SLASH2 File System
SLASH2 is an open source wide area network friendly distributed file system featuring multi residency at the file chunk level, system-managed data transfer, inline checksum verification, and much more.
Several needs arise concerning the management of large data sets:
- geographical replication for access locality
- replicas for valuable data
- the continuing emergence of cloud computing and the need for universal interfaces
- research collaboration
- data set migration
In solving these issues, often the burden is placed upon users themselves. This requires all researchers needing access to data sets to learn tools and deal with environments to manage data transfers on their own. Replication is often doing manually instead of being handled by the system according to policy.
Current Approaches and Their Issues
The current toolset of choices storage system administrators have is often at a level and interface too high or too low which gives researchers either too much or not enough control over this data management.
High level data management interfaces (from GridFTP to scp) are built on top of the storage system and completely externalize operations such as replica management. This places the burden of learning interfaces to utilitize these features as well as monitoring failed transfers and retries on the user. Also, users vie for network performance depending other system activity. This means at certain times network resources are under and over utilized. With very large data sets, this problems escalates further out of control.The low level approach adapts parallel file system techniques to the WAN. This places the burden of verifying file integrity of replicas on the user if unreliable transports are used during transmission.
The SLASH2 Solution
SLASH2 handles these tasks at the system level, alleviating burden from the user. It provides a POSIX interface and a small set of additional tools for managing replicas. SLASH2 also imposes no restrictions on underlying storage systems which allows administrators to run their systems unaltered as well as support many diverse types of storage systems.
More information is available in our white paper.
Technical System Architecture
The following diagram shows the internal structure of the metadata server:
See design documents for more details.