HPC computation results in the terabyte range are not uncommon. The problem in this context is not so much storing the data at rest, but the performance of the necessary copying back and forth in the course of the computation job flow and the dependent job turn-around time.
For interim results during a job runtime or for fast storage of input and results data, parallel file systems have established themselves as the standard to meet the ever-increasing performance requirements of HPC storage systems.
The open-source solutions BeeGFS from the Fraunhofer Competence Center for High Performance Computing or Lustre are widely used in the Linux HPC world, and also several other free as well as commercial parallel file system solutions like Panasas exist.
Parallel NFS (pNFS) is gradually emerging as the future standard to meet requirements in the HPC environment. From the industry’s as well as the user’s perspective, the benefits of utilising standard solutions are indisputable: besides protecting end user investment, standards also ensure a defined level of interoperability without restricting the choice of products available. As a result, less user and administrator training is required which leads to simpler deployment and at the same time, a greater acceptance. As part of the NFS 4.1 Internet Standard, pNFS will not only adopt the semantics of NFS in terms of cache consistency or security, it also represents an easy and flexible extension of the NFS 4 protocol.
transtec has a great experience in deploying all of the existing parallel file system technologies: Lustre, BeeGFS (previously known as FraunhoferFS or FhGFS), IBM’s GPFS, and Panasas. Our overall experience and customer-oriented approach assures that customers get the best possible HPC storage solution available.