Performance of the stream* Data-parallel file system

Although hardware supporting parallel file I/O has improved greatly since the introduction of first-generation parallel computers, the programming interface has not. Each system provides a different logical view of parallel files as well as non portable operations for manipulating files. Neither do parallel languages provide standards for performing I/O. In this paper we describe a view of parallel files for data-parallel languages, dubbed stream*, in which each virtual processor writes to and reads from its own stream. In this scheme each virtual processor''s I/O operations have the same familiar, unambiguous meaning as in a sequential C program. We demonstrate how I/O operations in stream* can run as fast as those of vendor-specific parallel file systems on the operations most often encountered in data-parallel programs. We show how this system supports general virtual processor operations for debugging and elemental functions. Finally, we present empirical results from a prototype stream* system r