Last modified 11 years ago Last modified on 02/21/11 16:19:38


This document describes how to build and install IOFSL v 0.1. IOFSL consists of several software components with software dependencies. A brief overview of the IOFSL software components, instructions for obtaining and installing external software dependencies, instructions for obtaining and installing IOFSL, and instructions for testing and verifying that IOFSL was built correctly are provided by this document.

IOFSL Software Overview

The IOFSL code consists out of multiple parts:

  • code on the client (e.g. compute node side):
    • fuse client for non MPI-IO programs
    • ROMIO driver for MPI-IO programs
  • The forwarding server
    • POSIX interface
    • PVFS2 interface
    • libsysio interface

Required External Software Dependencies

There are several external software components required to build IOFSL. This section describes how to obtain, install, and configure those components for IOFSL.

Flex and Bison

flex and bison are required to build IOFSL from source. Please ensure that flex and bison are in your path and can be found by the build environment.

More information on how to obtain lex, flex, yacc, or bison can be found here.

Boost C++ Libraries

IOFSL requires the Boost C++ libraries for a variety of functions. Obtain the Boost source code from .

To install Boost from source, issue the following commands:

$ ./ --with-libraries=date_time,program_options,regex,thread,test --prefix=/opt/boost
$ ./bjam  --prefix=/opt/boost --libdir=/opt/boost/lib --includedir=/opt/boost-1.39/include install


IOFSL uses the Parallel Virtual File System's (PVFS) Buffer Message Interface (BMI) as the transport layer for the server and client. IOFSL requires the latest version of BMI available through the PVFS SVN repository.

Obtain the latest PVFS source from the SVN repository using the following commands:

cvs -d :pserver:[email protected]:/anoncvs login
cvs -d :pserver:[email protected]:/anoncvs co pvfs2
cvs -d :pserver:[email protected]:/anoncvs logout

Once the PVFS source is retrieved from the repository, install BMI using the following commands:

$ ./configure --enable-bmi-only --prefix=/opt/bmi-2.8.1
$ make
$ make install

ROMIO and MPICH2 Client Interface

IOFSL uses ROMIO to provide a MPI-IO-enabled IOFSL interface to applications. Recent versions of MPICH2 (v1.3a2 and more recent) contain an IOFSL compatible ROMIO driver. To build MPICH2 with ZOIDFS support, run the following commands:

$ export CFLAGS=-I/path/to/zoidfs/headers
$ ./configure --enable-romio --with-file-system=ufs+zoidfs
$ make
$ make install

To build the latest version of MPICH2 with ZOIDFS support from the MPICH2 SVN repository, obtain the latest MPICH2 source from the SVN repository using the following commands:

$ svn checkout mpich2

The following commands will build ROMIO. Notice that the path to zoidfs.h must be added to the build process so that the IOFSL interface can be properly built. Within the IOFSL source tree, zoidfs.h is located in the src/zoidfs/ directory.

$ cd mpich2
$ ./maint/updatefiles
$ export CFLAGS=-I/path/to/zoidfs/headers
$ ./configure --enable-romio --with-file-system=ufs+zoidfs
$ make
$ make install

Optional External Software Dependencies


We use OpenPA for atomic primitives support. Instructions for obtaining and building OpenPA can be found here.

To build IOFSL with OpenPA support, use the --with-openpa=<location of openpa> flag.

FUSE Client Interface

The IOFSL client provides a POSIX-compatible client interface using a Filesystem in Userspace (FUSE) Linux kernel module. This interface requires that the FUSE libraries and development files are available on the IOFSL build target. The IOFSL configure processes automatically detects and builds support for FUSE if the FUSE development files and libraries are located

PVFS2 Server Interface

The IOFSL server provides an interface to PVFS2 file systems. This interface requires that the PVFS2 libraries and development files are available on the IOFSL build target. Use the --with-pvfs2 configure option to provide PVFS2 support for the IOFSL server.

libsysio Server Interface

IOFSL supports a libsysio ( server interface.

For libsysio support obtain the latest version of libsysio from the sourceforge CVS repository using the following commands:

$ cvs -d:pserver:[email protected]:/cvsroot/libsysio login 
$ cvs -z3 -d:pserver:[email protected]:/cvsroot/libsysio co -P libsysio

The following commands will build and install libsysio for IOFSL. Note that we build libsysio with alternate symbols (--with-alternate-symbols="_zfs_sysio_").

$ ./configure --prefix=/opt/libsysio --with-statvfs=yes --with-native-driver=yes --with-incore-driver=no \ 
                         --with-file-handle-interface=yes --with-threads=posix --with-stdfd-dev=no --with-stddev-dev=no \
$ make
$ make install

Building and Installing IOFSL

Once the necessary optional and required external software dependencies are installed, the IOFSL client and server can be built and installed.

The latest IOFSL software is available through the IOFSL Git repository (see GitRepository for instructions on how to access the public IOFSL repository).

The following commands will build IOFSL with the ROMIO client and POSIX server interfaces:

$ ./prepare
$ ./configure
$ make

To enable the IOFSL unit tests, CUnit and Boost is required. CUnit is available at . Make sure to get the tar.gz package since the package does not build on Linux hosts. After installing CUnit, IOFSL support for CUnit can be enabled using the --with-cunit=/path/to/cunit flag. Since IOFSL requries Boost, the Boost unit tests are enabled by default.

If MPICH2, BMI, or Boost are installed in non-standard locations, ensure that IOFSL is configured to search for the library and development files for these components using the --with-mpi=<dir>, --with-bmi=<dir>, and --with-boost=<dir> configure flags.

Using and Testing IOFSL

To start the IOFSL server, issue the following commands from the base of the IOFSL source code directory. In this example, we assume that the client and server will operate on the localhost TCP network interface using port 12345.

$ export ZOIDFS_ION_NAME="tcp://localhost:12345"
$ ./src/iofwd/iofwd --config config.cfg

An example configuration file can be found in the root directory of the source distribution. ( By default, the server will listen for requests at tcp://localhost:12345; To change this, see the frontend section in the configuration file.

The server uses BMI, and consequently supports all protocols supported by BMI: currently this is TCP, Myricom MX, Infiniband and the BG/P tree network.

Several self tests are included in the test directory in the IOFSL source code and are built by default during IOFSL installation. These tests include IO and metadata tests that verify the correctness of the IOFSL server interfaces. To build the following unit tests, IOFSL must be configured with CUnit support enabled (

$ export ZOIDFS_ION_NAME="tcp://localhost:12345"
$ ./test/unit-tests/zoidfs-io-cunit /tmp/iofsl-test-dir
$ ./test/unit-tests/zoidfs-md-cunit /tmp/iofsl-test-dir

The tests above expect the given directory to be empty and writable. It also assumes that a IOFSL server is running on the localhost on TCP port 12345.

There is also a set of automated unit tests. To run them, use make check.

To test the ROMIO client interface of IOFSL, file paths must be prefixed with zoidfs:/.