Motivation

Array variables comprise the bulk of the data in a netCDF dataset, and for accesses to large regions of single array variables, PnetCDF attains very high performance. However, the current PnetCDF interface only allows access to one array variable per call. If an application instead accesses a large number of small-sized array variables, this interface limitation can cause significant performance degradation, because high end network and storage systems deliver much higher performance with larger request sizes. Moreover, the record variables data is stored interleaved by record, and the contiguity information is lost, so the existing MPI-IO collective I/O optimization can not help.

We provided a new mechanism for PnetCDF to combine multiple I/O operations for better I/O performance. This mechanism can be used in a new function that takes arguments for reading/writing multiple array variables, allowing application programmers to explicitly access multiple array variables in a single call. It can also be used in the implementation of asynchronous I/O functions, so that the combination is carried out implicitly, without changes to the application. Our performance results demonstrate significant improvement using well-known application benchmarks.

Usage

There are two ways to use this feature:

Explicit Method: The caller explicitly accesses multiple variables at once. New routines added to the library (e.g. ncmpi_put_mvara_all) take a list of variables to access.

The function calls for the explicit method look like this:

int ncmpi_put_mvara_all(int ncid, int nvars, int varid[],
                   MPI_Offset *start[], MPI_Offset *count[],
                   void **buf, MPI_Offset *bufcount,
                   MPI_Datatype *datatype);

int ncmpi_get_mvara_all(int ncid, int nvars, int varid[],
                   MPI_Offset *start[], MPI_Offset *count[],
                   void **buf, MPI_Offset *bufcount,
                   MPI_Datatype *datatype);

Do note that we do not have Fortran bindings for these new routines in parallel-netcdf-1.1.0, but want very much to introduce Fortran bindings for this feature in a future release. In the meantime, Fortran codes can use the implicit method (below).

Implicit Method The library accesses multiple variables implicitly. Several variable accesses can be "scheduled" with the nonblocking routines. Then, when the application waits for completion of those accesses, the library will service them all in a single call.

Parallel-netcdf has had nonblocking routines in the C interface for some time. With this release we extend those routines to the Fortran interface. This feature is deliberately similar to the MPI non-blocking routines:

In C:

ncmpi_iget_vara_all
ncmpi_iput_vara_all
ncmpi_wait
ncmpi_waitall

In Fortran:

nfmpi_iget_vara_all
nfmpi_iput_vara_all
nfmpi_wait
nfmpi_waitall

In the 'wait' and 'waitall' methods, the library actually batches up all outstanding nonblocking operations. In this way, we can carry out this optimization without needing a thread.

References

You can see examples of this new API in action in the test/mcoll_perf directory: https://trac.mcs.anl.gov/projects/parallel-netcdf/browser/trunk/test/mcoll_perf

We wrote a paper describing the API, the implementation, and some results:

Kui Gao, Wei-keng Liao, Alok Choudhary, Robert Ross, and Robert Latham. "Combining I/O Operations for Multiple Array Variables in Parallel NetCDF". In the Proceedings of the Workshop on Interfaces and Architectures for Scientific Data Storage, held in conjunction with the the IEEE Cluster Conference, New Orleans, Louisiana, September 2009. PDF