Reverse I/O Stream library (RIOS)

Table of Contents

1 About RIOS

Reverse mode IO stream (RIOS) are C++ I/O streams to support large stacks, in reverse mode automatic differentiation (AD). It uses

  • asynchronous IO, with either MPI-IO or AIO
  • IO in blocks
  • configurable buffer size

2 License

RIOS is licensed with the GNU General Public License v3 (GPL). See the file COPYING and the main header file rios.hh for details.

3 Obtaining

The RIOS project's source code is kindly hosted at

The project info page is and the project home page is

You can download the latest version from the project info page or pick any of the tarballs at the file release page

4 Requirements

RIOS can be with or without MPI-IO support, and works with any of the major MPI libraries. However we recommend to use MPICH2.

5 Build

5.1 Build types

Basically there are two ways to use RIOS:

  1. compile a library and link you program to it. Together with the library the test program stream-test is also build.
  2. just include the generated all-sources file or for Windows builds into you program.

5.2 Building the library

Invoke make, possibly with setting the MPI_IMPL variable to an adequate value. The Makefile variable MPI_IMPL controls the use of MPI-IO. Possible values are:

  • none
  • openmpi
  • openmpi_naked
  • mpich2
  • mpich2_naked
  • mpich2_compiled
  • mpich2_compiled_naked

The default value is mpich2.

The setting none means no MPI-IO support. In this case the macro RIOS_WITH_MPI is not defined, in all other cases it is defined.

Here <name>_naked means that the regular C++ compiler is used and the required libraries added manually. This will likely break sooner or later as the set of libraries changes in one MPI distro ro the other.

Further, the idea is that openmpi and mpich2 use the system libraries when the relevant packages have been installed on a Debian system.

The setting mpich2_compiled is for an MPICH2 library compiled and installed by hand. It's location is determined by the variable MPICH2_HOME (mpich2_compiled) or MPICH2_INC and MPICH2_LIB (in the case of mpich2_compiled_naked). These are variables that are commonly set by the module environment on a cluster.

make MPI_IMPL=mpich2

5.2.1 Configuring

The is not configure script yet, but the file config.h can be used to set certain macros. For example, when building on a non-GNU system, which does not have the aio_init function, comment out the HAVE_AIO_INIT macro there.

5.3 Builting the all-sources file

The two all-sources files and are already present in the sources. They have a target in the Makefile that depends on Makefile, so when you change the list of library source files in the Makefile, they will be regenerated.

6 Install

To install the library and headers invoke make install and set the variable prefix to the directory where the file should go:

make install prefix=/usr/local/rios

7 Usage

7.1 Setting up

To use RIOS in your project, include the main header file rios.hh. In addition, link your program to you library or also include the all-sources file (or for Windows).

7.2 Initialization and shutting down

At the beginning of your program, you should call RIOS::init(). This will take care of initializing the MIP library. If you use RIOS without MPI-IO support, this may be omitted.

Likewise, at the end of your program, call RIOS::finalize(). If you use RIOS without MPI-IO support, this may be omitted.

7.3 Creating RIOS streams

The function to create a RIOS iostream is called RIOS::create, with the following signature:

RiosStreamWrapper *RIOS::create(std::string const &outfileName, RiosStreamType const streamType, 
				size_t bufferSize, size_t numBuffers, int verbosity);

streamType specifies the type of stream you wish to create:

  • RIOS_nullstream
  • RIOS_stringstream
  • RIOS_fstream
  • RIOS_astream
  • RIOS_SBB_Files
  • RIOS_MB_SyncIO
  • RIOS_MB_OSHints
  • RIOS_CFile
  • RIOS_WinFile

Some of these are experimental and not all are available on all systems. We recommend that you use RIOS_MB_AIO or RIOS_MB_MPIO. Using RIOS_stringstream or RIOS_fstream will create a plain std::stringstream or std::fstream, resp. RIOS_CFile will use a streambuffer that is a thin wrapper around plain C I/O.

outfileName is the name of the file on disk (ignored in some cases, and a template in the case of RIOS_SBB_Files).

bufferSize specifies the size of the main memory buffer to use.

numBuffers specifies the number of the buffers to use.

verbose specifies the verbosity. The higher the number, the more messages you will get.

The function returns an RiosStreamWrapper object that hold the iostream created. It has two methods:

  • getStream returns a pointer to the std::iostream
  • deleteFiles is a utility function to delete the files created by the stream. This is useful in the case of RIOS_SBB_Files, which creates several files.

7.4 Releasing RIOS streams

To release the resources associated with a RIOS stream, pass the pointer returned by RIOS::create to the RIOS::release function.

8 Configuration

Configuration of RIOS streams is done in two ways: The main parameters, which are the stream type, the buffer size, and the number of buffers to use are given to the RIOS::create function. Further configuration details are passed to the RIOS library via environment variables.

8.1 Environment variables

  • RIOS_USE_AIO_INIT: whether to call aio_init when AIO is used (0 or 1). Default is 1
  • AIO_THREADS: set the number of threads AIO should use
  • AIO_NUM: set the expected number of simultaneous requests
  • AIO_IDLE_TIME: set the idle timeout for AIO threads
  • RIOS_BUFFER_CACHING: Whether to use posix_fadvise to turn OS buffer caching off. Only used when the UNIX file descriptor is known, i.e. with RIOS_MB_AIO, RIOS_MB_SyncIO, and RIOS_MB_OSHints.
  • RIOS_ODIRECT: When set to 1, try to turn off system caches and buffers. With UNIX files, this adds the flag O_DIRECT when the file is open()'ed. On Windows, this add the flag FILE_FLAG_NO_BUFFERING to the CreateFile call.
  • RIOS_MPI_WAITSOME_VARIANT: whether to use MPI_Waitsome from the MPI library (0) or the custom implementation that is included in RIOS, myMPI_Waitsome (1). Default is 1.

Author: Johannes Willkomm

Created: 2013-09-03 Di 10:27

Emacs 23.4.1 (Org mode 8.0.7)

Validate XHTML 1.0