MCL is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

MCL is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see http://www.gnu.org/licenses/.

Software Requirements

CMake 3.12+
Python 3.6+
C++ compiler (C++17 standard)
Fortran compiler (Fortran 2008 standard)
MPI library (MPI 2.1 standard)

Optional dependencies

GoogleTest & GoogleMock
pFUnit 4.4+

Compilation

An example of a compilation procedure is shown below:

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=<path-to-install> <path-to-mcl-source>
make
make install

Unit & Integration Tests

Building and running unit and integration tests requires GoogleTest and in case of a build with Fortran API also pFUnit . Note that you may need to provide the path to the CMake module directory of GoogleTest CMake module using the CMAKE_PREFIX_PATH CMake option. Building tests is an optional step enabled by the -DBUILD_TESTS=ON CMake option, for example:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTS=ON ..
make
ctest --output-on-failure

Usage

Execution of the server and client programs linked against the MCL depends on the selected communication mode. At the moment, MCL offers two MPI-based modes that are controlled by setting the environment variable MCL_COMM_MODE:

MCL_COMM_MODE=MPI_MPMD MPI-based multiple-program multiple-data mechanism [default]
MCL_COMM_MODE=MPI_PORT MPI-based port mechanism
MCL_COMM_MODE=TEST_STUB stub mechanism used for the interface testing in external programs

The actual execution of the server and client programs linked against the MCL depends on the selected communication mode chosen. The following examples feature options as provided by MPICH mpirun implementation.

MPI MPMD Communication Mode

If the MPI MPMD mechanism is used, then the commands to execute the programs will look like this:

export MCL_COMM_MODE=MPI_MPMD

mpirun -n X <server_exec> [<args1>] : -n Y <client1_exec> [<args2>] : -n Z <client2_exec> [<args3>]

Here, the first executable will be the server, and the following ones will become client number one and two.

MPI Port Communication Mode

If the MPI Port mechanism is used, then the programs are run as follows:

export MCL_COMM_MODE=MPI_PORT
export MCL_PORT_DIR=$PWD
mpirun -n X -env MCL_PROGRAM=0 <server_exec>  [<args1>] &
mpirun -n Y -env MCL_PROGRAM=1 <client1_exec> [<args2>] &
mpirun -n Z -env MCL_PROGRAM=2 <client2_exec> [<args3>] &
wait

where the MCL_PROGRAM label assigns the program IDs within the MCL environment. The program with MCL_PROGRAM=0 is the server and programs with MCL_PROGRAM=1 and MCL_PROGRAM=2 are client programs number one and two, respectively. Since this communication mode relies on opening ports to enable communication between independent MPI processes, as a measure to avoid waste of resources in case of a problem, we impose a time limit for MCL initialization, which can be adjusted with the MCL_TIMEOUT variable (integer value in milliseconds).

Test Stub Communication Mode

If the Test Stub mechanism is used, the program is executed like this:

export MCL_COMM_MODE=TEST_STUB
export MCL_TEST_DATA=<file_name>
export MCL_PROGRAM=<number>
mpirun -n X <exec> [<args>]

In this mode, no actual communication takes place, but instead, the data is read from a file provided through MCL_TEST_DATA. This file can be created during a normal simulation by setting MCL_RECORD_COMMUNICATION=ON. As a result, files named MCL_LOG_id containing the log of all data transfers from the point of view of the program 'id', which refers to the ID of the program in that particular simulation. To reduce the size of these files, it is possible to skip the recording of outgoing transfers (Send in the invoking program) by setting MCL_RECORD_DISCARD_SEND=ON, however, this also disables checks of Send transfers when these logs are used for a test run.

Note that when running a test, the program has to be executed with the same number of MPI processes as when the transfer log was recorded. Otherwise, this will result in discrepancies between the data expected by the external program, on the ones provided through this mechanism. The same applies to setting MCL_PROGRAM to the ID of the program in the simulation when the data were recorded.

Available APIs

Apart from the default C/C++ API (compatible with pure C codes), MCL also provides a Fortran 2008 compliant API that can be used in a Fortran-based program. Be sure to include mcl.mod in the compilation procedure and to link additionally against libmcl.a (static) or libmclf.so (shared). The Fortran API can be disabled by adding the -DBUILD_FORTRAN_API=OFF CMake option in the configuration step.

Implementation

To use MCL in the program, the communication layer has to be initialized by

  C/C++      int MCL_Initialize(void *param)
  Fortran    call mcl_initialize(communicator, err)

NOTE: This function must be called before any collective on MPI_COMM_WORLD is invoked as this will result in a deadlock! If the MPI MPMD communication mechanism is chosen, then one needs to pass a copy of the MPI_COMM_WORLD communicator stored in a variable, e.g., MPI_Comm world_comm = MPI_COMM_WORLD, into the function, which will then return a split communicator that will serve as world in the individual programs. For the MPI Port mode, nullptr will also work.

After these two steps the communicator is ready to send data by

  C          int MCL_Send(void *data, int length, int data_type, int tag, int destination)
  C++        int MCL_Send(dtype *data, int length, int tag, int destination)
  Fortran    call mcl_send(data, length, tag, destination, err)

and receive data by

  C          int MCL_Receive(void *data, int length, int data_type, int tag, int source)
  C++        int MCL_Receive(dtype *data, int length, int tag, int source)
  Fortran    call mcl_receive(data, length, tag, source, err)

in which dtype is the data type, and destination and source refer to the program ID.

Finally, the communication has to be deinitialized by invoking

  C/C++      int MCL_Finalize()
  Fortran    call mcl_finalize(err)

before MPI communication is finalized.

All C/C++ API functions return an error code.

Communication Protocol

MiMiC implements a request-based client-server communication protocol realized through MCL, in which the server that issues requests encoded as integer constants, tagged with MCL_REQUEST tag, to the clients. A client waits until a request is received via a MCL_Receive function call, after which it will perform the requested action, e.g., send forces or compute forces. Sending/receiving data should be tagged with the MCL_DATA tag. All requests that are available in MCL are defined in the REQUESTS file.