Controlling ParaStation MPI communication paths

ParaStation MPI uses different communication paths, see the section called “ParaStation pscom communication library” for details. In order to restrict or prioritize the paths to use, a number of environment variables are recognized by ParaStation MPI.

PSP_SHM or PSP_SHAREDMEM (deprecated)

Don't use shared memory for communication within the same node.

PSP_TCP

Don't use TCP for communication.

PSP_P4S or PSP_P4SOCK (deprecated)

Don't use ParaStation MPI p4sock protocol for communication.

PSP_MVAPI

Don't use Mellanox InfiniBand vapi for communication.

PSP_OPENIB

Don't use InfiniBand verbs for communication.

PSP_GM

Don't use GM (Myrinet) for communication.

PSP_DAPL

Don't use DAPL for communication.

To disable the particular transport, the corresponding variable must be set to 0. To enable a transport, the variable must be set to 1 or the variable must not be defined. To prioritize a communication path, assign a value greater than 1.

Using the environment variable PSP_LIB, it is possible to define the communication library to use, independent of the variables mentioned above. This library must match the currently available interconnect and protocol, otherwise an error will occur.

The library name must be specified using the full path and filename, e.g.

  export PSP_LIB=/opt/parastation/lib64/libpscomopenib.so

This variable is automatically exported to all processes started by ParaStation MPI. Refer to the section called “ParaStation pscom communication library” for a full list of available library variants.

If more than one path for a particular interconnect exist, e.g. if the nodes are connected by two Gigabit Ethernet networks in parallel, it is desirable to pretend the interface and therefore the network to be used for application data. To do so, the environment variable PSP_NETWORK has to be defined.

Assuming the network 192.168.1.0 is dedicated to management data and the network 192.168.2.0 is intended for application data, the following configuration within parastation.conf would re-direct the application data to the network 192.168.2.0:

  Env PSP_NETWORK 192.168.2.0
  Nodes {
    node0   0  # 192.168.1.1
    node1   1  # 192.168.1.2   
    ...
  }

Refer to ps_environment(5) for details.