Chapter 5. Insight ParaStation MPI

Table of Contents

ParaStation pscom communication library
ParaStation protocol p4sock
Directory /proc/sys/ps4/state
Directory /proc/sys/ps4/ether
Directory /proc/sys/ps4/local
p4stat
Controlling process placement
Using the ParaStation MPI queuing facility
Exporting environment variables for a task
Using non-ParaStation MPI applications
ParaStation MPI TCP bypass
Controlling ParaStation MPI communication paths
Authentication within ParaStation MPI
Homogeneous user ID space
Single system view
Parallel shell tool
Nodes and CPUs
Integration with AFS
Integrating external queuing systems
Integration with PBS PRO
Integration with OpenPBS
Integration with Torque
Integration with LSF
Integration with LoadLeveler
Multicasts
Copying files in parallel
Using ParaStation MPI accounting
Using ParaStation MPI process pinning
Using memory binding
Spawning processes belonging to all groups
Changing the default ports for psid(8)
Relocating packages during installation

This chapter provides more technical details and background information about ParaStation MPI.

ParaStation pscom communication library

The ParaStation MPI communication library libpscom offers secure and reliable end-to-end connectivity. It hides the actual transport and communication characteristics from the application and higher level libraries.

The libpscom library supports a wide range of interconnects and protocols for data transfers. Using a generic plug-in system, this library may open connections using the following networks and protocols:

  • TCP: uses standard TCP/IP sockets to transfer data. This protocol may use any interconnect. Support for this protocol is built-in to the libpscom.

  • P4sock: uses an optimized network protocol for Ethernet (see the section called “ParaStation protocol p4sock, below). Support for this protocol is built-in to the libpscom.

  • InfiniBand: based on a verbs kernel layer and a libverbs library, typically provided by the hardware vendor, the libpscom may use InfiniBand to actually transfer data. The corresponding plug-in library is called libpscom4openib.

  • Myrinet: using the GM library and kernel level module, the libpscom library is able to use Myrinet for data transfer. The particular plug-in library is called libpscom4gm.

  • Shared Memory: for communication within a SMP node, the libpscom library uses shared memory. Support for this protocol is built-in to the libpscom.

  • DAPL: The libpscom supports a DAPL transport layer. Using the libpscom4dapl plug-in, it may transfer data across various networks like InfiniBand or 10G Ethernet using a vendor-provided libdapl.

  • QsNet: The libpscom supports the QsNetII transport layer. Using the libpscom4elan plug-in, it may transfer data using the libelan.

The interconnect and protocol used between two distinct processes is chosen while opening the connection between those processes. Depending on available hardware, configuration (see the section called “Configuration of the ParaStation MPI system”) and current environment variables (see the section called “Controlling ParaStation MPI communication paths”), the library automatically selects the fastest available communication path.

The library routines for sending and receiving data handle arbitrary large buffers. If necessary, the buffers will be fragmented and reassembled to meet the underlying transport requirements.

The application is dynamically linked with the libpscom.so library. At runtime, this library loads plug-ins for various interconnects, see above. For more information on controlling ParaStation MPI communication pathes, refer to the section called “Controlling ParaStation MPI communication paths”.