Tell the ParaStation MPI daemon to find all the ParaStation MPI related files in inst-dir. The default is /opt/parastation.

Tell the ParaStation MPI daemon how to handle a distinct hardware. Usually it is not necessary to edit these entries, since the template version of the configuration file contains up to date entries of all supported hardware types. Furthermore a deeper insight into the low-level functionality of ParaStation MPI is needed in order to create such an entry.

Nevertheless a brief overview on the structure of the Hardware entries is given here.

The following five types of parameters within the Hardware environment will get a special handling from the ParaStation MPI daemon psid(8). These define different script files called in order to execute various operations towards the corresponding communication hardware.

All these entries have the form of the parameter's name followed by the corresponding value. The value might be enclosed by single or double quotes in order to allow a space within.

The values are interpreted as absolute or relative paths. Relative paths will be looked up relative to InstallDir. If one or more of the scripts are not defined, no corresponding action will take place for this hardware.

All further parameters defined within a Hardware section are interpreted as environment variables when calling the above defined scripts. Again these parameters have the form of the parameters name - interpreted as the environments variables name - followed by the corresponding value. The values might be single strings not containing whitespace characters or enclosed by single or double quotes, too.

The impact of the environment variables on the scripts of course depend on the scripts itself.

Various hardware types are defined within the template configuration file coming with the ParaStation MPI software distribution. These hardware types, the corresponding scripts and the environment variables the scripts understand are briefly discussed within the following lines.

This configuration parameter is no longer required and will be silently ignored.

Define the default communication hardware available on the nodes of the ParaStation MPI cluster. This may be overruled by an explicit HWType option in a Node statement.

The hardware types used within this command have to be defined in Hardware declarations before.

Further hardware declarations might be defined by the user, but this is pretty much undocumented.

It is possible to enable more than one hardware type, either as default or on a per node basis.

The default value of HWType is none.

If the argument is one of yes, true or 1, all nodes declared within a Node statement will allow to start parallel tasks, unless otherwise stated.

If the argument is one of no, false or 0, starting will be not allowed.

It might be useful to prohibit the startup of parallel task from the frontend machine if a batch system is used. This will force all users to use the batch system in order to start their tasks. Otherwise it would be possible to circumvent the batch system by starting parallel task directly from the frontend machine.

The default is to allow the starting of parallel tasks from all nodes.

If the argument is one of yes, true or 1, all nodes declared within a Node statement will allow to run processes of parallel tasks, unless otherwise stated.

If the argument is one of no, false or 0, ParaStation MPI will not start processes on these nodes.

It might be useful to prohibit the start of processes on a frontend machine since usually this machine is reserved for interactive work done by the users. If the execution of processes is forbidden on a distinct node, parallel tasks might be started from this node anyhow.

The default is to allow all nodes to run processes of parallel tasks.

Define one or more nodes to be part of the ParaStation MPI cluster.

This is the first example of a parameter that supports the environment mode. This means there are two different notations to use this parameter. The first one may be used to define a single node, the second one will allow to register more than one node within a single command. It is a convenient form that prevents from typing the keyword once per entry again and again.

Each entry has to have at least two items, the hostname and the id. This will tell the ParaStation MPI system that the node called hostname will act as the physical node with ParaStation MPI ID id.

hostname is either a resolvable hostname or an IP address in dot notation (e.g. 192.168.1.17). Id is an integer number in the range from 0 to maximum number of nodes minus one.

Further optional items as HWType-entry, starter-entry or runJobs-entry may overrule the default values of the hardware type on the node, the ability to start parallel jobs from this node or the possibility to run processes on this node respectively. These entries have the same syntax as the stand alone commands to set the corresponding default value.

E.g. the line

Node node17 16 HWType { ethernet p4sock } starter yes runJobs no

will define the node node17 to have the ParaStation MPI ID 16. Furthermore it is expected to have a Ethernet communication using both TCP and p4sock protocols. It is allowed to start parallel tasks from this node but the node itself will not run any process of any parallel task (except the ParaStation MPI logger processes of the tasks started on this node).

The option environment or env allows per node environment variables to be set. Using the first form, the variable name is set to value. More then one name/value pair may be given. More complex values may be given using quotation marks:

Node node17 16 environment LD_LIBRARY_PATH /mypath
Node node18 17 env { PSP_P4S "2" PSP_OPENIB "0" }

This example will define the variable LD_LIBRARY_PATH to /mypath for node node17 and the variables PSP_P4S and PSP_OPENIB to 2 and 0 for node node18.

The $GENERATE allows to define a group of nodes at once using a simple syntax. Using the parameters from and to, a range may be defined, incremented by step. Each entry in this range may be referenced within the nodestr and idstr using a syntax of $[{offset[,width[,base]]}]. Eg., the entry

$GENERATE 1-96  node${0,2} ${0}

define the nodes node01 up to node96 using the id's 1 - 96, respectively. More node specific attributes may be defined as described above.

These entries are silently ignored by this version of ParaStation MPI.

Set the timeout of the central select(2) of the ParaStation MPI daemon psid(8) to time seconds.

The default value is 2 seconds.

The ParaStation MPI daemon psid(8) will declare other daemons as dead after num consecutively missing multicast pings.

After declaring a node as dead, all processes residing on this node are also declared dead. This results in sending signals to all processes on the local node that have requested to get informed about the death of one of these processes.

The default value is 10.

For now, the multicast period is set to two seconds, i.e. every daemon sends a multicast ping every two seconds. This results in declaring a daemon as dead after 20 seconds for the default value.

Set the debugging level of the ParaStation MPI daemon psid(8) to num.

Set the logging output's destination for the ParaStation MPI daemon psid(8). Usually the daemon prints logging output using the syslog(3) mechanism, unless an alternative logging file is requested via psid(8)'s -l option.

In order to collect all the ParaStation MPI specific log messages into a special file, the facility argument of the openlog(3) function call in cooperation with a suitable setup of the syslogd(8) may be used. This parameter will set the argument to one of the mentioned values.

The default value is LOG_DAEMON.

Tell psid(8) to use the multicast group group-num for multicast communication to other daemons.

The default group to use is 237

Tell psid(8) to use the UDP port portno for multicast communication to other daemons.

The default port to use is 1889

Tell psid(8) to use the UDP port portno for the RDP communication protocol to other daemons.

The default port to use is 886.

Set various resource limits to the psid(8) and thus to all processes started from it.

All limits are set using the setrlimit(2) system call. For a detailed description of the different types of limits please refer to the corresponding manual page.

If no RLimits are set within the ParaStation MPI configuration files, no changes are made to the systems default value.

The following (soft) resource limits may be set:

Set environment variables for the ParaStation MPI daemon psid(8) and any application started via this daemon.

This command again has two different modes. While within the first form exactly one variable is set, within the environment form of this command as many variables as wanted may be set. The general form of the latter case is one variable per line.

The value part of each line either is a single word or an expression enclosed by single or double quotes. The expression might contain whitespace characters. If the expression is enclosed by single quotes, it is allowed to use balanced or unbalanced double quotes within this expression and vice versa.

This command might be used for example in order to set the PSP_NETWORK environment variable globally without the need of every user to adjust this parameter in his own environment.

If the argument is one of yes, true or 1, suspending a task by sending the signal SIGTSTP to the logger will handle all resources (CPUs) currently claimed by this task as free.

If the argument is one of no, false or 0, ParaStation MPI will not claim resources as free after sending SIGTSTP.

If the argument is one of yes, true or 1, compatibility mode for applications linked with ParaStation MPI version 4.0 up to 4.0.6 will be enabled. Keep in mind that this behavior might collide with the freeOnSuspend feature.

If the argument is one of no, false or 0, ParaStation MPI will disable compatibility mode.

If the argument is one of yes, true or 1, keep alive messages from the ParaStation MPI daemon psid(8) are sent using Multicast messages.

If the argument is one of no, false or 0, ParaStation MPI will use it's own RDP protocol for keep alive messages. This is the default.

Define the default sorting strategy for nodes when attaching them to a partition. The different possible values have the following meaning:

This only comes into play, if the user does not define a sorting strategy explicitly via PSI_NODES_SORT. Be aware of the fact that using a batch-system like PBS or LSF *will* set the strategy explicitly, namely to NONE.

If the argument is one of yes, true or 1, all nodes may be overbooked by the user using the PSI_OVERBOOK environment variable.

If the argument is one of no, false or 0, ParaStation MPI will deny overbooking of the nodes, even if PSI_OVERBOOK is set.

It might be useful to prohibit the start of processes on a frontend machine since usually this machine is reserved for interactive work done by the users. When the execution of processes is forbidden on a distinct node, parallel task might be started from this node anyhow.

The default is to allow all nodes to run processes of parallel tasks.

Define the maximum number of processes per node.

This parameter can be set during runtime via the set maxproc directive within the ParaStation MPI administration and management tool psiadmin(1).

Enables or disables process pinning for compute tasks. If enabled, tasks will be pinned down to particular CPU-slots. The mapping between those CPU-slots and physical CPUs and cores is made using a mapping list. See CPUmap below.

The pinProcs parameter can be set during runtime via the set pinprocs directive within the ParaStation MPI administration and management tool psiadmin(1).

This parameter must be set to true if nodes providing non-Uniform memory access (NUMA) should use 'local' memory for the tasks.

This parameter can be set during runtime via the set bindmem directive within the ParaStation MPI administration and management tool psiadmin(1).

Set the map used to assign CPU-slots to physical cores to map. Map is a quoted string containing a space-separated permutation of the number 0 to Ncore-1. Here Ncore is the number of physical cores available on this node. The number of cores within a distinct node may be determined via list hw. The first number in map is the number of the physical core the first CPU-slot will be mapped to, and so on.

This parameter can be set during runtime via the set bindmem directive within the ParaStation MPI administration and management tool psiadmin(1).

This parameter must be set to true if processes spawned by ParaStation MPI should belong to all groups defined for this user. Otherwise, they will only belong to the primary group.

This parameter can be set during runtime via the set supplementaryGroups directive within the ParaStation MPI administration and management tool psiadmin(1).

Set the maximum number of retransmissions within the RDP facility. If more than this number of retransmission would have been necessary to deliver the packet to the remote destination, this connection is declared to be down.

See also psiadmin(1).

Set the maximum number of status broadcasts per round. This is used to limit the number of status-broadcasts per status-iteration. Too many broadcast might lead to running out of message-buffers within RDP on huge clusters.

If more than this number of broadcasts are triggered during one status-iteration, all future broadcasts will be ignored. The corresponding counter is reset upon start of the next status iteration.

A value of 0 will completely suppress sending of status-broadcasts. In this case information on dead nodes will be propagated by sending ACTIVENODES messages upon receive of too many wrong LOAD messages, only.

Only relevant, if MCast is *not* used.

See also psiadmin(1).

The timeout of the actual timer registered by RDP in milliseconds. Each time the corresponding timer is elapsed, handleTimeoutRDP() is called handling all resend activities necessary. This parameter steers the actual load introduced by RDP. Within the daemon, there is a lower limit for all timeout-timers of 100 msec. Thus, the minimal value here is 100, too.

Dead-limit of the RDP status module. After this number of consecutively missing RDP-pings the master declares the node to be dead.

Only relevant, if MCast is *not* used.

Timeout of the RDP status module. After this number of milliseconds a RDP-ping is sent to the master daemon. Additionally, the master daemon checks for received ping-messages. Within the daemon, there is a lower limit for all timeout-timers of 100 msec. Thus, the minimal value here is 100, too.

Only relevant, if MCast is *not* used.

The closed timeout within the RDP facility in milliseconds. If a RDP-connection is closed, during this timeout all messages from the corresponding partner are ignored. Thus, reconnection is avoided during this period. This helps handling packets still on the wire on connection close.

The resend timeout within the RDP facility in milliseconds. If a pending message is available and not yet acknowledged, this is the timeout after which the message is retransmitted to the remote host.

The maximum number of pending ACKs within the RDP facility. If this number of packets is received from a remote node consecutively without any retransmission, an explicit ACK is sent. Otherwise the ACK is sent piggyback within the next regular packet to this node or as soon as a retransmission occurred.

If set to 1, each RDP packet received is acknowledged by an explicit ACK.

Enable pinning of processes to distinct processor cores.

Enable binding of processes to distinct memory nodes on NUMA systems.

Define map assigning logical process-slots to physical processor cores.

Enable users to influence local mapping of processes via providing a __PSI_CPUMAP environment variable.

This script is called during startup of the ParaStation daemon. It's either an absolute path or relative to InstallDir. Parsing the configuration file will fail, if the script is not found. Depending on its return value the daemon continues startup without any action (0), some output of the script is written to the daemons log (-1) or the daemon stops immediately (-2).

As a default no script is defined and, thus, nothing is called.

Maximum number of tries to stat() an executable before spawning new processes. Increasing this number might help on overloaded NFS-servers.

Flag the RDP statistics. If set to 1, statistics on total number of messages sent and mean-time to ACK are determined per connection.

This script is called by the currently elected master of the ParaStation daemons every time a node becomes active from ParaStation's point of view. I.e. the node's daemon connects to the master daemon for the first time after being down.

This might be used to pass this type of information into a batch-system or some monitoring facility like the GridMonitor.

While calling the script two additional arguments are passed within the environment:

As a default no script is defined and, thus, nothing is called.

This script is called by the currently elected master of the ParaStation daemons every time a node goes down from ParaStation's point of view. I.e. the node's daemon disconnects from the master daemon after being connected.

This might be used to pass this type of information into a batch-system or some monitoring facility like the GridMonitor.

While calling the script two additional arguments are passed within the environment:

As a default no script is defined and, thus, nothing is called.