ParaStation provides its own communication protocol for Ethernet, called p4sock. This protocol is designed for extremely fast and reliable communication within a closed and homogeneous compute cluster environment.
The protocol implements a reliable, connection-oriented communication layer, especially designed for very low overhead. As a result, it delivers very low latencies.
The p4sock protocol is encapsulated within the kernel module p4sock.ko. This module is loaded on system startup or whenever the ParaStation daemon psid(8) starts up and the p4sock protocol is enabled within the configuration file parastation.conf(5).
The p4sock.ko module inserts a number of
entries within the /proc filesystem. All
ParaStation entries are located within the subdirectory
/proc/sys/ps4. Three different
subdirectories, listed below, are available.
To read a value, e.g. just type
# cat /proc/sys/ps4/state/connections
to get the number of currently open connections. To modify a value, for e.g. type
# echo 10 > /proc/sys/ps4/state/ResendTimeout
to set the new value for ResendTimeout.
Within this state directory, various
entries showing protocol counters.
All these entries, except polling, are
read only!
HZ: reads the number of timer
interrupts per second for this kernel ("jiffies").
A jiffy is the base unit for system timers, used by the Linux kernel. So all timeouts within the kernel are based on this timer resolution. On kernels with version 2.4, this it typically 100Hz (= 10ms). But there are kernel versions available, e.g. for newer SuSE Linux versions, which include patches to change this to a much higher value!
connections: reads the current number of open
connections.
polling: returns the current value
for the polling flag: 0 = never poll, 1 = poll if
otherwise idle (number of runable processes < number
of CPUs), 2 = always poll. Writing this value will
immediately change the polling strategy.
recv_net_ack: number of received
ACKs.
recv_net_ctrl: number of received
control packets (ACK, NACK, SYN, SYNACK, ...).
recv_net_data: number of received
data packets.
recv_net_nack: number of received
NACKs.
recv_user: number of packets
delivered to application buffers.
send_net_ack: number of sent ACKs.
send_net_ctrl: number of sent
control packets.
send_net_data: number of sent data
packets.
send_net_nack: number of sent
NACKs.
send_user: number of packets sent
by the application.
sockets: number of open sockets
connecting to the ParaStation protocol module.
timer_ack: number of expired
delayed ACK timers.
timer_resend: number of expired
resend timers.
Within this directory, all Ethernet related parameters for the ParaStation p4sock protocol are grouped. All these entries can be read and written, newly written values will be used immediately.
AckDelay: maximum delay in "jiffies"
for ACK messages. If no message is sent within this time
frame, where an ACK for already received packets can be
"hooked up", a single ACK message will generated.
Must be less then ResendTimeout.
MaxAcksPending: maximum number of
pending ACK messages until an "urgent" ACK messages will
be sent.
MaxDevSendQSize: maximum number of
entries of the (protocol internal) send queue to the
network device.
MaxMTU: maximum packet size used
for network packets. For sending packets, the minimum of
MaxMTU and service specific MTU
will be used.
MaxRecvQSize: size of the protocol
internal receive queue.
MaxResend: Number of retries until
a connection is declared as dead.
MaxSendQSize: size of the protocol
internal send queue.
ResendTimeout: delay in "jiffies"
for resending packets not acknowledged up to now. Must
be greater then AckDelay.
The command p4stat can be used to list open sockets and network connections of the p4sock protocol.
$ /opt/parastation/bin/p4stat -s
Socket #0 : Addr: <00><00><00><00><00><'........' last_idx 0 refs 2
Socket #1 : Addr: <70><6f><72><74><33><'port384.' last_idx 0 refs 10
Socket #2 : Addr: <70><6f><72><74><31><'port144.' last_idx 0 refs 10
$ /opt/parastation/bin/p4stat -n
net_idx SSeqNo SWindow RSeqNo RWindow lusridx lnetidx rnetidx snq rnq refs
84 30107 30467 30109 30468 84 84 230 0 0 2
85 30106 30466 30106 30465 85 85 231 0 0 2
86 30107 30467 30109 30468 86 86 84 0 0 2
87 30106 30466 30106 30465 87 87 85 0 0 2
88 30107 30467 30109 30468 88 88 217 0 0 2
89 30106 30466 30106 30465 89 89 218 0 0 2
90 30106 30466 30106 30465 90 90 220 0 0 2
91 30106 30466 30106 30465 91 91 221 0 0 2
92 30001 30361 30003 30362 92 92 232 0 0 2
93 30001 30361 30003 30362 93 93 219 0 0 2
94 30000 30000 30001 30360 94 94 233 0 0 2
95 30000 30000 30001 30360 95 95 222 0 0 2
96 30000 30000 30001 30360 96 96 222 0 0 2
This command shows some protocol internal parameters, like open connections, sequence numbers, reference counters, etc. For more information, see p4stat(8).