This is the first post in a two part series where I will try and explain the importance of the lssrad output when you are reviewing the data output from the mpstat command.

In this first post, I will explain my understanding of the lssrad output and part two will cover the mpstat command and the association with the lssrad output.  Note that the output from mpstat is based on the lssrad output placement for logical processors in regard to thread redispatch and logical processor memory affinity.

 

Explaining the lssrad output.

# lssrad -av
REF1        SRAD          MEM        CPU
0
               0         7662.69     0-11

 

REF1 is the Node that the LPAR is running on.  This is referred to as the SECondary DOMAIN in a Hypervisor Resource Dump, although the SECondary DOMAIN numbers will not match the REF1 numbers.

Scale Out Servers with one processor module should only ever have a REF1 value that is equal to 0 (zero).

Scale Out Servers with two processor modules could have a REF1 value that is equal to 0 and 1.

Power 770's, 780's, and 870's with up to four nodes could have a REF1 value that is 0, 1, 2 or 3.

Power 795's with up to eight nodes could have a REF1 value that is 0, 1, 2, 3, 4, 5, 6 or 7.

Power E880's with up to four nodes could have a REF1 value that is 0, 1, 2 or 3.

 

SRAD is the Scheduler Resource Affinity Domain, similar to those used in Resource Sets in AIX.  This is referred to as the PRImary DOMAIN in a Resource Dump, although the PRImary DOMAIN numbers will not match the SRAD numbers.  These are the Chips within the Node that your LPAR has been allocated its home cores from.

MEM is the amount of memory that the LPAR has allocated from this REF1 (Node) and SRAD.

CPU lists the Logical CPU numbers in AIX that map to the Virtual Processors on this REF1 and SRAD.

 

Example 1

In this lssrad output, AIX has been told that all of the CPU and memory have been allocated from a single Chip in a single Node.

# lssrad -av
REF1        SRAD        MEM        CPU
0
               0        7662.69     0-11

 

Example 2

In this lssrad output, AIX has been told that the LPAR has been allocated CPU and memory from three different Chips in a single Node.

# lssrad -av
REF1        SRAD        MEM        CPU
0
               0      19456.00     0-11 16-19 28-31 40-43 52-55 64-67 76-79
                                   88-91 100-103 112-115 124-127
               1      13145.38     12-15 24-27 36-39 48-51 60-63 72-75 84-87
                                   96-99 108-111 120-123
               2      14820.00     20-23 32-35 44-47 56-59 68-71 80-83 92-95
                                   104-107 116-119

 

Example 3

In this lssrad output, AIX has been told that the LPAR has been allocated CPU from three different Chips in two Nodes and memory from five different Chips in 3 nodes.  I believe that extra memory was DLPAR'd in to this LPAR at a time when there was limited free memory, hence the memory being spread over many Nodes and Chips.  This output shows memory only SRAD's, where there is memory with no CPU.

# lssrad -av
REF1        SRAD        MEM        CPU
0
               0       32749.12    0-63
               1        9462.00    64-67 72-75 80-83 88-91
1
               2        2471.19    92-95
2
               3        1992.00
               4         249.00

 

From the AIX man page for mpstat we see the following.

S0rd ==> The percentage of thread redispatches within the same logical processor with scheduling affinity domain 0.

S1rd ==> The percentage of thread redispatches within the same physical processor or core with scheduling affinity domain 1.

S2rd ==> The percentage of thread redispatches within the same chip set, but not within the same processor core with scheduling affinity domain 2.

S3rd ==> The percentage of thread redispatches within the same MCM (multiple chip module) , but not within the same chip set with scheduling affinity domain 3.

S4rd ==> The percentage of thread redispatches on different MCMs within the same CEC or Plane with scheduling affinity domain 4.

S5rd ==> The percentage of thread redispatches on a different CEC or Plane with scheduling affinity domain 5.

 

So what exactly is an Affinity Domain?

Affinity Domain 0 means your running Thread was redispatched to the same Logical CPU as it previously ran.  This Logical CPU is mapped to the same Virtual Processor.

Affinity Domain 1 means your running Thread was redispatched to a different Logical CPU on the same Virtual Processor.

Affinity Domain 2 is for machines with Multi-Chip modules, but I am unable to confirm this.

Affinity Domain 3 means your running Thread was redispatched to a different Logical CPU on a Different Virtual Processor within the same Chip and same node.

Affinity Domain 4 means your running Thread was redispatched to a different Logical CPU on a Different Virtual Processor that is mapped to a Different Chip in the same node.

Affinity Domain 5 means your running Thread was redispatched to a different Logical CPU on a Different Virtual Processor that is mapped to a Different Chip that is in a Different Node.

 

Next we have the thread dispatch statistics relative to the memory that they need to access.  These statistics are generally accurate as the hypervisor must inform AIX where the memory even if it is spread across different chips and/or nodes.  This information generally correct as this cannot change unless you run a DPO operation.  

S3hrd ==> The percentage of local thread dispatches on this logical processor.

These are threads that have been dispatched to a core within the chip that has the memory modules directly attached.

S4hrd ==> The percentage of near thread dispatches on this logical processor.

These are threads that have been dispatched to a core on another chip within the same node to the chip that has the memory modules directly attached.  E.g.  There will be a chip to chip memory access for these threads.

S5hrd ==> The percentage of far thread dispatches on this logical processor.

These are thread that have been dispatched to a core on another chip in another node to the chip that has the memory modules directly attached.  E.g.  There will be a node to node memory access for these threads and this will be the slowest memory access.