Just when you thought you understood idle CPU time!!
Myself and many other people were of the opinion that the percentage columns for user, system, wait and idle in the vmstat output were percentages in relation to the Physical Consumed (PC) column, well this as I discovered recently, this is not always true.
From the man page for vmstat it states:
If the current physical processor consumption of the uncapped partitions exceeds the entitled capacity, the percentage becomes relative to the number of physical processor consumed (pc).
What the man page fails to state is that if the current physical processor consumption of the uncapped partition is less than the entitled capacity, the percentage is in relation to the entitled capacity of the LPAR.
The idle system - no workload except vmstat running.
Note the LPAR has 4 Virtual Processors and starts with an Entitlement of 1 core.
r b p w avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec hr mi se
0 0 0 0 443929 1585346 0 0 0 0 0 0 1 372 173 0 0 99 0 0.01 0.8 19:18:09
0 0 0 0 443930 1585345 0 0 0 0 0 0 0 50 167 0 0 99 0 0.00 0.4 19:18:10
0 0 0 0 443930 1585345 0 0 0 0 0 0 2 100 206 0 0 99 0 0.01 0.5 19:18:11
0 0 0 0 443930 1585345 0 0 0 0 0 0 5 50 169 0 0 99 0 0.00 0.4 19:18:12
Now starting 1 spinning CPU thread.
So a single thread running gives the 59 user and 41 idle percentages as expected for p8.
1 0 0 0 443978 1585296 0 0 0 0 0 0 1 47 167 59 0 41 0 1.00 100.3 19:18:50
1 0 0 0 443978 1585296 0 0 0 0 0 0 5 99 207 59 0 41 0 1.00 100.4 19:18:51
1 0 0 0 443978 1585296 0 0 0 0 0 0 1 56 174 59 0 41 0 1.01 100.5 19:18:52
1 0 0 0 443978 1585296 0 0 0 0 0 0 1 47 169 59 0 41 0 1.00 100.2 19:18:53
Now I add more Entitlement using DLPAR.
Upping the Entitlement to 2 changes the Entitled percentage from 100% to 50%, but does not change the PC column, we are still running a single spinning thread on 1 CPU. Yet, user drops to 30%, half of the 60% in the above test. We just added 100% more Entitlement, and idle increases to 70%, that’s 50% idle for the extra entitlement, plus 20% of the Entitlement that is being used (20% of 2 Entitlement = 40% of 1 Entitlement).
So if the LPAR is below entitlement, these values are percentages of the Entitlement, not the PC column.
1 0 0 0 443988 1585259 0 0 0 0 0 0 0 52 195 30 0 70 0 1.00 50.2 19:19:37
1 0 0 0 443988 1585259 0 0 0 0 0 0 1 82 186 30 0 70 0 1.00 50.2 19:19:38
1 0 0 0 443988 1585259 0 0 0 0 0 0 0 60 183 30 0 70 0 1.00 50.2 19:19:39
1 0 0 0 443988 1585259 0 0 0 0 0 0 0 55 184 30 0 70 0 1.00 50.2 19:19:40
So now I add another spinning thread and we go back to the expected 59% user and 41% idle. Why? Well both cores are running a single thread on the primary SMT, so we get the expected 59% user and 41% idle as per p8. As the LPAR is now running above entitlement, the user, system, wait and idle percentages are now in relation to the PC column.
2 0 0 0 444053 1585188 0 0 0 0 0 0 2 70 170 59 0 41 0 2.00 100.2 19:20:14
2 0 0 0 444053 1585188 0 0 0 0 0 0 0 46 165 59 0 41 0 2.00 100.1 19:20:15
2 0 0 0 444053 1585188 0 0 0 0 0 0 0 128 178 59 0 41 0 2.00 100.2 19:20:16
2 0 0 0 444053 1585188 0 0 0 0 0 0 2 47 169 59 0 41 0 2.00 100.1 19:20:17
Adding another thread does not change the percentage values for user and idle, as the new core is running only 1 SMT thread.
3 0 0 0 436105 1593130 0 0 0 0 0 0 5 46 165 59 0 41 0 3.00 150.0 19:20:39
3 0 0 0 436105 1593130 0 0 0 0 0 0 9 56 167 59 0 41 0 3.00 150.0 19:20:40
3 0 0 0 436105 1593130 0 0 0 0 0 0 1 58 201 59 0 41 0 3.02 151.0 19:20:41
3 0 0 0 436105 1593130 0 0 0 0 0 0 2 47 167 60 0 40 0 2.98 149.1 19:20:42