Measuring CPU performance with SAR command:

We will continue our journey with our SAR command. We already covered on SAR command in our first post on how to install it. In this post we will see how to use SAR command to monitor CPU, RAM(Memory), disk and IO stats.

Example3: To report on CPU performance in sar command, we use the -u flag. Here is an example:

[sahil@linuxnix.com:~] $ sar -u 1 2
Linux 2.6.32-642.13.1.el6.x86_64 (linuxnix.com)    10/12/2017      _x86_64_        (2 CPU)
09:13:11 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
09:13:12 AM     all      0.00      0.00      1.01      0.00      0.00     98.99
09:13:13 AM     all      0.50      0.00      0.50      0.00      0.00     98.99
Average:        all      0.25      0.00      0.75      0.00      0.00     98.99

Here we can see that sar is reporting CPU utilization twice at an interval of one second and we have an average line in the end. The time is as per the systems’ time zone and displays it in real time, i.e., SAR begins reporting at 09:13:11 AM and ends at 09:13:13 AM which was the actual time on the server when the command finished.

Exploring the meaning of each SAR field reported

%user: Percentage of CPU utilization that occurred while executing at the user level which includes application processes, user running jobs, etc.

%nice: Percentage of CPU utilization that occurred while executing at the user level with nice priority. I won’t go into depth regarding nice values at this point but to briefly explain we can use them to modify the priority at which a process would run on the system.

%system: Percentage of CPU utilization that occurred while executing at the kernel level. This constitutes the memory used by the OS kernel itself to maintain optimum system function.

%iowait: Percentage of time that the CPUs were idle during which the system had an outstanding disk I/O request. An example of high I/O wait is when a backup job is running, but there is no tape drive loaded to take the backup resulting in the job getting stuck.

%steal: Percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor. This category of CPU utilization would occur in case of virtual machines only.

%idle: Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

Using sar with the -u flag will report CPU utilization statistics for all processor cores present in the system.

Example 4: Reporting processor utilization

If we are interested in getting resource utilization data for a processor core we could use sar with the -P flag as demonstrated below:

[sahil@linuxnix.com:~] $ sar -P 1 1 2
Linux 2.6.32-642.13.1.el6.x86_64 (linuxnix.com)    10/13/2017      _x86_64_        (2 CPU)
02:06:18 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
02:06:19 AM       1     23.23      0.00      8.08      1.01      0.00     67.68
02:06:20 AM       1     47.00      0.00      3.00      0.00      0.00     50.00
Average:          1     35.18      0.00      5.53      0.50      0.00     58.79
[sahil@linuxnix.com:~] $
[sahil@linuxnix.com:~] $ sar -P 2 1 2
Linux 2.6.32-642.13.1.el6.x86_64 (linuxnix.com)    10/13/2017      _x86_64_        (2 CPU)
02:06:25 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
[sahil@linuxnix.com:~] $
[sahil@linuxnix.com:~] $ date
Fri Oct 13 02:06:32 UTC 2017

The command “sar -P 1 1 2” reports utilization of processor core #1 twice at an interval of 1 second. We notice that we get an empty output when we run the same command for processor core #2 and this implies that this core is not in use or completely idle now.

As another example, here are the utilization statistics for cores #2 and #3 on a 4-core system.

[sahil@linuxnix.com2:~] $ sar -P 2 1 2
Linux 2.6.32-696.6.3.el6.x86_64 (linuxnix.com2.dev.test.org)         10/13/2017      _x86_64_        (4 CPU)
02:12:41 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
02:12:42 AM       2      9.80      0.00      1.96      0.00      0.00     88.24
02:12:43 AM       2      1.96      0.00      0.98      0.00      0.00     97.06
Average:          2      5.88      0.00      1.47      0.00      0.00     92.65
[sahil@linuxnix.com2:~] $
[sahil@linuxnix.com2:~] $ sar -P 3 1 2
Linux 2.6.32-696.6.3.el6.x86_64 (linuxnix.com2.dev.test.org)         10/13/2017      _x86_64_        (4 CPU)
02:14:17 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
02:14:18 AM       3     66.34      0.00     30.69      0.00      0.00      2.97
02:14:19 AM       3     69.70      0.00     29.29      0.00      0.00      1.01
Average:          3     68.00      0.00     30.00      0.00      0.00      2.00

Measuring memory utilization with SAR:

Example5: To report memory utilization, we use the -r flag with sar. Here is an example:

[sahil@linuxnix.com:~] $ sar -r 1 2
Linux 2.6.32-642.13.1.el6.x86_64 (linuxnix.com)    10/13/2017      _x86_64_        (2 CPU)
02:20:06 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
02:20:07 AM    356312   3168012     89.89    471872   1315312   2149304     27.85
02:20:08 AM    356312   3168012     89.89    471876   1315312   2149304     27.85
Average:       356312   3168012     89.89    471874   1315312   2149304     27.85

I’ll briefly describe the fields reported in the above sar command output as follows:

kbmemfree: Amount of free memory available in kilobytes.

kbmemused: Amount of used memory in kilobytes. This does not consider memory used by the kernel itself.

%memused: Percentage of used memory.

kbbuffers: Amount of memory used as buffers by the kernel in kilobytes.

kbcached: Amount of memory used to cache data by the kernel in kilobytes.

kbcommit: Amount of memory in kilobytes needed for the current workload. This is an estimate of how much RAM/swap is required to guarantee that there never is out of memory.

%commit: Percentage of memory needed for the current workload in relation to the total amount of memory (RAM+swap). This number may be higher than 100% because the kernel usually overcommits memory.

Anyone familiar with the Linux operating system would consider the output given by the free command to be more flexible and more comfortable to interpret. But the benefit of using sar is that we can access historical data when we need to troubleshoot a recent performance bottleneck. I’ll describe retrieving historical data in a later article.

Example6: To view statistics on swap memory we use the -S flag. Here’s an example:

[sahil@linuxnix.com:~] $ sar -S 1 2
Linux 2.6.32-642.13.1.el6.x86_64 (linuxnix.com)    10/13/2017      _x86_64_        (2 CPU)
02:28:37 AM kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
02:28:38 AM   3520820    673480     16.06     30732      4.56
02:28:39 AM   3520820    673480     16.06     30732      4.56
Average:      3520820    673480     16.06     30732      4.56

Now let’s understand the various fields reported in the above output.

kbswpfree: Amount of free swap space in kilobytes.

kbswpused: Amount of used swap space in kilobytes.

%swpused: Percentage of used swap space.

kbswpcad: Amount of cached swap memory in kilobytes. This is the memory that once was swapped out, is swapped back in but still also is in the swap area  (if memory is needed it doesn’t need to be swapped out again because it is already in the swap area. This saves I/O).

%swpcad: Percentage of cached swap memory in relation to the amount of used swap space.

Measuring disk and I/O performance with SAR:

Example7: To estimate disk utilization and performance we use the -d flag with sar.

[sahil@linuxnix.com:~] $ sar -d 1 2
Linux 2.6.32-642.13.1.el6.x86_64 (linuxnix.com)    10/13/2017      _x86_64_        (2 CPU)
02:34:38 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
02:34:39 AM    dev8-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM   dev8-48      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM   dev8-16      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM   dev8-32      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM   dev8-64      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM  dev253-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM  dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM  dev253-3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM  dev253-4      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM  dev253-5      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM  dev253-6      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:39 AM  dev253-7      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

02:34:39 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
02:34:40 AM    dev8-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM   dev8-48      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM   dev8-16      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM   dev8-32      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM   dev8-64      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM  dev253-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM  dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM dev253-3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM  dev253-4      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM  dev253-5      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM  dev253-6      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:34:40 AM  dev253-7      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average:       dev8-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:      dev8-48      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:      dev8-16      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:      dev8-32      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:     dev8-64      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:     dev253-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:     dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:     dev253-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:     dev253-3      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:     dev253-4      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:     dev253-5      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:     dev253-6      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:     dev253-7      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
[sahil@linuxnix.com:~] $

The device names as mentioned in the DEV column don’t look very familiar. Let’s understand this. The device name dev253-7 implies a block device with a major number of 253 and a minor number of 7. The device name dev8-64 means a block device with a major number of 8 and a minor number of 64. The major number is used to indicate the module used to access the device, and the minor number is used to distinguish between individual devices in the device hierarchy. A major number of 8 represents the SCSI module and a major number of 253 represents the device-mapper module.

Example8: To modify the output to familiar device names, we may use the -p flag before the -d flag as shown below:

[sahil@linuxnix.com:~] $ sar -p -d 1 2
Linux 2.6.32-642.13.1.el6.x86_64 (linuxnix.com)    10/13/2017      _x86_64_        (2 CPU)
02:36:28 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
02:36:29 AM       sda      2.97      0.00     55.45     18.67      0.00      1.33      1.33      0.40
02:36:29 AM       sdd      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:29 AM       sdb      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:29 AM       sdc      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:29 AM       sde      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:29 AM os_vg-root_lv      0.99      0.00      7.92      8.00      0.00      3.00      3.00      0.30
02:36:29 AM os_vg-swap_01_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:29 AM kdump_vg-kdump_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.0002:36:29 AM xfer_vg-lvol1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:29 AM os_vg-tmp_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:29 AM os_vg-var_lv      5.94      0.00     47.52      8.00      0.00      0.17      0.17      0.10
02:36:29 AM os_vg-hpds_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:29 AM os_vg-kickstart_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00     0.00

02:36:29 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
02:36:30 AM       sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM       sdd      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM       sdb      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM       sdc      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM       sde      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM os_vg-root_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM os_vg-swap_01_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM kdump_vg-kdump_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM xfer_vg-lvol1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM os_vg-tmp_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM os_vg-var_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM os_vg-hpds_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:36:30 AM os_vg-kickstart_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average:          sda      1.49      0.00     27.86     18.67      0.00      1.33      1.33      0.20
Average:          sdd      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:          sdb      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:          sdc      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:          sde      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    os_vg-root_lv      0.50      0.00      3.98      8.00      0.00      3.00      3.00      0.15
Average:    os_vg-swap_01_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    kdump_vg-kdump_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    xfer_vg-lvol1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    os_vg-tmp_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    os_vg-var_lv      2.99      0.00     23.88      8.00      0.00      0.17      0.17      0.05
Average:    os_vg-hpds_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    os_vg-kickstart_lv      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Note the output displays statistics for all physical and logical volumes detected for the interval and counts specified. Now let’s describe the meaning of the fields shown in the output:

tps: Indicate the number of transfers per second issued to the device.  Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.

rd_sec/s: Number of sectors read from the device. The size of a sector is 512 bytes.

wr_sec/s: Number of sectors written to the device. The size of a sector is 512 bytes.

avgrq-sz: The average size (in sectors) of the requests issued to the device.

avgqu-sz: The average queue length of the requests issued to the device.

await: The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in the queue and the time spent servicing them.

svctm: The average service time (in milliseconds) for I/O requests issued to the device.

%util:  Percentage of elapsed time during which I/O requests were issued to the device. Device saturation occurs when this value is close to 100%.

Example9: To report on I/O statistics, we use the -b flag with sar. Here’s an example:

[sahil@linuxnix.com:~] $ sar -b 1 2
Linux 2.6.32-642.13.1.el6.x86_64 (linuxnix.com)    10/13/2017      _x86_64_        (2 CPU)
02:34:57 AM       tps      rtps      wtps   bread/s   bwrtn/s
02:34:58 AM     30.00      0.00     30.00      0.00    320.00
02:34:59 AM      0.00      0.00      0.00      0.00      0.00
Average:        15.00      0.00     15.00      0.00    160.00

I’ll briefly describe the fields reported in the output:

tps: Total number of transfers per second that were issued to physical devices.  A transfer is an I/O request to a physical device.  Multiple logical requests can be combined into a single I/O request to the device.  A transfer is of indeterminate size.

rtps: Total number of reading requests per second issued to physical devices.

wtps: Total number of write requests per second issued to physical devices.

bread/s: Total amount of data read from the devices in blocks per second.  Blocks are equivalent to sectors with 2.4 kernels and newer and therefore have a size of 512 bytes. With older kernels, a block is of indeterminate size.

bwrtn/s: Total amount of data written to devices in blocks per second.

In the next article, we’ll explore how we use other utilities provided by sysstat in conjunction with sar to store and retrieve historical performance data.

Post Views: 35,660
The following two tabs change content below.

Sahil Suri

He started his career in IT in 2011 as a system administrator. He has since worked with HP-UX, Solaris and Linux operating systems along with exposure to high availability and virtualization solutions. He has a keen interest in shell, Python and Perl scripting and is learning the ropes on AWS cloud, DevOps tools, and methodologies. He enjoys sharing the knowledge he's gained over the years with the rest of the community.