SAR file collection and historical reporting data:

In a previous article, we described how sar reports data on various system performance metrics in real time. In this article, we talk about sar data is collected, stored and retrieved.

When we install sysstat, it adds the following additional utilities which are responsible for collecting and storing sar data:

  • sadc: This tool collects sar data to make it available for retrieval at a later date.
  • sa1: It saves system activities in a binary data file. sa1 depends on sadc to provide the data.
  • sa2: It creates a daily summary of the collected statistics.
  • sadf: Allows generation of sar report in CSV, XML, and other formats.

Both sa1 and sa2 utilities run as cron jobs. The default sysstat cron file is /etc/cron.d/sysstat and is as follows:

sudo cat /etc/cron.d/sysstat
# Run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# Generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

The file is root readable and thus requires superuser privileges to access.

ls -l /etc/cron.d/sysstat
-rw------- 1 root root 235 Mar  8  2016 /etc/cron.d/sysstat

Let’s understand the two cron jobs mentioned above:

The first sysstat cron job: /usr/local/lib/sa/sa1

  • It runs every 10 min and collects sar data for historical reference.
  • This writes the data to /var/log/sa/saXX file. XX is the day of the month.
  • sa1 creates binary files so we can’t open them via a text editor.
  • It accepts two parameters to sa1: interval (in seconds) and counts.
  • In the above crontab example: sa1 1 1 means that sa1 will collect data once with an interval of 1 second every ten mins.

The second sysstat cron job: /usr/local/lib/sa/sa2

  • It runs close to midnight (at 23:53) to create the daily summary report of sar data.
  • sa2 creates /var/log/sa/sarXX file.
  • It creates ASCII text files and therefore can be opened with a text editor or displayed to stdout.
  • This file contains summary information on all metrics whose statistics are captured by sar.

Sysstat saves the files generated by sa1 and sa2 for 28 days by default, but this can be changed by modifying the HISTORY variable in the /etc/sysconfig/sysstat file.

grep HISTORY /etc/sysconfig/sysstat
HISTORY=28

Reading historical SAR data from saXX files

Example10: To read data from a saXX file we use the following syntax:

sar [metric flag] -f /var/log/sa/saXX

The -f flag denotes that we’ll be reading input from a file followed by the file name. The metric flag will specify the metric whose information we’d like to access. For example, the command “sar -u -f /var/log/sa12” will display the recorded CPU utilization information for the 12th day of the month as shown below:

sahil@linuxnix:~] $ sudo sar -u -f /var/log/sa/sa12 | head
Linux 2.6.32-642.13.1.el6.x86_64 (linuxnix.test.org)    10/12/2017      _x86_64_        (2 CPU)
12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM     all      1.63      0.00      1.17      0.07      0.00     97.12
12:20:01 AM     all      1.61      0.00      1.14      0.06      0.00     97.19
12:30:01 AM     all      1.57      0.00      1.11      0.06      0.00     97.26
12:40:01 AM     all      1.64      0.00      1.11      0.06      0.00     97.19
12:50:01 AM     all      1.65      0.00      1.14      0.06      0.00     97.15
01:00:01 AM     all      1.59      0.00      1.12      0.06      0.00     97.24
01:10:01 AM     all      1.66      0.00      1.15      0.05      0.00     97.14

 

Example11: If we need to monitor the system for a particular time window, then we could specify an output file and conveniently run sar in the background.

Here is an example:

sudo sar -o /tmp/SAR_datafile 60 5 >/dev/null 2>&1 &
[1] 16587

This command will run sar every 1 minute 5 times and write the output to /tmp/SAR_datafile file. The file /tmp/SAR_datafile is a binary file, and we’ll need to use sar -f to open it.

Example 12: In the following example, I’ll be extracting the CPU utilisation data from the data file /tmp/SAR_datafile

[sahil@linuxnix:~] $ sar -u -f /tmp/SAR_datafile
Linux 2.6.32-642.13.1.el6.x86_64 (linuxnix.test.org)    10/13/2017      _x86_64_        (2 CPU)
04:19:35 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:20:35 AM     all      1.65      0.00      1.30      0.05      0.00     96.99
04:21:35 AM     all      1.49      0.00      1.02      0.05      0.00     97.44
04:22:35 AM     all      1.49      0.00      0.98      0.05      0.00     97.49
04:23:35 AM     all      1.52      0.00      1.06      0.06      0.00     97.36
04:24:35 AM     all      1.51      0.00      1.09      0.06      0.00     97.34
Average:        all      1.53      0.00      1.09      0.05      0.00     97.32
[sahil@linuxnix:~] $ file /tmp/SAR_datafile
/tmp/SAR_datafile: data

Formatting sar data using sadf:

We can use the sadf command to read data files generated by sar and display them in different formats depending on our requirements.

Before getting to examples I’ll describe the various flags I’ll be using:

Note: All the activity flags of sar may be entered on the command line to indicate activities to report. Before specifying them, put a pair of dashes (–) on the command line in order not to confuse the flags with those of sadf.

 

-d: Print the contents of the data file in a format that can easily be ingested by a relational database system. The output consists of fields separated by a semicolon.

-e [ hh:mm:ss ]: Set the ending time of the report, given in local time.

-s [ hh:mm:ss ]: Set the starting time of the data (presented in local time), causing the sadf command to extract records time-tagged at, or following, the time specified.

-t: When this option is used  together  with options -d or -x, the timestamp is displayed in local time instead of UTC

-x: Print the contents of the data file in XML format.

 

Given below are a couple of examples:

 

Example 13: This extracts CPU utilization data from data file /var/log/sa/sa21 within the time window 01:30:01 to 02:30:01

[sahil@linuxnix:~] $ sadf -t -d /var/log/sa/sa21 -- -u -s 01:30:01 -e 02:30:01
# hostname;interval;timestamp;CPU;%user;%nice;%system;%iowait;%steal;%idle
linuxnix.test.org;599;2017-09-21 01:40:01;-1;1.58;0.00;1.05;0.15;0.00;97.22
linuxnix.test.org;599;2017-09-21 01:50:01;-1;1.53;0.00;1.03;0.05;0.00;97.39
linuxnix.test.org;599;2017-09-21 02:00:01;-1;1.53;0.00;1.12;0.05;0.00;97.30
linuxnix.test.org;598;2017-09-21 02:10:01;-1;12.83;0.00;2.22;0.31;0.00;84.63
linuxnix.test.org;598;2017-09-21 02:20:01;-1;11.67;0.00;2.69;0.27;0.00;85.37
linuxnix.test.org;599;2017-09-21 02:30:01;-1;1.50;0.00;1.07;0.04;0.00;97.38

 

Example 14: This does the same as above example but instead of data for all CPUs we limiting the output to the first core only.

[sahil@linuxnix:~] $ sadf -t -d /var/log/sa/sa17 -- -P 1 -s 01:30:01 -e 02:30:01
# hostname;interval;timestamp;CPU;%user;%nice;%system;%iowait;%steal;%idle
linuxnix.test.org;599;2017-09-17 01:40:01;1;1.50;0.00;1.10;0.05;0.00;97.35
linuxnix.test.org;599;2017-09-17 01:50:01;1;1.46;0.00;1.16;0.05;0.00;97.33
linuxnix.test.org;599;2017-09-17 02:00:01;1;1.62;0.00;1.05;0.04;0.00;97.29
linuxnix.test.org;598;2017-09-17 02:10:01;1;8.35;0.00;1.75;0.23;0.00;89.68
linuxnix.test.org;598;2017-09-17 02:20:01;1;9.53;0.00;2.37;0.17;0.00;87.92
linuxnix.test.org;598;2017-09-17 02:30:01;1;1.48;0.00;1.04;0.08;0.00;97.40

Example 15: Here we’ve used sed to convert the semicolons to commas so that the data could be redirected to a CSV file.

[sahil@linuxnix:~] $ sadf -t -d /var/log/sa/sa17 -- -P 1 -s 01:30:01 -e 02:30:01  | sed 's/;/,/g'
# hostname,interval,timestamp,CPU,%user,%nice,%system,%iowait,%steal,%idle
linuxnix.test.org,599,2017-09-17 01:40:01,1,1.50,0.00,1.10,0.05,0.00,97.35
linuxnix.test.org,599,2017-09-17 01:50:01,1,1.46,0.00,1.16,0.05,0.00,97.33
linuxnix.test.org,599,2017-09-17 02:00:01,1,1.62,0.00,1.05,0.04,0.00,97.29
linuxnix.test.org,598,2017-09-17 02:10:01,1,8.35,0.00,1.75,0.23,0.00,89.68
linuxnix.test.org,598,2017-09-17 02:20:01,1,9.53,0.00,2.37,0.17,0.00,87.92
linuxnix.test.org,598,2017-09-17 02:30:01,1,1.48,0.00,1.04,0.08,0.00,97.40

Example 16: In this example, we use sadf to extract data and report it in XML format.

[sahil@linuxnix:~] $ sadf -t -x /var/log/sa/sa17 -- -u -s 01:30:01 -e 02:30:01 | head
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Configure PUBLIC "DTD v2.5 sysstat //EN"
"http://pagesperso-orange.fr/sebastien.godard/sysstat.dtd">
<sysstat>
        <sysdata-version>2.5</sysdata-version>
        <host nodename="linuxnix.test.org">
           <sysname>Linux</sysname>
           <release>2.6.32-642.13.1.el6.x86_64</release>
           <machine>x86_64</machine>
           <number-of-cpus>2</number-of-cpus>
-------------------------------------------------------------------------------------
-----------------------------------output truncated for brevity

That’s for this post, in our next SAR post we will see how to use ksar command in detail.

Post Views: 26,633
The following two tabs change content below.

Sahil Suri

He started his career in IT in 2011 as a system administrator. He has since worked with HP-UX, Solaris and Linux operating systems along with exposure to high availability and virtualization solutions. He has a keen interest in shell, Python and Perl scripting and is learning the ropes on AWS cloud, DevOps tools, and methodologies. He enjoys sharing the knowledge he's gained over the years with the rest of the community.