Linux iostat command

The iotop is a free and useful Linux top command like monitoring tool meant to monitor disk I/O activity on the server. You must have a kernel version of 2.6.20 or later to use iotop. It requires root privileges to execute. We could also use sar command or iostat to monitor I/O activity on the server, but iotop is different from these tools because it displays columns for the I/O bandwidth utilized by each process/thread during the sampling period. It also shows the percentage of time the thread/process spent while swapping in and while waiting on I/O. Having I/O utilization information based on process IDs instead of disks makes identification of processes causing I/O bottlenecks straightforward.

The iotop is not available by default on most systems. So, let’s see the installation process on Redhat/Centos and Ubuntu/Debian based systems.

Installing iotop on Debian/Ubuntu systems

root@linuxnix:~# apt-get install iotop
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
iotop
0 upgraded, 1 newly installed, 0 to remove and 12 not upgraded.
Need to get 23.8 kB of archives.
After this operation, 127 kB of additional disk space will be used.
Get:1 http://ap-southeast-1.ec2.archive.ubuntu.com/ubuntu/ trusty/universe iotop amd64 0.6-1 [23.8 kB]
Fetched 23.8 kB in 0s (36.7 kB/s)
Selecting previously unselected package iotop.
(Reading database ... 118631 files and directories currently installed.)
Preparing to unpack .../archives/iotop_0.6-1_amd64.deb ...
Unpacking iotop (0.6-1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up iotop (0.6-1) ...

We could query information about the package via dpkg command as shown below:

root@linuxnix:~# dpkg -p iotop
Package: iotop
Priority: optional
Section: admin
Installed-Size: 124
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Version: 0.6-1
Depends: python (>= 2.7.1-0ubuntu2), python (<< 2.8)
Size: 23828
Description: simple top-like I/O monitor
iotop does for I/O usage what top(1) does for CPU usage. It watches I/O
usage information output by the Linux kernel and displays a table of
current I/O usage by processes on the system. It is handy for answering
the question "Why is the disk churning so much?".
iotop can only run under a Linux 2.6.20 or later kernel built with the
CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING and
CONFIG_VM_EVENT_COUNTERS build config options on.
Original-Maintainer: Paul Wise <pabs@debian.org>
Homepage: http://guichaz.free.fr/iotop/

 

Installing iotop on Redhat/Centos systems

[root@linuxnix ~]# yum install iotop -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.s.uw.edu
* epel: mirror.nes.co.id
* extras: mirrors.usc.edu
* nux-dextop: li.nux.ro
* updates: centos.unixheads.org
Resolving Dependencies
--> Running transaction check
---> Package iotop.noarch 0:0.6-2.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===========================================================================================================================================================================================
Package Arch Version Repository Size
===========================================================================================================================================================================================
Installing:
iotop noarch 0.6-2.el7 base 52 k
Transaction Summary
================================================================================
Install 1 Package
Total download size: 52 k
Installed size: 156 k
Downloading packages:
iotop-0.6-2.el7.noarch.rpm | 52 kB 00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : iotop-0.6-2.el7.noarch 1/1
Verifying : iotop-0.6-2.el7.noarch 1/1
Installed:
iotop.noarch 0:0.6-2.el7
Complete!

Let’s query the package via the following rpm command

[sahil@linuxnix:~] $ rpm -qa | grep iotop
iotop-0.3.2-7.el6.noarch
[ssuri@pbox:~] $ rpm -qi iotop
Name        : iotop                       
Relocations: (not relocatable)
Version     : 0.3.2                             
Vendor: Red Hat, Inc.
Release     : 7.el6                         
Build Date: Wed 18 Sep 2013 08:02:59 AM UTC
Install Date: Fri 12 Dec 2014 01:51:26 PM UTC      
Build Host: x86-022.build.eng.bos.redhat.com
Group       : Applications/System          
Source RPM: iotop-0.3.2-7.el6.src.rpm
Size        : 146237                           
License: GPLv2
Signature   : RSA/8, Thu 19 Sep 2013 08:09:14 AM UTC, Key ID 199e2f91fd431d51
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
URL         : http://guichaz.free.fr/iotop/
Summary     : Top like utility for I/O
Description :
Linux has always been able to show how much I/O was going on
(the bi and bo columns of the vmstat 1 command).
iotop is a Python program with a top like UI used to
show of behalf of which process is the I/O going on.

 

Now that we’ve installed iotop and also verified the install we’ll now focus on its usage.

Example1: Using iotop without any options will display all processes running on the system including those not performing any I/O operations. To restrict the output to only those processes which are currently performing I/O, use the -o option.

[sahil@linuxnix:~] $ sudo iotop -o
Total DISK READ: 0.00 B/s | Total DISK WRITE: 54.43 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 1074 be/3 root        0.00 B/s   27.21 K/s  0.00 %  0.10 % [jbd2/dm-5-8]
54963 be/4 root        0.00 B/s    6.80 M/s  0.00 %  0.00 % rsync --server -vlogDtprxze.iLsfxC . /var/www/html/pub/linux_build/
16153 be/4 root        0.00 B/s    3.89 K/s  0.00 %  0.00 % java -Dgosh.args=-sc telnetd -p6666 start -Xms512m -Xmx1536m -X~anager/bundle-cache/ -b /opt/asrmanager/felix-framework/bundle/

 

So, the above iotop output only shows the processes engaged in performing any I/O activities at the instant when iotop reported I/O bandwidth utilization on the system.

I’ll briefly describe the different fields reported on by iotop:

  • TID: PID of process
  • PRIO: the I/O priority at which the process is running. This can be modified using the ionice command.
  • USER: The user id which owns the process
  • DISK READ: data read per second
  • DISK WRITE: data written per second
  • SWAPIN: Time spent swapping, i.e., copying of memory pages back and forth between memory and swap.
  • IO: Time spent by the process waiting for I/O during the sampling interval. While debugging I/O performance issues, take note that a high value of IO column will result in a performance bottleneck.
  • COMMAND: The command being executed by the process.

We can filter out the output further to limit it to a user with the -u option or even a process with the -p option which I’ll demonstrate in the next few examples.

Example2: Here is an example to filter out all processes owned by grid user which are presently performing an I/O operation.

[sahil@linuxnix:~] $ sudo iotop -o -u grid
Total DISK READ: 107.11 K/s | Total DISK WRITE: 95.00 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
15883 rt/4 grid      953.70 B/s    0.00 B/s  0.00 %  0.04 % ocssd.bin
15875 rt/4 grid      953.70 B/s    0.00 B/s  0.00 %  0.03 % ocssd.bin
15879 rt/4 grid      953.70 B/s    0.00 B/s  0.00 %  0.02 % ocssd.bin
15877 rt/4 grid        0.00 B/s  476.85 B/s  0.00 %  0.02 % ocssd.bin
15872 rt/4 grid        0.00 B/s  476.85 B/s  0.00 %  0.02 % ocssd.bin
15881 rt/4 grid        0.00 B/s  476.85 B/s  0.00 %  0.01 % ocssd.bin
87309 be/4 grid        0.00 B/s    3.73 K/s  0.00 %  0.00 % gipcd.bin
22947 be/4 grid        0.00 B/s    7.45 K/s  0.00 %  0.00 % tnslsnr LISTENER -inherit

 

Example 3: In this case, only the I/O operations being performed by process id 15883 are reported.

[sahil@linuxnix:~] $ sudo iotop -o -p 15883
Total DISK READ: 2.99 K/s | Total DISK WRITE: 148.73 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
15883 rt/4 grid     1022.16 B/s    0.00 B/s  0.00 %  0.23 % ocssd.bin

 

Example 4: Like top command, we can also use iotop in batch mode so that we could make use of the displayed output in scripts or reporting purposes. We use the -b option with iotop to indicate batch mode. The -b option is usually accompanied by -n option which shows the number of iterations iotop should run before stopping.

So, if I needed to check I/O activity performed on the server only once i.e., for just one sample interval, I could type the following:

[sahil@linuxnix:~] $ sudo iotop -b -n 1 -o
Total DISK READ: 429.62 K/s | Total DISK WRITE: 211.50 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
55044 be/4 obryann1  423.01 K/s  211.50 K/s  0.00 %  3.01 % ora_ckpt_bryann12
15875 rt/4 grid        6.61 K/s    0.00 B/s  0.00 %  1.43 % ocssd.bin

I hope this article has helped you to understand how we may use iotop instead of sar or iostat when we need to identify an I/O bottleneck based on a process instead of a disk device.

Post Views: 19,497
The following two tabs change content below.

Sahil Suri

He started his career in IT in 2011 as a system administrator. He has since worked with HP-UX, Solaris and Linux operating systems along with exposure to high availability and virtualization solutions. He has a keen interest in shell, Python and Perl scripting and is learning the ropes on AWS cloud, DevOps tools, and methodologies. He enjoys sharing the knowledge he's gained over the years with the rest of the community.