iotop: Per Process I/O Usage

Based on a reader comment, we take iotop for a spin to see if it can be used for monitoring the IO usage of individual processes on a system. The result? It has some interesting capability that we haven't found in other tools.


If you have been reading some of my articles you know my frustration at the lack of good storage management and monitoring tools for Linux. Many of the tools provide bits and pieces of the information storage administrators need but they are disconnected and collectively don’t provide the information required. So while we (or someone else) is developing appropriate tools for managing and monitoring storage, we have to use what tools are available.

So I started examining various storage monitoring tools, starting with iostat which is a part of the sysstat tool set. Iostat can be used to watch the overall IO statistics of a particular or all block device(s) along with CPU usage statistics. This information can be useful since it provides something of an overall view of the IO usage but it can’t tell you which process or processes are creating (or abusing) the IO performance.

In the comments section of the iostat article a user named mossholderm pointed to a tool called iotop that can help monitor the IO usage of the processes on a system. So I decided to take a look.

Installing iotop

The first step in using any new tool is installing it. Sometimes there are pre-built binaries and sometimes you have to install from source. Iotop is written in Python (a great language IMHO), so I decided to install it from source (plus I could not find an rpm for the latest version that matched my distribution but I didn’t look too hard to be honest). The last update to iotop was in 2007 so I was a little worried about being able to build it correctly. But before I could build it and run it there are some prerequisites that I needed to address.

These prerequisites for iotop were a little beyond the usual, “find this package and install it” approach. For iotop to work correctly, it requires certain kernel versions and options to be configured. In particular, it states the following prerequisites are needed:

  • Python >= 2.5 -or- Python 2.4 + ctypes
  • 2.6.20 kernel (or greater) with IO accounting and VM event counters configured

The system I was using is a CentOS 5.4 system with an older 2.6.18 kernel. Plus it came with Python 2.4.3 so I needed to add ctypes to my system as well as find a kernel that either had the IO accounting patches back ported or was a newer kernel than 2.6.20.

Installing ctypes was very easy. From the website, ctypes adds the following functionality:

ctypes allows to call functions in dlls/shared libraries and has extensive facilities to create, access and manipulate simple and complicated C data types in Python – in other words: wrap libraries in pure Python. It is even possible to implement C callback functions in pure Python.

Installation is very simple with two steps. The first step is,

# python setup.py build

And the second step is,

# python setup.py install

The next step in satisfying the prerequisites for iotop was to sort out the kernel. I initially tried using the stock CentOS kernel but it kept telling me it didn’t have IO accounting or VM event counters defined. So I chose to build my own kernel. I used 2.6.32 since I was building it for other reasons. The IO accounting kernel options is on the first very option as show below in Figure 1.

Figure 1: 2.6.32 Kernel Configuration Screenshot

Look right in the middle of the screen on the right hand side and you will see the options for enabling “per-task” storage I/O accounting”. Then all you do is build and install the kernel (for details on this process, just use Google – there are lots of articles describing how to build and install your own kernel).

The next step is very easy – installing iotop. After uncompressing and untarring the iotop binary, the command is simply the following.

# ./setup.py install

That’s pretty much it – iotop is installed and ready to go.

Running iotop

Once iotop was installed the next step is to actually run it. Figure 2 below is the screenshot when I first started iotop without running any I/O intensive applications.

Figure 2: Initial iotop Screenshot

Notice the iotop looks sort of like “top”.

At the very top of the output you see the sum of the “DISK READ” and “DISK WRITE” bandwidth in B/s (Bytes per second). After this is a list of all processes running on the system. Each process has a column labeled “DISK READ” and “DISK WRITE” as well as “SWAPIN” and “IO”. Finally the “COMMAND” column is the name of the process. By default iotop monitors all users on the system and all processes even if they really don’t do any IO. However, you can toggle iotop to show just the applications doing real I/O using “O” (capital O). My test system wasn’t doing any I/O so toggling the system from Figure 2 shows nothing so I’m not showing it. However, I can do something more interesting by running IOzone on the system and then using iotop.

So I started running IOzone on an Intel X-25E SSD and then started iotop. Figure 3 below is iotop showing only the I/O intensive processes when running IOzone.

Figure 3: Iotop Screenshot when running IOZone

The filesystem is ext4 if you’re interested. Notice at the top of the output from iotop that the total disk throughput is listed as 216.73 MB/s. You can see the individual IOzone process below the top (PID 4307).

To show you how many processes there are on a system and how many are really doing IO, compare Figure 3 to Figure 2. You can see that in this case there were few processes actually doing any IO.

Just to be sure that iotop was actually doing what it said it was, I checked that iotop is actually measuring I/O by looking at gkrellm. A screen capture of the output is shown in Figure 4.

Figure 4: Gkrellm Screenshot when running IOZone

There are other options for iotop. The man pages are reasonable enough to figure out what the options do for you.


I’m a bit partial to Python because I’ve been doing a great deal of coding with it lately (probably not very Pythonic coding though). So when user mossholderm wrote about iotop in a comment about the iostat article, I had to try it out. It’s is a simple tool for watching the I/O usage on a per process basis on a system. The interface is something like a “top” interface which is something we all use in Linux so it’s pretty easy to understand the output.

I’m still on my quest for a good management/monitoring tool for Linux storage. Iostat has some aspects I like, particularly the overall I/O view it provides as well as the ability to target individual devices within a system along with information on CPU usage and I/O scheduler requests. Iotop allows us to see the I/O usage of individual processes. Nfsiostat allows us to see NFS requests on NFS clients. Now we need to combine these tools into one while adding in some other bits and pieces. Anyone game?

Comments on "iotop: Per Process I/O Usage"

Perfect answer! That really gets to the heart of it! You nailed it!

Such an impressive answer! You’ve beaten us all with that! Good used car purchasing tips!

?i to every body, it’s my fir?t go to see of this webpage; this
blog includes ?mazing and in fact excellent material in favor of r?a?ers.

At last, someone who comes to the heart of it all

Well I guess I don’t have to spend the weekend figuring this one out! Buying a used car indeed is a tedious process.

I’m extremely impressed with your writing skills and also with the layout on your blog. Is this a paid theme or did you customize it yourself? Either way keep up the nice quality writing, it is rare to see a great blog like this one these days..

I’ve read several good stuff here. Definitely worth bookmarking for revisiting. I surprise how a lot attempt you put to make one of these great informative website.

Nice blog here! Also your site loads up very fast! What web host are you using? Can I get your affiliate link to your host? I wish my site loaded up as quickly as yours lol|

Good article and straight to the point. I don’t know if this is really the best place to ask but do you people have any ideea where to get some professional writers? Thanks ????

I would like to thnkx for the efforts you’ve put in writing this site. I’m hoping the same high-grade site post from you in the upcoming as well. In fact your creative writing abilities has inspired me to get my own blog now. Really the blogging is spreading its wings rapidly. Your write up is a great example of it.

I love reading through an article that will make men and women think. Also, thank you for permitting me to comment!|

I have not checked in here for a while as I thought it was getting boring, but the last several posts are good quality so I guess I¡¦ll add you back to my daily bloglist. You deserve it my friend ????

I¡¦ve learn a few excellent stuff here. Certainly value bookmarking for revisiting. I wonder how a lot attempt you put to make this kind of fantastic informative website.

Hey There. I found your blog using msn. This is a really well written article. I’ll make sure to bookmark it and come back to read more of your useful information. Thanks for the post. I’ll definitely return.

Hey very cool web site!! Man .. Beautiful .. Wonderful .. I will bookmark your blog and take the feeds additionally…I am satisfied to find numerous useful information right here in the put up, we want work out extra techniques in this regard, thanks for sharing.

Great ¡V I should definitely pronounce, impressed with your website. I had no trouble navigating through all tabs as well as related info ended up being truly easy to do to access. I recently found what I hoped for before you know it in the least. Quite unusual. Is likely to appreciate it for those who add forums or anything, website theme . a tones way for your client to communicate. Nice task..

“Hello There. I found your blog using msn. This is an extremely well written article. I’ll be sure to bookmark it and come back to read more of your useful info. Thanks for the post. I will definitely return.”

Outstanding quest there. What occurred after? Take care!

My grand father all the time used to watch YouTube comical videos, hehehehehe, because he needs to be happy always.

What i don’t understood is if truth be told how you are now not really a lot more neatly-favored than you may be right now. You are very intelligent. You know thus considerably relating to this subject, made me individually consider it from so many varied angles. Its like women and men are not fascinated unless it is one thing to accomplish with Woman gaga! Your personal stuffs great. All the time maintain it up!

Leave a Reply