# Beyond iostat: Storage performance analysis with blktrace

DevFeed: [Beyond iostat: Storage performance analysis with blktrace](<https://devfeed.tech/articles/beyond-iostat-storage-performance-analysis-with-blktrace-12449.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2013/07/14/io-performance.html>)

Author: Marc Brooker

Published: 2013-07-14T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog.md>), [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog-2.md>)

Topics: [IO](<https://devfeed.tech/topics/io.md>), [Linux](<https://devfeed.tech/topics/linux.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [Processes](<https://devfeed.tech/topics/processes.md>), [benchmarking](<https://devfeed.tech/topics/benchmarking.md>)

Tags: [io](<https://devfeed.tech/tags/io.md>), [linux](<https://devfeed.tech/tags/linux.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [performance](<https://devfeed.tech/tags/performance.md>), [performance-analysis](<https://devfeed.tech/tags/performance-analysis.md>), [processes](<https://devfeed.tech/tags/processes.md>), [smart](<https://devfeed.tech/tags/smart.md>), [tool](<https://devfeed.tech/tags/tool.md>)

## AI overview

This article explains how to investigate Linux storage I/O performance beyond the aggregate metrics provided by iostat. It introduces blktrace and related tools for capturing detailed disk I/O traces, then using blkparse to analyze them and identify process-level or time-sensitive performance issues.

## Source excerpt

Beyond iostat: Storage performance analysis with blktrace An under appreciated set of IO analysis tools. If you've spent much time at all investigating IO performance on Linux, you're no doubt already familiar with iostat from the venerable sysstat package. iostat is the go-to tool for Linux storage performance monitoring with good reason: it's available nearly everywhere, it works on the vast majority of Linux machines, and it's relatively easy to use and understand. Some of what it measures can be subtle, and the exact definitions of its measurements can be confusing, and even contentious, but it's still a great start. Sometimes, though, you need to go into more detail than iostat can provide. The aggregate view from iostat is simple, but makes it difficult to tell which processes are doing which IOs. Averaging over a period of time can hide subtle performance issues and the real causes of may IO-related problems. To get around these issues, you'll want to go deeper. If you have the guts for it, a recent kernel, and a good understanding of IO performance issues, you'll want to reach for blktrace and friends. The blktrace toolkit provides an extremely powerful way to look at the exact IO performance of a Linux machine, at a wide range of levels of detail, and is vastly more capable than the simple iostat. For a start, let's look at the performance of a random read workload to a magnetic drive, with 16k IOs. The manufacturer's spec sheet says this drive should be delivering about 120 IOs per second on a completely random load. iostat -x has this to say about the drive: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz sdb 0.00 0.00 124.67 0.00 3989.33 0.00 32.00 avgqu-sz await svctm %util 1.00 8.01 8.02 100.00 As expected, we're doing about 125 random IOs per second, each at 16k (32.00 512 byte sectors), at a mean service time of around 8ms. That's pretty much exactly what we would expect from a 7200 RPM magnetic drive. Nothing to see there, then. Next up, is a d