Monitoring Disk I/O Performance with iostat on Linux

Disk I/O bottlenecks can severely impact application performance. When my trading server experienced slow database queries, iostat revealed the disk was at 100% utilization. This guide covers using iostat to diagnose and monitor disk performance.

Environment

• OS: Ubuntu 22.04 LTS
• Disk: NVMe SSD (500GB)
• Tool: sysstat 12.5.1

$ iostat --version
sysstat version 12.5.1
(C) Sebastien Godard (sysstat  orange.fr)

The Problem

# Slow database queries
$ time psql -c "SELECT count(*) FROM trades;"
 count
-------
 5000000
(1 row)

Time: 45234.125 ms (00:45.234)

# Check system load
$ uptime
 15:30:00 up 30 days,  2:30,  1 user,  load average: 4.50, 3.80, 2.90

# Check iowait
$ top
%Cpu(s):  5.2 us,  2.1 sy,  0.0 ni, 45.3 id, 46.1 wa,  0.8 hi,  0.5 si
# 46.1% iowait - disk is the bottleneck!

Analysis

Step 1: Basic iostat Usage

# Show CPU and device statistics
$ iostat
Linux 5.15.0-84-generic (server)   05/12/2025   _x86_64_    (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.34    0.00    3.45   46.12    0.00   38.09

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
nvme0n1          1234.50      567.89      1234.56   12345678   98765432

Step 2: Detailed Statistics

# Show extended device statistics
$ iostat -dx
Device             tps     kB/s    rrqm/s   wrqm/s     r/s     w/s   rMB/s   wMB/s await r_await w_await svctm  %util
nvme0n1        1234.50  1802.45     12.34    45.67   890.00  344.50   1.23    1.56  12.34    8.90   21.00   0.81 100.00

Step 3: Monitor in Real Time

# Update every 2 seconds
$ iostat -dx 2
Linux 5.15.0-84-generic (server)   05/12/2025   _x86_64_    (4 CPU)

Device             tps     kB/s    rrqm/s   wrqm/s     r/s     w/s   rMB/s   wMB/s await r_await w_await svctm  %util
nvme0n1        1234.50  1802.45     12.34    45.67   890.00  344.50   1.23    1.56  12.34    8.90   21.00   0.81 100.00

Device             tps     kB/s    rrqm/s   wrqm/s     r/s     w/s   rMB/s   wMB/s await r_await w_await svctm  %util
nvme0n1         456.78   567.89      5.67    23.45   234.00  222.78   0.45    0.78  45.67   32.10   60.00   2.17 100.00

Understanding iostat Metrics

Key Metrics Explained

Interpreting Output

# Example: High %util with high await = disk bottleneck
Device             tps     kB/s  await  %util
nvme0n1        1234.50  1802.45  12.34  100.00  # BAD - disk saturated

# Example: Low %util with low await = healthy disk
nvme0n1         234.56   567.89   2.34   15.00  # GOOD - disk healthy

Solution: Optimize Disk I/O

Step 1: Identify I/O Heavy Processes

# Find processes using most I/O
$ sudo iotop -o
Total DISK READ:  123.45 M/s | Total DISK WRITE:  456.78 M/s
Current DISK READ:  123.45 M/s | Current DISK WRITE:  456.78 M/s
  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
12345 be/4  postgres   45.67 M/s  234.56 M/s  0.00 %  85.23 %  postgres: writer process
 6789 be/4  mysql      23.45 M/s  123.45 M/s  0.00 %  45.67 %  mysqld

Step 2: Optimize I/O Scheduler

# Check current scheduler
$ cat /sys/block/nvme0n1/queue/scheduler
[none] mq-deadline kyber bfq

# For NVMe SSDs, 'none' or 'mq-deadline' is best
$ echo mq-deadline | sudo tee /sys/block/nvme0n1/queue/scheduler

# Make persistent
$ echo 'ACTION=="add|change", KERNEL=="nvme*", ATTR{queue/scheduler}="mq-deadline"' | \
    sudo tee /etc/udev/rules.d/60-scheduler.rules

Step 3: Increase I/O Limits

# Check current limits
$ cat /proc/sys/fs/aio-max-nr
65536

# Increase if needed
$ sudo sysctl -w fs.aio-max-nr=131072

Step 4: Add Monitoring Script

#!/bin/bash
# /usr/local/bin/disk-io-monitor.sh

LOGFILE="/var/log/disk-io-$(date +%Y%m%d).csv"
INTERVAL=60

echo "timestamp,device,tps,kb_read_s,kb_wrtn_s,await,pct_util" > "$LOGFILE"

while true; do
    TIMESTAMP=$(date +%Y-%m-%d_%H:%M:%S)
    iostat -dx 1 2 | tail -n +4 | head -1 | while read line; do
        DEVICE=$(echo $line | awk '{print $1}')
        TPS=$(echo $line | awk '{print $2}')
        KB_READ=$(echo $line | awk '{print $3}')
        KB_WRTN=$(echo $line | awk '{print $4}')
        AWAIT=$(echo $line | awk '{print $10}')
        UTIL=$(echo $line | awk '{print $14}')
        
        echo "${TIMESTAMP},${DEVICE},${TPS},${KB_READ},${KB_WRTN},${AWAIT},${UTIL}" >> "$LOGFILE"
    done
    sleep $INTERVAL
done

Quick iostat Commands

# Basic stats
$ iostat

# Extended device stats
$ iostat -dx

# Show only devices with activity
$ iostat -dx -p ALL

# Specific device
$ iostat -dx nvme0n1

# Show memory stats too
$ iostat -dxm 2 5  # 5 samples, 2 second interval

# Show in MB/s
$ iostat -dxk 2

Lessons Learned

1. %util >80% indicates disk saturation - investigate I/O-heavy processes.
2. await >20ms for SSDs suggests the disk cannot keep up with requests.
3. Use iotop alongside iostat to identify which processes cause I/O.
4. NVMe SSDs perform best with none or mq-deadline scheduler.
5. Monitor continuously during peak hours to catch performance degradation.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.