Linux Disk I/O Performance: iostat and iotop Analysis

Introduction

Disk I/O is often the bottleneck in server performance. When applications slow down, the cause is frequently excessive disk reads or writes. iostat and iotop are the primary Linux tools for diagnosing I/O performance. This post covers how to use these tools to identify bottlenecks and optimize storage performance.

Environment

The examples use Ubuntu 22.04 LTS with a mix of NVMe and SATA drives, running PostgreSQL and file-intensive workloads.

iostat --version
# iostat: sysstat version 11.7.5

iotop --version
# iotop 0.6.2

Problem

The server is experiencing slow response times:

iostat -x 1 5
# Device   r/s     w/s   rkB/s   wkB/s  rrqm/s  wrqm/s  %rrqm  %wrqm  r_await  w_await  aqu-sz  rareq-sz  wareq-sz  svctm  %util
# nvme0n1 12.00  45.00   48.00  180.00    0.00     2.00   0.00   4.26     1.20     3.50    0.18      4.00      4.00   1.50    8.55
# sda     25.00 120.00  100.00  480.00    5.00    30.00  16.67  20.00    45.00   125.00   17.50      4.00      4.00   7.14   88.00

The sda drive shows 88% utilization and high await times, indicating a bottleneck.

Analysis

Key I/O metrics:

High %util combined with high await indicates a bottleneck.

Solution

1. Basic iostat monitoring

# Show all devices
iostat -x 1 5

# Show specific device
iostat -x sda 1 5

# Show with timestamps
iostat -x -t 1 5

2. Identify the busiest processes

# Install iotop if not available
sudo apt-get install iotop

# Run as root
sudo iotop -o
# Total DISK READ: 12.34 M/s | Total DISK WRITE: 45.67 M/s
# Current DISK READ: 12.34 M/s | Current DISK WRITE: 45.67 M/s
#   PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
# 12345 be/4 postgres   8.00 M/s   32.00 M/s  0.00 %  65.00 %  postgres: writer process

3. Monitor I/O with vmstat

vmstat 1 5
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
#  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
#  5  2  10240 8192000 262144 8388608    0    0 10240 45678    0    0 45  5 50  0

The b column shows processes blocked on I/O.

4. Use iotop in batch mode for logging

# Log I/O usage every 5 seconds
sudo iotop -o -b -d 5 > /var/log/iotop.log

5. Identify specific files causing I/O

# Use lsof to find open files
sudo lsof +D /var/lib/postgresql/16/main/

# Use fatrace for real-time file access monitoring
sudo apt-get install fatrace
sudo fatrace --current-mount --sort t /var/lib/postgresql/

6. Optimize I/O scheduler

# Check current scheduler
cat /sys/block/sda/queue/scheduler
# [mq-deadline] kyber bfq none

# For NVMe drives, use none or mq-deadline
echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

# For SATA drives, use mq-deadline or bfq
echo mq-deadline | sudo tee /sys/block/sda/queue/scheduler

7. Tune read-ahead

# Check current read-ahead
sudo blockdev --getra /dev/sda
# 256

# Increase for sequential workloads
sudo blockdev --setra 4096 /dev/sda

# Decrease for random I/O workloads
sudo blockdev --setra 256 /dev/sda

8. Use ionice to control process priority

# Find a process using too much I/O
sudo iotop -o

# Lower its I/O priority
sudo ionice -c3 -p $PID

# Run a command with lowest I/O priority
ionice -c3 tar czf backup.tar.gz /data/

9. Monitor with sar

# Install sysstat
sudo apt-get install sysstat

# Enable data collection
sudo systemctl enable sysstat

# Check I/O statistics
sar -d 1 5
# Linux 5.15.0-91-generic    04/15/2024      _x86_64_    (4 CPU)
# 12:00:01 PM       DEV       tps    rkB/s    wkB/s  areq-sz  aqu-sz  await  svctm  %util
# 12:00:02 PM    nvme0n1    120.00   480.00  1800.00    18.00    0.50   1.50   1.20   8.55

10. Create an I/O monitoring script

#!/bin/bash
# io-monitor.sh

LOG_FILE="/var/log/io-monitor.log"
INTERVAL=5

while true; do
  echo "=== $(date) ===" >> "$LOG_FILE"
  iostat -x 1 1 | tail -n +4 >> "$LOG_FILE"
  echo "" >> "$LOG_FILE"
  
  # Check for high utilization
  UTIL=$(iostat -x 1 1 | awk '/sda/ && !/Device/ {print $NF}')
  if (( $(echo "$UTIL > 80" | bc -l) )); then
    echo "WARNING: sda utilization at ${UTIL}%" | logger -t io-monitor
  fi
  
  sleep $INTERVAL
done

11. Optimize filesystem mount options

# For databases
mount -o noatime,nodiratime,data=writeback,commit=60 /dev/sdb1 /var/lib/postgresql

# For web servers
mount -o noatime,nodiratime /dev/sda1 /var/www

Lessons Learned

Disk I/O bottlenecks are identified by high %util combined with high await times. Use iotop to identify which processes are causing the most I/O. Tune the I/O scheduler based on your workload: none for NVMe, mq-deadline for general use. And always monitor I/O with sar or iostat during peak hours to establish baseline performance metrics.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.