Diagnosing High 'iowait' and Storage Bottlenecks using iotop and fio

One of the most frustrating experiences for a system administrator is watching a server crawl to a halt while the CPU and RAM usage appear completely normal. Users complain about timeouts, SSH sessions lag, and database queries that usually take milliseconds are suddenly taking seconds.

When your compute and memory aren't the culprits, the silent killer is almost always storage. Specifically, your system is likely suffering from a severe Input/Output (I/O) bottleneck, manifesting as high iowait.

In this comprehensive tutorial, we will strip away the mystery of storage performance. We will move beyond the basic top command and utilize industry-standard tools—iostat, iotop, and fio—to identify rogue processes, analyze disk utilization, and benchmark your hardware to determine if your SATA or NVMe drives are failing.

What You'll Learn

Prerequisites

To follow along with this tutorial, you will need:

• A Linux-based server (Ubuntu/Debian, CentOS/RHEL, or similar).

• Root or sudo privileges to install packages and run disk-level diagnostics.

• Basic familiarity with the Linux command-line interface.

Understanding I/O Wait (The Silent Bottleneck)

Before we start installing tools, we need to understand what we are looking for. When you run a standard performance monitor like top or htop, you will see a CPU summary line that looks something like this:

%Cpu(s): 5.2 us, 1.5 sy, 0.0 ni, 42.1 id, 49.8 wa, 0.0 hi, 1.4 si, 0.0 st

Most admins look at us (user space) and sy (system space), see that they are low, and assume the CPU is fine. But the critical metric here is wa (iowait), sitting at a massive 49.8%.

What exactly is wa?

I/O wait is the percentage of time that your CPU cores are sitting completely idle because they are waiting for a disk (or network) operation to complete.

The CPU is thousands of times faster than a storage drive. If a process asks the disk to retrieve data, the CPU halts execution of that thread and waits for the disk to fetch the data. If the disk is saturated or failing, the CPU spends all of its time waiting, essentially locking up the system even though the processor isn't doing any actual math.

If your wa consistently spikes above 10-15% on a multi-core system, you have a storage bottleneck that requires immediate investigation.

High-Level Diagnosis with iostat

To confirm that the disk is the issue, we need to look at the hardware itself. The best tool for this is iostat, which is part of the sysstat package.

Installing sysstat

Update your package manager and install the tool:

For Ubuntu/Debian:

sudo apt update && sudo apt install sysstat -y

For CentOS/RHEL/AlmaLinux:

sudo dnf install sysstat -y

Running iostat

To get a real-time, human-readable view of your disks, run iostat with extended statistics (-x), in megabytes (-m), refreshing every 2 seconds (2):

iostat -xm 2

Example Output:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
nvme0n1           0.00     2.50  150.00  300.00     0.60     4.20    21.84     0.45    1.00    0.50    1.25   0.80   36.00
sda               0.00    15.00   25.00  110.00     0.10     1.50    24.29     3.50   28.50   12.00   32.20   7.40  100.00

Interpreting the Metrics

Do not be overwhelmed by the wall of numbers. Focus your attention on these four specific columns:

• r/s and w/s (IOPS): Reads and Writes per second. This tells you how many individual requests the disk is handling.

• rMB/s and wMB/s (Throughput): The actual volume of data moving back and forth.

• await (Latency): The average time (in milliseconds) for I/O requests issued to the device to be served. This is crucial. For NVMe drives, this should be sub-millisecond. For SATA SSDs, under 5ms. For spinning HDDs, anything consistently over 20-30ms indicates severe struggling. In our example above, sda is struggling at 28.5ms.

• %util (Utilization): The percentage of CPU time during which I/O requests were issued to the device. Think of this as how "busy" the disk is. In our example, sda is at 100.00%. It is completely maxed out, establishing our bottleneck.

Pinpointing Rogue Processes with iotop

Now that iostat has confirmed that sda is at 100% utilization, we need to know what is causing it. Is it a runaway database query? A daily backup script? Or a compromised system thrashing swap space?

For this, we use iotop, which acts exactly like top but exclusively for storage.

Installing iotop

# Ubuntu/Debian
sudo apt install iotop -y

# CentOS/RHEL/AlmaLinux
sudo dnf install iotop -y

Running iotop

You must run iotop with sudo because it requires kernel-level hooks to map I/O usage to specific PIDs. Use these flags for the cleanest output:

sudo iotop -oPa

• -o (Only): Only show processes actually doing I/O right now.

• -P (Processes): Show processes instead of individual threads (reduces clutter).

• -a (Accumulated): Show the total accumulated I/O since iotop started, rather than the momentary speed. This is invaluable for catching processes that "spike" for a second and then sleep.

Example Output:

Total DISK READ :       0.00 B/s | Total DISK WRITE :      25.50 M/s
Actual DISK READ:       0.00 B/s | Actual DISK WRITE:      28.10 M/s
    PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
  14502 be/4 mysql         0.00 B      5.20 G  0.00 % 85.20 % mysqld
  21004 be/4 root          0.00 B      1.10 G  0.00 % 12.00 % rsync -a /var/www /backup
    992 be/4 root          0.00 B     15.00 M  0.00 %  0.10 % systemd-journald

The Diagnosis

The IO> column represents the percentage of time the process spent waiting on I/O. In this scenario, we caught the culprits red-handed.

A MySQL database (PID 14502) is writing gigabytes of data and spending 85% of its time waiting on the disk. Simultaneously, an rsync backup script is running (PID 21004).

The Solution here is software-based: You would schedule the rsync backup for off-peak hours (e.g., 3:00 AM) or use ionice to lower its I/O priority, preventing it from competing with your database for disk write cycles.

Benchmarking Storage Health with fio

If iotop shows that your applications are barely moving any data (e.g., a few megabytes per second), but iostat shows the disk at 100% utilization and iowait is skyrocketing, you likely have a failing hardware drive.

To prove this, we need to test the maximum capabilities of the disk.

Note: Important Note on dd: Many older tutorials suggest using the dd command (e.g., dd if=/dev/zero of=testfile...). Do not use dd for benchmarking. It is single-threaded, easily fooled by RAM caching, and cannot test random I/O (which is how databases actually work). Use fio.

Installing fio

# Ubuntu/Debian
sudo apt install fio -y

# CentOS/RHEL/AlmaLinux
sudo dnf install fio -y

The Golden Rule of Benchmarking

Never run write tests directly on block devices (like /dev/sda) if they contain data. It will overwrite your file system and destroy your server. Always run fio against a test file located inside a mounted directory (e.g., /tmp or /var/www).

Running fio Tests

Storage drives behave differently depending on the workload. We will run two distinct tests: Random I/O (simulating databases/web servers) and Sequential I/O (simulating large file transfers/backups).

Test 1: Random Read/Write (Database Simulation)

This test creates a 4GB file and hits it with randomized 4-kilobyte chunks, perfectly simulating a heavy database workload.

sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 \
--name=database_test --filename=/tmp/fio_test_file.tmp --bs=4k \
--iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

Understanding the Parameters:

Test 2: Sequential Write (Throughput Simulation)

This test writes large 1-Megabyte blocks in a straight line, simulating dumping a massive backup archive or migrating a large database SQL file.

sudo fio --name=throughput_test --filename=/tmp/fio_test_file.tmp \
--ioengine=libaio --direct=1 --bs=1M --iodepth=64 --size=4G --readwrite=write

Note: Clean up your test file after you are done by running sudo rm /tmp/fio_test_file.tmp

Interpreting fio Results

When fio finishes, it will dump a significant amount of data. Here is a simulated output of a Random Read test on a struggling SATA SSD, and how to read it:

database_test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B
...
read: IOPS=1250, BW=4.88MiB/s (5120kB/s)(3072MiB/629145msec)
   slat (usec): min=2, max=412, avg=8.45, stdev=4.20
   clat (usec): min=150, max=185400, avg=51200.50, stdev=1250.25
   lat (usec): min=155, max=185405, avg=51208.95, stdev=1254.30

Key Metric 1: IOPS and BW

Look at the line: read: IOPS=1250, BW=4.88MiB/s.

• IOPS (Input/Output Operations Per Second): This SSD is achieving 1,250 IOPS.

• BW (Bandwidth): It is moving 4.88 Megabytes per second.

Is this good or bad?

• A standard Enterprise NVMe drive should push 100,000+ IOPS on this test.

• A healthy SATA SSD should push 10,000 to 80,000 IOPS.

• A spinning 7200RPM Hard Drive will usually cap out around 100 to 200 IOPS.

Conclusion: If this drive is advertised as an SSD, 1,250 IOPS means it is severely degraded, misconfigured, or throttling due to overheating.

Key Metric 2: Latency (clat)

Latency is measured in microseconds (usec). Divide by 1000 to get milliseconds (ms).

• slat: Submission latency (time taken to submit the request to the OS).

• clat: Completion latency (time taken for the disk hardware to actually do the work).

• lat: Total latency (slat + clat).

Focus on clat. In our example, the average clat is 51200.50 usec (which is 51.2 milliseconds). The max latency spiked to 185400 usec (185 milliseconds).

The Verdict: Any SSD showing 50ms average completion latency on a simple 4k random read test is experiencing massive hardware strain. The controller is likely failing, or the NAND flash cells are degraded. This hardware needs to be replaced.

Remediation Strategies

If you have completed this tutorial and confirmed a storage bottleneck, here are your next steps depending on the findings:

1. If iotop pointed to software:

1. Database Tuning: If MySQL/PostgreSQL is the culprit, check your slow query logs. A missing index forces the database to scan the entire disk instead of reading from RAM. Increase your innodb_buffer_pool_size so data lives in memory, not on disk.

2. Swap Thrashing: If the SWAPIN column in iotop is high, your server is out of RAM and is using the disk as emergency memory. Add more physical RAM to the server.

3. Schedule Management: Move intensive I/O operations (backups, log rotations, virus scans) to off-peak hours. Use the ionice -c 3 command to force backup scripts into "Idle" I/O priority.

2. If fio pointed to hardware limits:

1. Change Schedulers: For SATA SSDs and NVMe drives, ensure your Linux I/O scheduler is set to none or mq-deadline rather than bfq (which is optimized for spinning platters).

2. Upgrade Hardware: You cannot defeat physics. If your application legitimately requires 50,000 IOPS and you are running on a 7200RPM SATA drive, no amount of Linux tuning will save you. You must migrate your data to an NVMe block storage volume.

Summary

High iowait doesn't have to be a mystery. By systematically checking your overall CPU states (top), analyzing real-time disk saturation (iostat), isolating the exact processes writing to the disk (iotop), and proving the physical limits of your hardware (fio), you can definitively solve storage bottlenecks and restore your server's performance.