tooolbox/blog/RAID Array Degraded State Recovery on Linux: A Step-by-Step Guide
troubleshooting2025-01-15·12 min·227/348

RAID Array Degraded State Recovery on Linux: A Step-by-Step Guide

How to diagnose and recover a degraded RAID array on Linux, including drive replacement, rebuild procedures, and monitoring with mdadm.

Introduction

A RAID array in degraded state is operating with one or more failed drives. The array still functions thanks to redundancy, but it is vulnerable. A second failure means data loss. Recognizing degraded state early and knowing how to recover is critical for any system administrator managing Linux servers with software RAID.

Environment

The examples use mdadm 4.2 on Debian 12, with a RAID 5 array built from three 4TB SATA drives. One drive has failed, putting the array in degraded state.

mdadm --version
# mdadm - v4.2 - 2023-01-15

cat /proc/mdstat
# md0 : active raid5 sdc1[2] sdb1[1]
#       7814026240 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]

Problem

A critical alert fires: the RAID array is degraded. The kernel logs show I/O errors on one drive:

dmesg | grep -i "error\|fail\|md"
# [  123.456] md/raid5:md0 not enough operational devices. (2 of 3, 2 needed)
# [  123.457] md/raid5:md0 Disk failure on sda1, disabling device.

The array is running in degraded mode with only two of three drives operational.

Analysis

RAID 5 can survive one drive failure because it distributes parity across all drives. When a drive fails, the missing data can be reconstructed from the remaining drives and parity.

Check the array status:

cat /proc/mdstat
# md0 : active raid5 sdc1[2] sdb1[1]
#       7814026240 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]

# The underscore indicates the failed device (sda1 missing)
sudo mdadm --detail /dev/md0
# /dev/md0
#    Version : 1.2
# Creation Time : Mon Jan 15 10:00:00 2025
#      Raid Level : raid5
#      Array Size : 7814026240 (7452.39 GiB 8001.56 GB)
#   Used Dev Size : 3907013120 (3726.19 GiB 4000.78 GB)
#    Raid Devices : 3
#   Total Devices : 2
#       Persistence : Superblock is persistent
#
#       State : clean, degraded
#  Active Devices : 2
# Working Devices : 2
#  Failed Devices : 1
#   Spare Devices : 0
#
#          Layout : left-symmetric
#      Chunk Size : 512K
#
#    Number   Major   Minor   RaidDevice State
#       0       0       0        0      removed
#       1       8      17        1      active sync   /dev/sdb1
#       2       8      33        2      active sync   /dev/sdc1
#
#    0       8       1        -      faulty   /dev/sda1

Solution

1. Identify the failed drive

sudo mdadm --detail --test /dev/md0
# Or check individual drives
sudo smartctl -H /dev/sda

2. Remove the failed drive from the array

sudo mdadm --fail /dev/md0 /dev/sda1
sudo mdadm --remove /dev/md0 /dev/sda1

3. Physically replace the drive

After installing the new drive, partition it to match the array configuration:

# Find the new drive
lsblk
# NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
# sda      8:0    0   4.0T  0 disk    # New drive
# sdb      8:16   0   4.0T  0 disk
# sdc      8:32   0   4.0T  0 disk

# Create partition
sudo parted /dev/sda mklabel gpt
sudo parted /dev/sda mkpart primary 0% 100%
sudo mdadm --examine /dev/sdb1    # Check superblock info for consistency

4. Add the new drive to the array

sudo mdadm --add /dev/md0 /dev/sda1

5. Monitor the rebuild

watch cat /proc/mdstat
# md0 : active raid5 sda1[3] sdc1[2] sdb1[1]
#       7814026240 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
#       [>....................]  recovery = 3.2% (125829120/3907013120) finish=345.6min speed=185432K/sec
# Check rebuild speed
cat /proc/mdstat | grep speed
# [>....................]  recovery = 3.2% speed=185432K/sec

6. Adjust rebuild speed

The default rebuild speed is conservative. Increase it if the system can handle the I/O:

# Increase minimum speed (bytes/sec)
echo 50000000 | sudo tee /proc/sys/dev/raid/speed_limit_min

# Increase maximum speed (bytes/sec)
echo 200000000 | sudo tee /proc/sys/dev/raid/speed_limit_max

7. Verify array health after rebuild

sudo mdadm --detail /dev/md0
# State : clean
#  Active Devices : 3
# Working Devices : 3
#  Failed Devices : 0

# Run a consistency check
sudo echo check > /sys/block/md0/md/sync_action

8. Update mdadm.conf

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

9. Set up monitoring

# Add email alerts
echo "MAILADDR admin@example.com" | sudo tee -a /etc/mdadm/mdadm.conf

# Enable the mdmonitor service
sudo systemctl enable mdmonitor
sudo systemctl start mdmonitor

Lessons Learned

RAID is not a backup. Degraded arrays are a warning, not a solution. Always maintain hot spares for critical arrays to minimize the window of vulnerability. Monitor arrays proactively with mdmonitor and email alerts. And test your recovery procedures regularly. Knowing how to recover an array before it fails is far better than learning under pressure.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.