Diagnosing Linux Kernel Panic: Root Cause Analysis

A kernel panic is the Linux equivalent of a Blue Screen of Death. When my trading server experienced a kernel panic at 3 AM during a market event, I had to diagnose it quickly to restore service. This guide covers systematic kernel panic analysis.

Environment

OS: Ubuntu 22.04 LTS
Kernel: 5.15.0-84-generic
Hardware: Dell PowerEdge R740, 64GB RAM, RAID controller

$ uname -r
5.15.0-84-generic

$ cat /proc/version
Linux version 5.15.0-84-generic (buildd@lcy02-amd64-045) (gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #94-Ubuntu SMP

The Problem

The server rebooted unexpectedly. After restarting, the system logs showed a kernel panic:

$ dmesg | grep -i panic
[    0.000000] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    0.000000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.15.0-84-generic #94-Ubuntu
[    0.000000] Call Trace:
[    0.000000]  dump_stack_lvl+0x4a/0x5e
[    0.000000]  panic+0x102/0x2e0
[    0.000000]  mount_block_root+0x155/0x220

Analysis

Step 1: Check Previous Boot Logs

# Check previous boot logs
$ journalctl -b -1 -k | tail -50
-- Logs begin at Fri 2025-06-24 08:00:00 WET, end at Fri 2025-06-25 03:15:00 WET. --
Jun 24 23:45:00 server kernel: BUG: unable to handle page fault at ffff888123456789
Jun 24 23:45:00 server kernel: PGD 0 P4D 0
Jun 24 23:45:00 server kernel: Oops: 0000 [#1] SMP NOPTI
Jun 24 23:45:00 server kernel: CPU: 3 PID: 12345 Comm: kworker/3:1 Tainted: G S
Jun 24 23:45:00 server kernel: RIP: 0010:process_data+0x1a5/0x300 [my_module]
Jun 24 23:45:00 server kernel: Call Trace:
Jun 24 23:45:00 server kernel:  worker_thread+0x35/0x3c0
Jun 24 23:45:00 server kernel: Kernel panic - not syncing: Fatal exception

Step 2: Check for kdump Crash Dumps

# Check if kdump is configured
$ systemctl status kdump.service
● kdump.service - kexec-jump based crash kernel
     Loaded: loaded (/lib/systemd/system/kdump.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

# Check for crash dumps
$ ls -la /var/crash/
total 0
# No crash dumps available - kdump was not enabled

Step 3: Analyze dmesg for Hardware Errors

$ sudo dmesg | grep -iE "error|fault|bug|warn|硬件"
[    1.234567] mpt3sas0: fault info: 0x12345678
[    1.234568] mpt3sas0: IOCStatus: 0x0048
[    1.234569] mpt3sas0: LogInfo: 0x31110000
[   45.678901] EDAC MC0: UE row 0, channel-a, label "DIMM_A1": Memory error

# Check for OOM kills before the panic
$ journalctl -b -1 | grep -i "oom\|out of memory"
Jun 24 23:40:00 server kernel: Out of memory: Killed process 12345 (python3)

Step 4: Check Kernel Configuration

# Check if kernel panic timeout is set
$ cat /proc/sys/kernel/panic
0

# Check crash kernel allocation
$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.0-84-generic root=UUID=xxxx ro quiet splash

# No crashkernel parameter - kdump cannot work

Step 5: Check Filesystem Health

# Check filesystem errors
$ sudo dumpe2fs -h /dev/sda2 | grep -i error
dumpe2fs 1.46.5 (30-Dec-2021)
Filesystem state:       clean with errors

# This indicates filesystem corruption that may have triggered the panic

Solution

Enable kdump for Future Crash Analysis

# Install kdump
$ sudo apt install linux-crashdump kdump-tools

# Enable kdump
$ sudo systemctl enable kdump.service

# Configure crash kernel memory (128MB recommended)
$ sudo sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="/GRUB_CMDLINE_LINUX_DEFAULT="crashkernel=128M /' /etc/default/grub
$ sudo update-grub

# Reboot to activate
$ sudo reboot

Fix Filesystem Errors

# Boot from live USB or recovery mode
$ sudo fsck -y /dev/sda2
fsck from util-linux 2.37.2
e2fsck 1.46.5 (30-Dec-2021)
/dev/sda2: Inode 1234567: block count larger than filesystem size
/dev/sda2: Inode 1234567: past end of inode block map
/dev/sda2: FREE_UNCLED INODES FOUND
/dev/sda2: Clearing free inode list
/dev/sda2: clean with errors, 123/65536 files, 12345/65536 blocks

Analyze Crash Dump (After Enabling kdump)

# Find crash dump
$ ls -la /var/crash/
drwx------ 2 root root 4096 Jun 25 03:00 127.0.0.1-2025-06-25-03:00:12

# Use crash utility
$ sudo apt install crash
$ sudo crash /usr/lib/debug/boot/vmlinux-5.15.0-84-generic \
    /var/crash/127.0.0.1-2025-06-25-03\:00\:12/dump.2025-06-25-03:00:12

crash> bt
PID: 12345  TASK: ffff8881234abcd0  CPU: 3   COMMAND: "kworker/3:1"
 #0 [ffffc90000abcde0] machine_kexec at ffffffff81056789
 #1 [ffffc90000abcde8] __crash_kexec at ffffffff81123456
 #2 [ffffc90000abcdf0] panic at ffffffff81234abc
 #3 [ffffc90000abcdf8] oops_end at ffffffff81056def

crash> log | tail -20
# Shows kernel log buffer at time of crash

Check Hardware

# Run memory test
$ sudo memtester 4G 1
# Or check existing ECC errors
$ sudo edac-util -s
$ sudo edac-util --status

# Check disk health
$ sudo smartctl -a /dev/sda | grep -i "reallocated\|pending\|uncorrectable"
Reallocated_Sector_Ct   0x0033   100   100   000    Pre-fail  -       0
Current_Pending_Sector  0x0033   100   100   000    Pre-fail  -       0
Offline_Uncorrectable   0x0033   100   100   000    Pre-fail  -       0

Prevention

# 1. Enable kernel panic timeout (auto-reboot)
$ echo "kernel.panic = 10" | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p

# 2. Enable NMI watchdog for hardware error detection
$ echo "kernel.nmi_watchdog = 1" | sudo tee -a /etc/sysctl.conf

# 3. Monitor for OOM conditions
$ sudo apt install earlyoom
$ sudo systemctl enable earlyoom

Lessons Learned

Enable kdump on production servers before you need it - you cannot retroactively get crash dumps.
Check journalctl -b -1 for the previous boot's kernel log to find panic causes.
Filesystem errors can cause kernel panics - run fsck periodically.
Hardware issues (bad RAM, failing disks) are a common root cause - run diagnostics.
Set kernel.panic = 10 to auto-reboot after panic for faster recovery.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.