Linux Memory Overcommit: Understanding vm.overcommit_memory

Introduction

Linux allows processes to allocate more memory than is physically available. This is called memory overcommit, and it is enabled by default. The idea is that most processes allocate memory they never fully use. But when processes do use all their allocated memory, the OOM killer intervenes. Understanding and tuning overcommit behavior is critical for running memory-sensitive workloads like databases.

Environment

The examples use Ubuntu 22.04 LTS with 32 GB of RAM, running PostgreSQL and Redis alongside application servers.

cat /proc/meminfo | head -5
# MemTotal:       32813564 kB
# MemFree:         2345678 kB
# MemAvailable:   18234567 kB
# Buffers:         1234567 kB
# Cached:         12345678 kB

sysctl vm.overcommit_memory
# vm.overcommit_memory = 0

Problem

Under memory pressure, the OOM killer terminates PostgreSQL:

dmesg | grep -i "oom\|killed"
# [  543.21] Out of memory: Killed process 12345 (postgres) total-vm:8388608kB, anon-rss:6291456kB

The issue is that overcommit allowed other processes to allocate memory that was never available, leading to a situation where PostgreSQL could not fulfill its allocations.

Analysis

Linux provides three overcommit modes, controlled by vm.overcommit_memory:

The commit limit is calculated as:

CommitLimit = SwapTotal + (MemTotal * overcommit_ratio / 100)

Check the current settings:

sysctl vm.overcommit_ratio
# vm.overcommit_ratio = 50

# Calculate commit limit
cat /proc/meminfo | grep -E "MemTotal|SwapTotal"
# MemTotal:       32813564 kB
# SwapTotal:       2097152 kB
# CommitLimit:    18503934 kB  (approximately)

Solution

1. Check overcommit behavior

# Check current mode
cat /proc/sys/vm/overcommit_memory
# 0 = heuristic, 1 = always, 2 = never

# Check how much memory is committed
cat /proc/meminfo | grep Committed
# Committed_AS:   24567890 kB

# Compare with commit limit
echo "Scale=2; 24567890 / 18503934 * 100" | bc
# 132.76  (Overcommitted by 32%)

2. Set never-overcommit for databases

# /etc/sysctl.d/99-memory.conf
vm.overcommit_memory = 2
vm.overcommit_ratio = 80

# Apply
sudo sysctl -p /etc/sysctl.d/99-memory.conf

With overcommit_memory=2, memory allocations that would exceed the commit limit fail immediately rather than being allowed and later killed by OOM.

3. Set always-overcommit for Redis

Redis benefits from overcommit_memory=1 because it uses fork() for persistence:

# Redis specific setting
echo "vm.overcommit_memory = 1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

4. Monitor overcommit

# Check committed memory vs limit
cat /proc/meminfo | grep -E "Committed_AS|CommitLimit"

# Create a monitoring script
#!/bin/bash
COMMITTED=$(grep Committed_AS /proc/meminfo | awk '{print $2}')
LIMIT=$(grep CommitLimit /proc/meminfo | awk '{print $2}')
PERCENT=$((COMMITTED * 100 / LIMIT))
if [ "$PERCENT" -gt 90 ]; then
  echo "ALERT: Memory commit at ${PERCENT}%" | logger -t mem-monitor
fi

5. Use cgroups for per-service limits

# Instead of global overcommit, use cgroup limits
echo "4G" | sudo tee /sys/fs/cgroup/postgres/memory.max
echo $POSTGRES_PID | sudo tee /sys/fs/cgroup/postgres/cgroup.procs

6. Test memory allocation behavior

# Test with a simple allocation program
cat > /tmp/test_overcommit.c << 'EOF'
#include 
#include 
#include 

int main() {
    size_t size = 4ULL * 1024 * 1024 * 1024;  // 4GB
    void *ptr = mmap(NULL, size, PROT_READ | PROT_WRITE,
                     MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (ptr == MAP_FAILED) {
        perror("mmap failed");
        return 1;
    }
    printf("Allocated 4GB at %p\n", ptr);
    // Touch the memory to force physical allocation
    memset(ptr, 0, size);
    printf("Memory touched successfully\n");
    munmap(ptr, size);
    return 0;
}
EOF
gcc /tmp/test_overcommit.c -o /tmp/test_overcommit
/tmp/test_overcommit

Lessons Learned

The default overcommit mode (0) works well for most workloads. For databases, overcommit_memory=2 prevents sudden OOM kills by refusing allocations that exceed the commit limit. For Redis, overcommit_memory=1 is recommended. Always monitor Committed_AS to understand your actual memory commitment. And remember that overcommit tuning is one part of a comprehensive memory management strategy that includes cgroups, swap configuration, and application-level memory management.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.