tooolbox/blog/Linux cgroup Memory Limits: Controlling Application Memory Usage
architecture2025-02-28·13 min·163/348

Linux cgroup Memory Limits: Controlling Application Memory Usage

How to set and manage memory limits for processes using Linux cgroups, including practical examples for Java, Node.js, and containerized workloads.

Introduction

Without memory limits, a single application can consume all available RAM and starve the entire system. Linux cgroups (control groups) provide a kernel-level mechanism to allocate, restrict, and account resource usage per process or group of processes. Understanding cgroups is essential for running multi-tenant systems, containers, and any environment where memory isolation matters.

Environment

The examples use cgroup v2 on Debian 12, which is the modern standard. For systems still running cgroup v1, the concepts are similar but the interface differs.

stat -f -c %T /sys/fs/cgroup/
# cgroup2fs

cat /proc/filesystems | grep cgroup
# nodev   cgroup
# nodev   cgroup2

Problem

A development server runs multiple services on a single machine. One service with a memory leak gradually consumes all available RAM, causing the OOM killer to terminate other services or the entire system becoming unresponsive.

# Without limits
ps aux --sort=-%mem | head -5
# USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
# appuser  1234  5.2 89.3 12345678 7340234 ?   Sl   10:00  45:23 java -Xmx6g -jar app.jar

The Java process is using 89% of system memory with no way for other processes to compete.

Analysis

cgroups work by assigning processes to groups and applying resource limits to those groups. For memory, cgroups can:

  • Set hard limits (memory.max) that trigger OOM when exceeded
  • Set soft limits (memory.high) that throttle before hitting the hard limit
  • Track memory usage (memory.current)
  • Account for cache vs anonymous memory
  • Control swap usage (memory.swap.max)

The cgroup filesystem is typically at /sys/fs/cgroup/:

ls /sys/fs/cgroup/
# cgroup.controllers  cgroup.stat  init.scope  system.slice  user.slice

Each directory represents a cgroup. Processes are assigned by writing their PID to cgroup.procs.

Solution

1. Create a cgroup with memory limits

sudo mkdir /sys/fs/cgroup/java-app
echo "5G" | sudo tee /sys/fs/cgroup/java-app/memory.max
echo "4G" | sudo tee /sys/fs/cgroup/java-app/memory.high
echo $JAVA_PID | sudo tee /sys/fs/cgroup/java-app/cgroup.procs

memory.max is the hard limit. When exceeded, the OOM killer activates within the cgroup. memory.high is a soft limit that triggers memory reclaim and throttling before the hard limit is reached.

2. Check memory usage of a cgroup

cat /sys/fs/cgroup/java-app/memory.current
# 4831838208    # ~4.5 GB in bytes

cat /sys/fs/cgroup/java-app/memory.stat
# anon 4294967296
# file 536870912
# slab 12345678

3. Use systemd to manage cgroup limits

# /etc/systemd/system/myapp.service
[Service]
MemoryMax=5G
MemoryHigh=4G
ExecStart=/usr/bin/java -jar /opt/myapp/app.jar

Apply immediately:

sudo systemctl daemon-reload
sudo systemctl restart myapp.service

Verify:

systemctl show myapp.service | grep -i memory
# MemoryMax=5368709120
# MemoryHigh=4294967296
# MemoryCurrent=4294967296

4. Set limits for a process tree

For a shell and all its children:

sudo mkdir /sys/fs/cgroup/build-env
echo "2G" | sudo tee /sys/fs/cgroup/build-env/memory.max
echo $$ | sudo tee /sys/fs/cgroup/build-env/cgroup.procs
make -j4    # This process and children are now limited

5. Monitor cgroup memory usage

# Real-time monitoring
watch -n 1 cat /sys/fs/cgroup/java-app/memory.current

# Or use systemd-cgtop
sudo systemd-cgtop
# Path                                     Tasks  %CPU   Memory  Input/s Output/s
# /system.slice/java-app.service              12   5.2    4.5G        -        -

6. Handle OOM within a cgroup

When a cgroup hits its memory limit:

# Check OOM events
cat /sys/fs/cgroup/java-app/memory.events
# oom 1
# oom_kill 1

# The OOM killer within the cgroup kills only processes in that cgroup
# Other cgroups are unaffected

7. Set swap limits

echo "1G" | sudo tee /sys/fs/cgroup/java-app/memory.swap.max

8. Use cgroup namespaces for isolation

sudo unshare --cgroup --mount-proc bash
# Now in a new cgroup namespace
echo "1G" > /sys/fs/cgroup/memory.max

9. Reset limits when done

echo "max" | sudo tee /sys/fs/cgroup/java-app/memory.max
echo "max" | sudo tee /sys/fs/cgroup/java-app/memory.high

10. Remove a cgroup

# First, move all processes out
for pid in $(cat /sys/fs/cgroup/java-app/cgroup.procs); do
  echo $pid | sudo tee /sys/fs/cgroup/cgroup.procs
done
sudo rmdir /sys/fs/cgroup/java-app

Lessons Learned

cgroups are the foundation of container resource management. Setting MemoryHigh (soft limit) before MemoryMax (hard limit) gives applications time to clean up before being killed. For Java applications, ensure the heap plus native memory stays below the cgroup limit. Monitor cgroup memory with systemd-cgtop to catch applications approaching their limits before they are terminated. And always test cgroup limits in staging before applying them to production services.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.