Python 메모리 관리 (garbage collection)

Introduction

Python manages memory automatically, which is one of its greatest conveniences. But this automation can lull developers into ignoring memory usage until a long-running process consumes all available RAM and crashes. Understanding how Python's garbage collector works is essential for building reliable applications, especially those that process large datasets or run for extended periods.

I discovered the importance of Python memory management when a cryptocurrency trading bot I built in Lisbon started consuming 8GB of RAM after running for 24 hours. The bot processed thousands of price updates per second, and without proper memory management, old data accumulated until the process was killed by the OOM killer.

Environment

Python 3.12.3
Linux 6.5.0 (Ubuntu 22.04)
8GB RAM

Problem

Symptom 1: Memory grows unbounded

import time

class PriceTracker:
    def __init__(self):
        self.history = []
    
    def add_price(self, price):
        self.history.append({
            "price": price,
            "timestamp": time.time()
        })

tracker = PriceTracker()

# Simulating 24 hours of trading
for i in range(1_000_000):
    tracker.add_price(100.0 + i * 0.01)
    # Memory grows indefinitely — no cleanup

print(f"History size: {len(tracker.history)}")
# This will consume several GB of RAM

Symptom 2: Circular references prevent cleanup

class Node:
    def __init__(self):
        self.parent = None
        self.children = []
    
    def add_child(self, child):
        child.parent = self
        self.children.append(child)

# Create circular reference
root = Node()
child = Node()
root.add_child(child)

# Delete references
del root
del child
# Memory is NOT freed because of circular reference
# Python's reference counting cannot handle this

Symptom 3: Global variables prevent garbage collection

import gc

class LargeObject:
    def __init__(self, data):
        self.data = data

# Store in global dict
cache = {}

def process_data():
    obj = LargeObject([i for i in range(1000000)])
    cache["data"] = obj  # Never cleaned up

process_data()
gc.collect()
# Memory is NOT freed because cache holds a reference

Analysis

Python uses two main memory management mechanisms.

Reference counting: Every object in Python has a reference counter. When the reference count drops to zero, the object is immediately deallocated. This handles most memory cleanup.

import sys

a = [1, 2, 3]
print(sys.getrefcount(a))  # 2 (a + getrefcount argument)

b = a  # Reference count increases
print(sys.getrefcount(a))  # 3

del b  # Reference count decreases
print(sys.getrefcount(a))  # 2

Generational garbage collector: The cyclic GC handles reference cycles — situations where objects reference each other, keeping their reference counts above zero. Python divides objects into three generations:

• Generation 0: New objects (checked most frequently)
• Generation 1: Survived one GC cycle
• Generation 2: Survived two GC cycles (checked least frequently)

Common memory leak patterns:

1. Caches that grow without bounds
2. Circular references between objects
3. Closures capturing large objects
4. Threads that never terminate
5. Event listeners that are never removed

Solution

Fix 1: Monitor memory usage

import tracemalloc
import psutil
import os

def get_memory_usage():
    """Get current memory usage in MB."""
    process = psutil.Process(os.getpid())
    return process.memory_info().rss / 1024 / 1024

def track_memory():
    """Track memory growth over time."""
    tracemalloc.start()
    
    initial = get_memory_usage()
    print(f"Initial memory: {initial:.1f} MB")
    
    # Your code here
    for i in range(1000000):
        data = [i for i in range(100)]
    
    current = get_memory_usage()
    print(f"Current memory: {current:.1f} MB")
    print(f"Growth: {current - initial:.1f} MB")
    
    # Show top memory allocations
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics('lineno')
    for stat in top_stats[:5]:
        print(stat)

track_memory()

Fix 2: Use weak references for caches

import weakref

class DataCache:
    def __init__(self):
        self._cache = weakref.WeakValueDictionary()
    
    def get(self, key):
        return self._cache.get(key)
    
    def set(self, key, value):
        self._cache[key] = value

cache = DataCache()

class ExpensiveObject:
    def __init__(self, data):
        self.data = data

# Objects can be garbage collected even if in cache
obj = ExpensiveObject([1, 2, 3])
cache.set("key1", obj)
del obj  # Object may be collected if no other references

Fix 3: Break circular references explicitly

class Node:
    def __init__(self):
        self.parent = None
        self.children = []
    
    def add_child(self, child):
        child.parent = self
        self.children.append(child)
    
    def cleanup(self):
        """Break circular references before deletion."""
        for child in self.children:
            child.parent = None
        self.children.clear()
        self.parent = None

root = Node()
child = Node()
root.add_child(child)

# Before deleting, clean up
root.cleanup()
del root
del child

Fix 4: Use __del__ and context managers

class DatabaseConnection:
    def __init__(self, connection_string):
        self.conn = None
        self.connection_string = connection_string
    
    def __enter__(self):
        self.conn = create_connection(self.connection_string)
        return self
    
    def __exit__(self, exc_type, exc_val, exc_tb):
        if self.conn:
            self.conn.close()
            self.conn = None

# Resource is cleaned up when exiting context
with DatabaseConnection("postgresql://...") as db:
    db.execute("SELECT * FROM users")
# Connection is closed automatically

Fix 5: Force garbage collection

import gc
import sys

# Collect all generations
gc.collect()

# Check for uncollectable objects
gc.set_debug(gc.DEBUG_SAVEALL)
gc.collect()
print(f"Uncollectable objects: {len(gc.garbage)}")

# Set GC thresholds for more frequent collection
gc.set_threshold(500, 5, 1)  # More aggressive GC

Lessons Learned

• Monitor memory usage in production using tracemalloc or psutil to catch leaks early.
• Use WeakValueDictionary and WeakSet for caches that should not prevent garbage collection.
• Break circular references explicitly when creating parent-child relationships.
• Use context managers (with statement) for resource cleanup instead of relying on __del__.
• Run gc.collect() periodically in long-running applications to force cleanup of circular references.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.