Celery worker 실행 에러 해결

Introduction

Celery is the most popular distributed task queue for Python, but getting a worker running smoothly can be surprisingly difficult. Errors range from simple import failures to complex broker connection issues and serialization mismatches. The error messages are often vague, pointing you to the wrong part of the codebase.

I spent an entire day debugging a Celery worker on a Lisbon VPS when it refused to process tasks after a Python upgrade. The worker started fine, connected to Redis, but silently dropped every task it received. The root cause turned out to be a pickle serialization incompatibility between Python 3.11 and 3.12.

Environment

Python 3.12.3
Celery 5.3.6
Redis 7.2.4 (broker)
Ubuntu 22.04 LTS

Problem

Error 1: Worker fails to start

$ celery -A myproject worker --loglevel=info

[2024-11-01 10:00:00,001: ERROR/MainProcess] Unrecoverable error: ImportError('cannot import name my_task from myproject.tasks')

Error 2: Worker starts but tasks fail

$ celery -A myproject worker --loglevel=info

[2024-11-01 10:00:01,234: INFO/MainProcess] Task myproject.tasks.process_order[abc123] received
[2024-11-01 10:00:01,235: ERROR/MainProcess] Task myproject.tasks.process_order[abc123] raised unexpected: SerializationError(...)

Error 3: Redis connection refused

$ celery -A myproject worker --loglevel=info

[2024-11-01 10:00:00,567: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 111 connecting to localhost:6379. Connection refused.

Error 4: Worker hangs on shutdown

$ celery -A myproject worker --loglevel=info
# ... runs fine ...
# Ctrl+C
[2024-11-01 12:00:00,000: WARNING/MainProcess] worker shutting down...
# Hangs for 30+ seconds

Analysis

Celery worker errors fall into several categories.

Category 1: Import errors. The worker cannot find the task module. This happens when the celery_app configuration does not include the correct modules, or when the working directory is wrong.

Category 2: Serialization errors. Celery defaults to pickle serialization, which can fail when task arguments contain objects that are not picklable, or when Python versions differ between producer and worker.

Category 3: Broker connection issues. Redis or RabbitMQ is not running, or the connection URL is wrong.

Category 4: Concurrency issues. The worker uses too many threads/processes for the available resources.

Check worker status:

# Check if worker is running
celery -A myproject inspect active

# Check registered tasks
celery -A myproject inspect registered

# Check worker stats
celery -A myproject inspect stats

Solution

Fix 1: Verify task discovery

# celery_app.py
from celery import Celery

app = Celery('myproject')
app.config_from_object('django.conf:settings', namespace='CELERY')

# Explicitly include task modules
app.autodiscover_tasks(['myproject.tasks'])

# Verify tasks are registered
@app.task
def process_order(order_id):
    return f"Processed order {order_id}"

# Check in shell
from celery_app import app
print(app.tasks.keys())

Fix 2: Use JSON serialization instead of pickle

# celery_app.py
app.config.update(
    task_serializer='json',
    accept_content=['json'],
    result_serializer='json',
    timezone='UTC',
    enable_utc=True,
    result_backend='redis://localhost:6379/0',
)

Or in Django settings:

# settings.py
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_RESULT_SERIALIZER = 'json'

Fix 3: Fix Redis connection

# Check if Redis is running
redis-cli ping

# Start Redis if not running
sudo systemctl start redis

# Test connection
celery -A myproject inspect ping

# Verify broker URL in config
import os
print(os.environ.get("CELERY_BROKER_URL", "redis://localhost:6379/0"))

Fix 4: Configure proper shutdown

# celery_app.py
app.config.update(
    worker_shutdown_timeout=30,  # Wait up to 30s for tasks to finish
    worker_prefetch_multiplier=1,  # Don't prefetch tasks
    task_acks_late=True,  # Acknowledge after completion, not before
)

Fix 5: Run worker with proper options

# Development
celery -A myproject worker --loglevel=info --concurrency=2

# Production with gevent
celery -A myproject worker --loglevel=info --pool=gevent --concurrency=100

# Production with prefork
celery -A myproject worker --loglevel=info --pool=prefork --concurrency=4

Fix 6: Monitor worker health

# Check worker status
celery -A myproject inspect active --timeout=5

# Check reserved tasks
celery -A myproject inspect reserved

# Check failed tasks
celery -A myproject inspect revoked

# Flower monitoring (install first: pip install flower)
celery -A myproject flower --port=5555

Lessons Learned

• Always use JSON serialization in production — pickle is a security risk and breaks across Python versions.
• Set task_acks_late=True to prevent task loss when a worker crashes mid-execution.
• Use worker_shutdown_timeout to give tasks time to finish gracefully.
• Test task discovery by running celery -A myproject inspect registered after startup.
• Monitor with Flower to get real-time visibility into worker health and task throughput.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.