Django Celery Task Queue Tutorial

What and Why

A web request should finish in milliseconds, not seconds. Anything slow, flaky, or external, like sending email, charging cards, generating reports, or calling third-party APIs, belongs in a background worker. Celery is the most common task queue in the Django ecosystem. It moves work off the request thread and onto a pool of workers that can scale independently.

Celery does three jobs: it serializes a function call into a message, persists that message in a broker, and runs it on a worker process. Around that core it adds retries, scheduling, chaining, and result tracking.

Mental Model

Picture two separate Python processes connected by a message queue. The Django process publishes tasks. A separate worker process consumes them. Neither knows the other exists at runtime, which is exactly the point: workers can crash and restart, and unfinished tasks remain in the broker.

The broker is typically Redis or RabbitMQ. You also have an optional result backend if you want to query task status or fetch return values. For most apps, results are write-and-forget.

Hands-on Example

Add Celery to a Django project. Install celery and redis, then create myproj/celery.py.

import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproj.settings")

app = Celery("myproj")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

In settings.py, point Celery at Redis.

CELERY_BROKER_URL = "redis://localhost:6379/0"
CELERY_RESULT_BACKEND = "redis://localhost:6379/1"
CELERY_TASK_ACKS_LATE = True
CELERY_WORKER_PREFETCH_MULTIPLIER = 1

In myproj/__init__.py, import the app so Django boots it.

from .celery import app as celery_app
__all__ = ["celery_app"]

Now write a task. Put it in emails/tasks.py.

from celery import shared_task
from django.core.mail import send_mail

@shared_task(bind=True, max_retries=3, default_retry_delay=30)
def send_welcome_email(self, user_id):
    from users.models import User
    try:
        user = User.objects.get(pk=user_id)
        send_mail(
            "Welcome",
            f"Hi {user.name}, glad to have you.",
            "noreply@example.com",
            [user.email],
        )
    except Exception as exc:
        raise self.retry(exc=exc)

Trigger it from a view or signal.

send_welcome_email.delay(user.id)

Run a worker locally.

celery -A myproj worker -l info

Django view -> task.delay() -> Broker (Redis)
                                |
                                v
                          Worker pool -> Task code
                                |
                                v
                        Result backend (optional)

For scheduled jobs, add django-celery-beat and define a periodic task in the admin, or use the static CELERY_BEAT_SCHEDULE setting.

from celery.schedules import crontab

CELERY_BEAT_SCHEDULE = {
    "purge-tokens": {
        "task": "auth.tasks.purge_expired_tokens",
        "schedule": crontab(hour=3, minute=0),
    },
}

Run beat alongside the worker.

celery -A myproj beat -l info

Common Pitfalls

• Passing model instances as arguments. Tasks serialize their arguments. Send IDs and re-fetch inside the task.
• Long-running database transactions in the calling view. If you call task.delay() inside a transaction and the task reads the same row, the worker might see stale state. Use transaction.on_commit(lambda: task.delay(id)).
• Forgetting idempotency. A retried task may run twice. Make sure double execution is safe or guarded by a uniqueness check.
• Mixing CPU-heavy and IO-heavy tasks in one queue. Route them to separate queues so a slow CPU job does not starve fast email tasks.
• Running beat twice. Two beat processes will fire schedules twice. Only one beat process should exist per environment.

Practical Tips

• Set task_acks_late=True so messages are only acknowledged after the task finishes, not when it is picked up.
• Use time_limit and soft_time_limit to keep stuck tasks from blocking workers forever.
• Name tasks explicitly with @shared_task(name="emails.send_welcome") so renames do not break in-flight messages.
• Monitor with Flower or your APM. A queue that grows without bound is a bug, not a feature.
• Keep tasks small. Compose them with chain, group, and chord rather than writing one giant task.

Wrap-up

Celery turns slow request paths into snappy ones by moving work to dedicated workers. Configure a broker, write small idempotent tasks, schedule recurring jobs with beat, and you have a robust foundation for asynchronous Django. From there, the path to specialized queues, priority routing, and distributed workflows is just configuration.