Airflow Executors Deep Dive: From Local to Kubernetes

What Executors Actually Do

The executor is the component that answers a deceptively simple question: how does Airflow actually run a task? When the scheduler decides that a task is ready to execute — all its dependencies have succeeded, the schedule has triggered, the pool has an open slot — it hands that task to the executor. The executor’s job is to take that task and run it somewhere.

“Somewhere” is the key word. Depending on which executor you choose, that task might run as a subprocess on the same machine as the scheduler, as a message dispatched to a distributed worker pool, or as a brand new container spun up in a Kubernetes cluster. The executor determines the concurrency model, the failure isolation, the resource limits, and the scalability ceiling of your entire Airflow deployment.

Think of the scheduler as a restaurant manager who decides which orders are ready to be cooked. The executor is the kitchen — it could be a single chef cooking one dish at a time, a team of chefs working in parallel on the same stove, a network of satellite kitchens receiving orders via radio, or a fleet of food trucks that spin up on demand. The orders are the same regardless of which kitchen you use, but the throughput, cost, and reliability are completely different.

SequentialExecutor: The Starting Point

The SequentialExecutor runs one task at a time, in the same process as the scheduler. It is the default executor that ships with Airflow, and it exists for exactly one purpose: making it easy to get started.

How It Works

When the scheduler finds a task ready to run, it executes it directly — synchronously, blocking, in the same Python process. No subprocess, no worker, no queue. The scheduler literally stops scheduling while the task runs. When the task finishes, the scheduler resumes looking for the next task to run.

Scheduler Process
  ├── Parse DAGs
  ├── Find ready tasks
  ├── Execute task_1  ← Blocks here until done
  ├── Execute task_2  ← Then blocks here
  └── ...

When to Use It

Only for local development and learning. The SequentialExecutor cannot run tasks in parallel, which means it cannot test parallelism in your DAGs. It uses SQLite as the metadata database (the only executor that supports SQLite), which means no concurrent access. It is useful for running through a tutorial or verifying that a DAG parses correctly, and that is it.

# airflow.cfg -- the default, usually what you want to move away from
[core]
executor = SequentialExecutor

Never use SequentialExecutor in production. It is like testing your web application on a single-threaded HTTP server — it verifies basic functionality but tells you nothing about how the system behaves under real load.

LocalExecutor: The First Real Executor

The LocalExecutor runs tasks as separate processes on the same machine as the scheduler. It is the simplest executor that supports parallelism, and it is a solid choice for small-to-medium deployments.

How It Works

When the scheduler identifies ready tasks, the LocalExecutor forks a new subprocess for each one. Each task runs in its own process with its own memory space, isolated from other tasks. The executor manages a pool of these processes, bounded by the parallelism setting.

Scheduler Process
  ├── Parse DAGs
  ├── Find ready tasks
  ├── Fork subprocess → task_1 (running)
  ├── Fork subprocess → task_2 (running)
  ├── Fork subprocess → task_3 (running)
  ├── Continue scheduling while tasks run...
  └── Collect results as subprocesses finish

The key difference from SequentialExecutor: the scheduler does not block. It forks the task process and immediately goes back to scheduling. Multiple tasks run simultaneously, limited only by the machine’s CPU and memory and the parallelism config.

Infrastructure Requirements

The LocalExecutor requires one significant upgrade from the default setup: a real database backend. SQLite does not support concurrent writes, and multiple task processes writing results simultaneously would corrupt it. You need PostgreSQL or MySQL.

# airflow.cfg
[core]
executor = LocalExecutor
parallelism = 32

[database]
sql_alchemy_conn = postgresql+psycopg2://airflow:password@localhost:5432/airflow

Strengths and Limitations

Strengths:

• Simple to set up — just change the executor config and point to a PostgreSQL database
• No additional infrastructure (no message broker, no separate workers)
• Good performance for up to 30-50 concurrent tasks on a reasonably sized machine
• Tasks share the same file system, making it easy to pass data via files

Limitations:

• Single machine bottleneck. All tasks run on one machine. If you need more CPU, memory, or I/O, your only option is a bigger machine (vertical scaling). You cannot add more machines.
• No resource isolation. A memory-hungry task can starve other tasks or crash the scheduler process. A task that consumes all available CPU slows down every other running task.
• No fault tolerance. If the machine goes down, everything goes down — the scheduler, the webserver, and all running tasks.

When to Use It

LocalExecutor is the right choice when:

• You have a small to medium number of DAGs (under 100)
• Your concurrent task count stays below 30-50
• You do not need fault tolerance beyond basic process restarts
• Your team does not want to manage Celery workers or a Kubernetes cluster

Many teams run LocalExecutor successfully for years. It covers the 80% case where you have a handful of important pipelines running on a single capable server.

CeleryExecutor: Horizontal Scaling

The CeleryExecutor distributes tasks across multiple worker machines using Celery, a distributed task queue framework. It is the traditional choice for medium-to-large Airflow deployments that need to scale beyond a single machine.

How It Works

The architecture introduces two new components: a message broker and Celery workers.

When the scheduler finds a task ready to run, it serializes the task into a message and pushes it onto the message broker (typically Redis or RabbitMQ). Celery workers — separate processes running on potentially separate machines — pull messages from the broker, execute the tasks, and report the results back to the Airflow metadata database.

┌─────────────┐     ┌───────────────┐     ┌──────────────┐
│  Scheduler  │────▶│ Message Broker│────▶│   Worker 1   │
│             │     │ (Redis/RMQ)   │     │  (machine A) │
└─────────────┘     │               │     └──────────────┘
                    │               │────▶┌──────────────┐
                    │               │     │   Worker 2   │
                    │               │     │  (machine B) │
                    └───────────────┘     └──────────────┘
                                          ┌──────────────┐
                                     ────▶│   Worker 3   │
                                          │  (machine C) │
                                          └──────────────┘

The broker acts as a buffer between the scheduler and the workers. The scheduler does not need to know which worker will pick up a task — it just puts it in the queue. Workers pull tasks when they have capacity. This decoupling is what enables horizontal scaling: you add capacity by adding more workers, not by upgrading a single machine.

Infrastructure Requirements

CeleryExecutor requires significantly more infrastructure than LocalExecutor:

1. Message broker — Redis or RabbitMQ. Redis is simpler to set up and more commonly used with Airflow. RabbitMQ is more feature-rich but adds operational complexity.
2. Multiple machines (or containers) running the Celery worker process
3. Shared access to DAG files — every worker needs the same DAG files available at the same path. Common solutions: NFS mount, git-sync sidecar, or baking DAGs into the Docker image.
4. Shared Python environment — every worker needs the same Python packages installed. A mismatch causes tasks to fail with import errors on some workers but not others.

# airflow.cfg
[core]
executor = CeleryExecutor

[celery]
broker_url = redis://redis:6379/0
result_backend = db+postgresql://airflow:password@postgres:5432/airflow
worker_concurrency = 16

Queues and Worker Specialization

One of CeleryExecutor’s powerful features is queue routing. You can assign tasks to specific queues, and configure workers to only consume from certain queues. This lets you specialize workers for different workloads.

# A GPU-intensive task goes to the GPU queue
@task(queue="gpu_workers")
def train_model():
    import tensorflow as tf
    # ...

# A memory-intensive task goes to the high-memory queue
@task(queue="highmem_workers")
def process_large_dataset():
    import pandas as pd
    # ...

# Default tasks go to the default queue
@task
def send_notification():
    # ...

# Start workers listening to specific queues
airflow celery worker --queues gpu_workers       # On GPU machines
airflow celery worker --queues highmem_workers   # On high-memory machines
airflow celery worker --queues default           # On standard machines

This is enormously useful in practice. You can route ML training tasks to machines with GPUs, large data processing to machines with 256 GB of RAM, and lightweight notification tasks to small, cheap instances. Without queue routing, you would need every worker to have the most expensive hardware configuration.

Strengths and Limitations

Strengths:

• Horizontal scaling — add workers to handle more load
• Queue routing enables hardware specialization
• Battle-tested at scale by many organizations
• Workers can be on different machines for fault tolerance

Limitations:

• Operational complexity. You are now running and monitoring Redis/RabbitMQ, multiple workers, and ensuring DAG files and Python environments are synchronized across all of them.
• Worker lifecycle management. Workers need to be provisioned, monitored, restarted on failure, and scaled up/down based on load. This is a non-trivial operational burden.
• Static resource allocation. Workers run continuously, consuming resources even when idle. You pay for the infrastructure whether or not tasks are running.
• Environment consistency. Every worker must have identical DAG files and Python packages. A version mismatch causes subtle, hard-to-debug failures.

KubernetesExecutor: Elastic and Isolated

The KubernetesExecutor takes a fundamentally different approach: instead of maintaining a pool of long-running workers, it creates a new Kubernetes pod for every single task. When the task finishes, the pod is destroyed.

How It Works

When the scheduler identifies a ready task, it tells the Kubernetes API to create a new pod. That pod runs the task, writes the result to the metadata database, and terminates. The pod’s container image, resource requests, environment variables, and secrets are all configurable per task.

┌─────────────┐     ┌────────────────────────┐
│  Scheduler  │────▶│   Kubernetes API        │
│             │     │                        │
└─────────────┘     │  ┌─────┐ ┌─────┐      │
                    │  │Pod 1│ │Pod 2│      │
                    │  │task_a│ │task_b│      │
                    │  └─────┘ └─────┘      │
                    │           ┌─────┐      │
                    │           │Pod 3│      │
                    │           │task_c│      │
                    │           └─────┘      │
                    └────────────────────────┘

Each pod is a completely isolated execution environment. It has its own file system, memory, CPU, and can even have its own Docker image. This means different tasks can use different Python versions, different package sets, or even different languages entirely.

Pod Templates and Per-Task Customization

The power of KubernetesExecutor comes from per-task configuration. You can specify resource limits, node selectors, tolerations, and even custom Docker images for individual tasks:

from kubernetes.client import models as k8s

# Custom pod configuration for a resource-intensive task
gpu_pod = k8s.V1Pod(
    spec=k8s.V1PodSpec(
        containers=[
            k8s.V1Container(
                name="base",
                resources=k8s.V1ResourceRequirements(
                    requests={"memory": "8Gi", "cpu": "4", "nvidia.com/gpu": "1"},
                    limits={"memory": "16Gi", "cpu": "8", "nvidia.com/gpu": "1"},
                ),
                image="company/airflow-ml:latest",
            )
        ],
        node_selector={"gpu": "true"},
    )
)

@task(executor_config={"pod_override": gpu_pod})
def train_model():
    import torch
    # This runs on a GPU node with 8Gi RAM and a dedicated GPU
    ...

# A lightweight task uses minimal resources
@task(executor_config={
    "pod_override": k8s.V1Pod(
        spec=k8s.V1PodSpec(
            containers=[
                k8s.V1Container(
                    name="base",
                    resources=k8s.V1ResourceRequirements(
                        requests={"memory": "256Mi", "cpu": "0.25"},
                        limits={"memory": "512Mi", "cpu": "0.5"},
                    ),
                )
            ]
        )
    )
})
def send_email():
    # Runs on a small pod, uses minimal cluster resources
    ...

Strengths and Limitations

Strengths:

• Elastic scaling. Pods are created on demand and destroyed after use. No idle workers consuming resources. During off-peak hours, resource usage drops to near zero.
• Complete isolation. A memory leak in one task cannot affect another. A crashed task does not take down other tasks.
• Per-task environments. Different tasks can run different Docker images, Python versions, and package sets.
• Resource limits. Kubernetes enforces CPU and memory limits per pod, preventing runaway tasks from affecting the cluster.

Limitations:

• Pod startup latency. Each task incurs 10-30 seconds of overhead for pod scheduling, image pulling, and container startup. For short tasks (under a minute), this overhead is significant.
• Requires Kubernetes. If your organization is not already running Kubernetes, adopting it just for Airflow is a large infrastructure investment.
• Debugging complexity. When a task fails, you need to check pod logs, pod events, and potentially node-level issues. The debugging surface area is larger than with local processes.
• Image management. You need a container registry, a build pipeline for your Airflow images, and a strategy for keeping images up to date.

Choosing the Right Executor

The decision matrix is simpler than it looks. Answer these questions in order:

Are you just learning or prototyping? Use SequentialExecutor. Change it later when you are ready for production.

Do you have fewer than 50 concurrent tasks and can live with single-machine risk? Use LocalExecutor. It is simple, requires minimal infrastructure, and handles the majority of real-world Airflow deployments.

Do you need to scale beyond one machine but do not have Kubernetes? Use CeleryExecutor. Add Redis, deploy workers, and scale horizontally.

Do you already run Kubernetes and want elastic scaling or per-task isolation? Use KubernetesExecutor. Pay the pod startup latency cost for better resource efficiency and isolation.

The Comparison Table

Feature	Sequential	Local	Celery	Kubernetes
Parallelism	None	Multi-process	Multi-machine	Pod per task
Scaling	None	Vertical only	Horizontal	Elastic
Infrastructure	SQLite	PostgreSQL	PostgreSQL + Broker + Workers	PostgreSQL + K8s cluster
Task isolation	None	Process-level	Process-level	Container-level
Startup overhead	None	Milliseconds	Milliseconds	10-30 seconds
Idle resource cost	Minimal	Fixed (one machine)	Fixed (worker pool)	Near zero
Complexity	Trivial	Low	Medium	High

Migration Path: Local → Celery → Kubernetes

Most organizations evolve through executors as their needs grow. Here is how to plan each migration.

Local to Celery

This migration is straightforward because the programming model does not change — your DAGs work the same way. The changes are infrastructure:

1. Set up a message broker. Start with Redis — it is simpler than RabbitMQ and works well for most deployments.
2. Deploy Celery workers. Start with one worker on the same machine as the scheduler. Verify everything works, then add workers on separate machines.
3. Ensure DAG file sync. All workers need the same DAG files. Set up git-sync, a shared NFS mount, or bake DAGs into the Docker image.
4. Ensure Python environment consistency. Pin all package versions in requirements.txt and use the same image or virtualenv across all workers.
5. Switch the executor config and restart.

# The key config changes for migration to Celery
[core]
executor = CeleryExecutor

[celery]
broker_url = redis://redis-host:6379/0
result_backend = db+postgresql://airflow:pass@postgres:5432/airflow
worker_concurrency = 16

Celery to Kubernetes

This migration is more involved because the execution model changes fundamentally. Tasks no longer run on persistent workers — they run in ephemeral pods.

1. Ensure your Airflow image is production-ready. Every task runs inside this image (unless overridden). It needs all your Python packages, DAG files, and configuration.
2. Test with a few DAGs first. Switch select DAGs to KubernetesExecutor while keeping others on Celery using the executor_config override.
3. Account for pod startup latency. Tasks that take 10 seconds to run now take 40 seconds (10s task + 30s pod overhead). Adjust SLAs and expectations.
4. Convert queue-based routing to pod templates. Where you previously used Celery queues to route tasks to specialized workers, use executor_config with pod overrides.
5. Monitor cluster resources. Kubernetes needs enough capacity to handle your peak pod count. Configure pod resource requests and limits to prevent over-provisioning.

The Hybrid Approach: CeleryKubernetesExecutor

Airflow also offers a CeleryKubernetesExecutor that lets you use both executors simultaneously. Tasks default to Celery workers for fast execution, but individual tasks can be routed to Kubernetes pods for isolation or custom resource needs.

[core]
executor = CeleryKubernetesExecutor

# This task runs on Celery (fast, no pod overhead)
@task
def quick_check():
    ...

# This task runs on Kubernetes (isolated, custom resources)
@task(executor_config={
    "KubernetesExecutor": {
        "pod_override": gpu_pod_spec
    }
})
def train_model():
    ...

This hybrid approach is often the best of both worlds during a migration. Lightweight tasks run on Celery workers with millisecond startup, while heavyweight tasks get their own isolated pods.

Monitoring Executor Health

Regardless of which executor you choose, monitor these metrics:

# Key metrics to watch (exposed via StatsD)
# executor.open_slots     -- Available capacity
# executor.queued_tasks   -- Tasks waiting for execution
# executor.running_tasks  -- Tasks currently executing

# For CeleryExecutor, also monitor:
# celery.worker.online    -- Number of active workers
# celery.worker.tasks     -- Tasks per worker

# For KubernetesExecutor, also monitor:
# kubernetes.pending_pods -- Pods waiting to be scheduled
# kubernetes.running_pods -- Pods currently running

If queued_tasks is consistently high while open_slots is zero, you need more capacity — either a bigger machine (Local), more workers (Celery), or more cluster resources (Kubernetes).

Next Steps

The executor is the foundation of your Airflow deployment. Choose based on your current scale, but design your DAGs to be executor-agnostic. Well-written tasks work the same way regardless of whether they run as a local subprocess, a Celery message, or a Kubernetes pod. When you outgrow your current executor, the migration is an infrastructure change, not a code change.

• Production Deployment — Deploy Airflow with your chosen executor using Docker Compose, Helm, or a managed service.
• Dynamic DAGs — Understanding executor performance helps you design dynamic DAG generation that does not overwhelm the scheduler.
• Best Practices — Review pool configuration and parallelism settings that work hand-in-hand with your executor choice.