Deploying Apache Airflow to Production

What Changes from Local Dev to Production

Running Airflow on your laptop is straightforward. You install it with pip, run airflow standalone, and everything works in a single process with SQLite. Production is a different world entirely, and the gap catches many teams off guard.

The differences are not just about scale — they are about reliability, security, and operational visibility. A local setup has no redundancy (the scheduler goes down, everything stops), no real authentication (anyone with network access can trigger DAGs), no monitoring (you discover failures by checking the UI manually), and no proper secrets management (connection passwords live in the database in plain text or in environment variables).

Production Airflow means answering questions your local setup ignores: What happens when the scheduler crashes at 3 AM? How do you rotate database credentials without downtime? How do you know that the scheduler is falling behind before tasks start queuing for hours? How do you prevent a junior engineer from accidentally deleting a production DAG?

This guide walks through the infrastructure decisions and configurations that answer those questions.

Database Backend: The Foundation

The metadata database is Airflow’s brain. Every DAG run, task instance, XCom value, variable, connection, and log pointer lives here. Choosing and configuring the right database is the single most important production decision after choosing your executor.

PostgreSQL vs MySQL

PostgreSQL is the recommended choice. It handles concurrent writes better (important when many tasks report results simultaneously), supports JSON columns natively (useful for XCom), and has better support for the FOR UPDATE SKIP LOCKED pattern that Airflow uses for task scheduling. The Airflow community tests primarily against PostgreSQL, and most production deployments use it.

MySQL works and is officially supported, but you are more likely to encounter edge cases and performance issues at scale. If your organization already runs MySQL and has strong operational expertise, it is a viable choice. Otherwise, default to PostgreSQL.

# airflow.cfg
[database]
sql_alchemy_conn = postgresql+psycopg2://airflow:${POSTGRES_PASSWORD}@db-host:5432/airflow

# Connection pool tuning for production
sql_alchemy_pool_size = 10
sql_alchemy_max_overflow = 20
sql_alchemy_pool_recycle = 1800

Database Sizing and Maintenance

The metadata database grows continuously. Every DAG run creates rows in dag_run and task_instance tables. XCom values (task outputs) can be large if tasks pass significant data between them. Log pointers accumulate for every task execution.

For production:

• Run database cleanup regularly. Airflow’s airflow db clean command removes old records. Schedule it as a weekly job.
• Monitor database size. Set alerts when the database exceeds 80% of available storage.
• Use a managed database service (RDS, Cloud SQL, Azure Database) for automated backups, failover, and patching.

# Clean records older than 90 days
airflow db clean --clean-before-timestamp "2026-04-01 00:00:00" --yes

Docker Compose: Small Team Deployment

Docker Compose is the sweet spot for teams that need a real production setup but do not want to manage Kubernetes. It gives you multi-container orchestration, process isolation, and reproducible environments on a single server or a small cluster.

The Architecture

A typical Docker Compose deployment runs these services:

┌─────────────────────────────────────────────────┐
│                 Docker Host                     │
│                                                 │
│  ┌────────────┐  ┌────────────┐  ┌───────────┐ │
│  │ Webserver  │  │ Scheduler  │  │  Worker   │ │
│  │  :8080     │  │            │  │ (Celery)  │ │
│  └────────────┘  └────────────┘  └───────────┘ │
│                                                 │
│  ┌────────────┐  ┌────────────┐  ┌───────────┐ │
│  │ PostgreSQL │  │   Redis    │  │ Flower    │ │
│  │  :5432     │  │  :6379     │  │  :5555    │ │
│  └────────────┘  └────────────┘  └───────────┘ │
└─────────────────────────────────────────────────┘

Docker Compose Configuration

# docker-compose.yaml
x-airflow-common: &airflow-common
  image: apache/airflow:2.9.0-python3.11
  environment: &airflow-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://redis:6379/0
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres:5432/airflow
    AIRFLOW__CORE__FERNET_KEY: ${FERNET_KEY}
    AIRFLOW__WEBSERVER__SECRET_KEY: ${WEBSERVER_SECRET_KEY}
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: "true"
    AIRFLOW__CORE__LOAD_EXAMPLES: "false"
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - ./config:/opt/airflow/config

services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 10s
      retries: 5

  redis:
    image: redis:7
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      retries: 5

  airflow-init:
    <<: *airflow-common
    command: >
      bash -c "
        airflow db migrate &&
        airflow users create
          --username admin
          --password ${ADMIN_PASSWORD}
          --firstname Admin
          --lastname User
          --role Admin
          --email admin@company.com
      "
    depends_on:
      postgres:
        condition: service_healthy

  webserver:
    <<: *airflow-common
    command: airflow webserver
    ports:
      - "8080:8080"
    depends_on:
      airflow-init:
        condition: service_completed_successfully
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      retries: 5

  scheduler:
    <<: *airflow-common
    command: airflow scheduler
    depends_on:
      airflow-init:
        condition: service_completed_successfully

  worker:
    <<: *airflow-common
    command: airflow celery worker
    depends_on:
      airflow-init:
        condition: service_completed_successfully

  flower:
    <<: *airflow-common
    command: airflow celery flower
    ports:
      - "5555:5555"

volumes:
  postgres-data:

Generating the Fernet Key

The Fernet key encrypts sensitive data in the metadata database (connection passwords, variable values). Generate it once and store it securely:

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Store this in a .env file (never committed to git) or in a secrets manager.

Helm Chart: Kubernetes Deployment

For teams already running Kubernetes, the official Airflow Helm chart is the production-grade deployment method. It handles the complexity of running multiple Airflow components as Kubernetes workloads with proper health checks, resource limits, and scaling configurations.

Installing the Helm Chart

# Add the Apache Airflow Helm repository
helm repo add apache-airflow https://airflow.apache.org
helm repo update

# Install with custom values
helm install airflow apache-airflow/airflow \
  --namespace airflow \
  --create-namespace \
  --values values.yaml

Key Configuration (values.yaml)

# values.yaml
executor: KubernetesExecutor

# Use your custom Airflow image with DAGs baked in
images:
  airflow:
    repository: company-registry.com/airflow
    tag: "2.9.0-v42"

# Or use git-sync to pull DAGs from a repository
dags:
  gitSync:
    enabled: true
    repo: git@github.com:company/airflow-dags.git
    branch: main
    subPath: dags
    wait: 60
    sshKeySecret: airflow-git-ssh

# Database -- use an external managed database
data:
  metadataConnection:
    user: airflow
    pass: ${POSTGRES_PASSWORD}
    protocol: postgresql
    host: airflow-db.abc123.us-east-1.rds.amazonaws.com
    port: 5432
    db: airflow

# Resource limits
scheduler:
  resources:
    requests:
      cpu: "1"
      memory: "2Gi"
    limits:
      cpu: "2"
      memory: "4Gi"

webserver:
  resources:
    requests:
      cpu: "0.5"
      memory: "1Gi"
    limits:
      cpu: "1"
      memory: "2Gi"

# Enable StatsD for monitoring
statsd:
  enabled: true

# Enable log persistence
logs:
  persistence:
    enabled: true
    size: 50Gi

Managed Services: Let Someone Else Operate It

If your team’s core competency is building pipelines, not operating infrastructure, a managed Airflow service eliminates the operational burden of running the scheduler, workers, database, and monitoring stack.

Astronomer (Astro)

Built by Airflow committers, Astronomer provides a managed Airflow platform with a developer-focused experience. You get environment management, CI/CD integration, monitoring, and support from people who know Airflow deeply. Available as a cloud service or self-hosted on your Kubernetes cluster.

Best for: Teams that want dedicated Airflow expertise and are willing to pay for a specialized platform.

Amazon MWAA (Managed Workflows for Apache Airflow)

AWS’s managed Airflow service. You provide DAG files in an S3 bucket, MWAA runs everything else. Integrates natively with IAM roles, VPCs, and other AWS services. The trade-off is that you are on AWS’s Airflow version schedule (typically a few months behind the latest release) and have limited control over the execution environment.

Best for: AWS-heavy teams that want seamless IAM and VPC integration without managing infrastructure.

Google Cloud Composer

GCP’s managed Airflow service, running on GKE (Google Kubernetes Engine) under the hood. Integrates with BigQuery, GCS, and other GCP services. Cloud Composer 2 uses a more efficient architecture with auto-scaling workers.

Best for: GCP-heavy teams, especially those using BigQuery as their primary data warehouse.

Which Managed Service?

Feature	Astronomer	MWAA	Cloud Composer
Cloud	Any / Self-hosted	AWS only	GCP only
Airflow version	Latest	Months behind	Months behind
Customization	High	Limited	Moderate
Pricing model	Per deployment	Per environment-hour	Per environment-hour
Best integration	Cloud-agnostic	AWS services	GCP services

Monitoring: Knowing Before Users Do

A production Airflow deployment without monitoring is a ticking time bomb. You need to know when the scheduler is falling behind, when tasks are failing, and when resource usage is approaching limits — before your stakeholders tell you.

StatsD + Prometheus + Grafana

Airflow emits metrics via StatsD. A common production stack pipes these through a StatsD exporter into Prometheus for storage and Grafana for visualization and alerting.

# airflow.cfg
[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 9125
statsd_prefix = airflow

Critical Metrics to Monitor

Scheduler health:

• scheduler.scheduler_heartbeat — Is the scheduler alive? Alert if this stops.
• dag_processing.total_parse_time — How long does it take to parse all DAGs? Alert if this exceeds 60% of min_file_process_interval.
• scheduler.tasks.starving — Tasks that are ready but cannot run due to pool or parallelism limits.

Task execution:

• ti.successes / ti.failures — Success and failure rates over time.
• dag_run.duration.{dag_id} — How long each DAG takes to complete.
• executor.queued_tasks — Tasks waiting for execution. High values indicate insufficient capacity.

Infrastructure:

• Database connection pool usage
• Worker CPU and memory utilization
• Disk usage for logs

# Example Grafana alert rule
- alert: AirflowSchedulerDown
  expr: absent(airflow_scheduler_heartbeat) == 1
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Airflow scheduler heartbeat missing for 2 minutes"

- alert: AirflowHighTaskFailureRate
  expr: rate(airflow_ti_failures[5m]) > 0.1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Task failure rate exceeds 10% over 5 minutes"

Logging: Finding the Needle

When a task fails at 3 AM, the first thing the on-call engineer needs is the log. Production logging configuration determines whether that engineer finds the answer in 30 seconds or 30 minutes.

Remote Logging

By default, Airflow writes task logs to the local file system. In a distributed deployment (Celery workers, Kubernetes pods), logs end up on the machine where the task ran. If that machine is gone (an autoscaled worker, a terminated pod), the logs are gone too.

Remote logging sends task logs to a centralized storage system:

# airflow.cfg -- S3 remote logging
[logging]
remote_logging = True
remote_log_conn_id = aws_default
remote_base_log_folder = s3://airflow-logs/task-logs/

# GCS alternative
# remote_base_log_folder = gs://airflow-logs/task-logs/

With remote logging, task logs persist regardless of what happens to the worker. The Airflow UI automatically reads from remote storage when you click “Log” on a task instance.

Log Aggregation

For operational visibility beyond individual task logs, pipe Airflow component logs (scheduler, webserver, worker) into a log aggregation system like the ELK stack (Elasticsearch, Logstash, Kibana) or Grafana Loki. This lets you search across all components, correlate events, and build dashboards for operational patterns.

Security: The Non-Negotiable Pieces

RBAC (Role-Based Access Control)

Airflow ships with a built-in RBAC system. In production, create specific roles with limited permissions rather than giving everyone Admin access:

# Create roles via CLI
# airflow roles create DataEngineer
# airflow roles create DataViewer

# Assign permissions
# airflow roles add-perms DataViewer
#   --action can_read --resource DAGs
#   --action can_read --resource DAG Runs

Common role structure:

• Admin — Full access, limited to platform team
• Engineer — Can create/edit DAGs, trigger runs, view logs
• Viewer — Read-only access to the UI, no ability to trigger or modify

Secrets Backends

Storing database passwords and API keys in Airflow’s metadata database (even encrypted with Fernet) is not ideal for production. Secrets backends let Airflow fetch credentials from external secrets managers at runtime:

# airflow.cfg -- HashiCorp Vault
[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {
    "connections_path": "connections",
    "variables_path": "variables",
    "url": "https://vault.company.com:8200",
    "auth_type": "approle"
}

# airflow.cfg -- AWS Secrets Manager
[secrets]
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {
    "connections_prefix": "airflow/connections",
    "variables_prefix": "airflow/variables"
}

With a secrets backend, credentials are never stored in Airflow’s database. They are fetched on demand from a system designed for secrets management — with audit logs, rotation policies, and fine-grained access control.

Network Security

• Run the webserver behind a reverse proxy (Nginx, ALB) with HTTPS
• Restrict metadata database access to Airflow components only
• Use private networking for worker-to-scheduler communication
• Enable CSRF protection in the webserver config

# airflow.cfg
[webserver]
enable_proxy_fix = True
cookie_secure = True
cookie_samesite = Lax

Pre-Production Checklist

Before declaring your Airflow deployment production-ready, verify:

• PostgreSQL (not SQLite) as the metadata database
• Executor is LocalExecutor, CeleryExecutor, or KubernetesExecutor
• Fernet key is generated and stored securely
• Webserver secret key is set and stored securely
• Remote logging is configured (S3, GCS, or Azure Blob)
• StatsD metrics are flowing to your monitoring stack
• RBAC is enabled with role-appropriate access
• Secrets backend is configured for sensitive credentials
• Database backups are automated and tested
• Health checks are configured for scheduler, webserver, and workers
• DAG files are deployed via git-sync or image bake (not manual copy)
• CI/CD pipeline runs DAG validation tests before deployment

Next Steps

Production Airflow is an ongoing operational commitment, not a one-time setup. Start with Docker Compose if you are a small team, graduate to Helm on Kubernetes as you scale, and consider managed services if operating infrastructure is not your core strength.

• Executors Deep Dive — Choose the right executor for your deployment before configuring infrastructure.
• Testing DAGs — Build the CI/CD pipeline that prevents broken DAGs from reaching your production deployment.
• Real-World Patterns — Design patterns for multi-environment setups, error handling, and idempotent pipelines.