Apache Airflow Best Practices for Production

Why Best Practices Matter

A DAG that works on your laptop can fail spectacularly in production. The scheduler bogs down because a well-intentioned variable lookup runs on every parse cycle. A retry produces duplicate records because nobody thought about what happens when a task runs twice. A deployment overwrites a running DAG mid-execution because there is no CI/CD pipeline enforcing safety.

These are not edge cases. They are the everyday reality of running Airflow at scale, and they are the difference between a data platform your team trusts and one that wakes people up at 3 AM. This guide is written from the perspective of lessons learned the hard way — each best practice comes with the “why” and the “what happens if you skip it” story.

DAG Design: Getting the Foundations Right

Use a Fixed start_date (And Never a Dynamic One)

Every DAG needs a start_date, and it is tempting to set it to something like days_ago(1) or datetime.now() so you do not have to think about it. Here is why that is dangerous: Airflow’s scheduler parses your DAG file repeatedly. Every time it parses, days_ago(1) evaluates to a new date. This means the scheduler sees a constantly shifting start date, which confuses its internal bookkeeping about which runs have happened and which have not. You end up with phantom runs, missed runs, or runs that trigger unexpectedly.

The fix is simple. Pick a fixed date — the day you create the DAG, the first of the current month, whatever makes sense — and hardcode it. It never needs to change, and your scheduler stays happy.

# BAD: start_date changes every parse cycle
dag = DAG('broken_dag', start_date=days_ago(1))

# GOOD: Fixed, deterministic start_date
dag = DAG('stable_dag', start_date=datetime(2026, 1, 1), catchup=False)

Set catchup=False Unless You Specifically Need Backfills

Airflow’s catchup parameter defaults to True, which means that when you deploy a new DAG or unpause one that has been off for a while, Airflow will create a DAG run for every single missed schedule interval between start_date and now. If your DAG is scheduled daily and you set start_date to six months ago, that is roughly 180 DAG runs firing at once.

Sometimes this is exactly what you want — for instance, if you are loading historical data and need to process each day independently. But most of the time, it is a surprise that floods your scheduler and overwhelms your downstream systems. Set catchup=False unless you have a specific reason to backfill.

Make Every Task Idempotent

Idempotency means that running a task once or running it five times with the same input produces exactly the same result. This sounds academic until you consider what actually happens in production.

Picture this: your pipeline runs nightly, loading the day’s orders into a warehouse table. Monday night’s run succeeds, inserting 1,200 rows. Tuesday morning, someone notices the numbers look off and re-runs Monday’s task. If the task uses INSERT, you now have 2,400 rows for Monday — 1,200 duplicates. Every downstream report, every dashboard, every metric that depends on this table is now wrong. And the worst part? Nothing threw an error. The task succeeded both times. You only find out days later when someone notices the revenue chart has a suspicious spike.

An idempotent version of the same task would delete Monday’s partition before inserting, or use an UPSERT that replaces existing records. Running it ten times still produces exactly 1,200 rows.

# BAD: Appends data on every run, creates duplicates on retry
@task
def load_data_bad(records):
    hook = PostgresHook('warehouse')
    hook.insert_rows('events', records)

# GOOD: Delete-then-insert ensures same result on every run
@task
def load_data_good(records, **context):
    hook = PostgresHook('warehouse')
    ds = context['ds']
    hook.run(f"DELETE FROM events WHERE partition_date = '{ds}'")
    hook.insert_rows('events', records)

Every task you write should answer the question: “What happens if this runs twice?” If the answer is “bad things,” redesign it.

Keep Tasks Atomic and Focused

A task should do one thing. Not “extract, transform, and load” in a single function, but three separate tasks: one to extract, one to transform, one to load. There are practical reasons for this beyond code cleanliness.

First, if a monolithic task fails at the “load” step, you have to re-run the entire thing — including the expensive “extract” step that already succeeded. With atomic tasks, you retry only the task that failed. Second, small tasks give you better visibility in the Airflow UI. You can see exactly which step failed, how long each step took, and where the bottleneck is. Third, small tasks enable parallelism. If extract, transform, and load are separate tasks, Airflow can run independent branches in parallel rather than waiting for one giant function to finish.

# BAD: One task does everything
@task
def do_everything():
    data = extract_from_api()
    transformed = clean_and_transform(data)
    load_to_warehouse(transformed)
    send_notification()

# GOOD: Separate, focused tasks
@task
def extract():
    return extract_from_api()

@task
def transform(data):
    return clean_and_transform(data)

@task
def load(data):
    load_to_warehouse(data)

Performance: Keeping the Scheduler Healthy

Never Call Variable.get() at the Module Level

This is covered in detail in the Connections and Variables guide, but it deserves repetition here because it is the most common performance killer in Airflow deployments.

The scheduler parses every DAG file roughly every 30 seconds. Any Variable.get() or BaseHook.get_connection() call at the top level of a file runs on every parse. With 100 DAGs each making two variable lookups, that is 400 database queries per minute just for parsing — before a single task has even run. The metadata database slows down, the scheduler falls behind, and tasks start queuing up.

Move all variable and connection lookups inside task functions or use Jinja templates. Both approaches defer the database query to execution time.

Move Heavy Imports Inside Task Functions

The same principle applies to Python imports. If your DAG file imports pandas, tensorflow, or a large custom library at the module level, that import runs on every parse cycle. These libraries can take seconds to load, and when the scheduler is parsing hundreds of files, those seconds add up fast.

# BAD: Heavy imports at module level slow down every parse
import pandas as pd
import tensorflow as tf

# GOOD: Import inside the task function
@task
def train_model():
    import pandas as pd
    import tensorflow as tf
    # ...

Use .airflowignore to Reduce Parsing Scope

The scheduler examines every file in your DAGs folder looking for DAG definitions. If your folder contains test files, utility modules, documentation, or virtual environments, the scheduler wastes time parsing files that will never contain DAGs.

Create a .airflowignore file in your DAGs folder (it works like .gitignore) to exclude non-DAG files:

tests/
utils/
README.md
__pycache__
.git

Production Configuration

Choosing the Right Executor

Your executor determines how Airflow actually runs tasks. Picking the wrong one for your scale is like putting a bicycle engine in a truck — it technically works, but not well.

The SequentialExecutor runs one task at a time. It is useful only for local development and testing. Never use it in production.

The LocalExecutor runs tasks as separate processes on a single machine. It is a solid choice for small to medium deployments — up to around 50 concurrent tasks. No extra infrastructure required beyond a PostgreSQL metadata database.

The CeleryExecutor distributes tasks across multiple worker machines using a message queue (typically Redis or RabbitMQ). This is the traditional choice for medium to large Airflow deployments. It requires more infrastructure to manage but scales horizontally by adding workers.

The KubernetesExecutor spins up a new Kubernetes pod for every task. Each task runs in its own isolated container, which is ideal for environments with variable load (scale to zero when idle, scale up when busy) and for tasks that have different dependency requirements. The trade-off is pod startup latency — each task takes a few seconds longer to start than with Celery.

Pools: Throttling Access to Shared Resources

Pools limit how many tasks can access a shared resource concurrently. Without them, you might accidentally open 50 simultaneous connections to a database that can only handle 10, or overwhelm an API with more requests than its rate limit allows.

Think of a pool as a velvet rope at a club. There is a fixed number of slots, and tasks have to wait their turn. You create a pool with a name and a slot count, then assign tasks to it.

# Create a pool: airflow pools set database_pool 5 "Limit concurrent DB connections"

@task(pool='database_pool')
def query_database():
    """Only 5 instances of this can run concurrently."""
    ...

Monitoring and Alerting: The 3 AM Problem

Here is a story that plays out at every company that runs Airflow without proper monitoring. It is 3 AM. A critical pipeline that feeds the morning executive dashboard fails. Nobody notices until 8 AM when the VP of Sales opens a blank dashboard and sends an urgent Slack message. The on-call engineer scrambles, discovers that the pipeline failed at 3:07 AM due to a connection timeout, and realizes that if they had been notified at 3:07, they could have fixed it and re-run the pipeline with time to spare.

This story illustrates why monitoring is not optional. Airflow provides several mechanisms to ensure you know about failures the moment they happen, not five hours later.

Failure Callbacks

The most fundamental alerting mechanism is the failure callback. You write a function that gets called whenever a task fails, and you attach it to your DAG or individual tasks. The callback receives Airflow’s context, which includes the DAG ID, task ID, execution date, and a link to the logs — everything an on-call engineer needs to start debugging.

def slack_failure_alert(context):
    task_instance = context['task_instance']
    dag_id = context['dag'].dag_id
    task_id = task_instance.task_id
    log_url = task_instance.log_url

    message = (
        f"Task Failed\n"
        f"DAG: {dag_id}\n"
        f"Task: {task_id}\n"
        f"Log: {log_url}"
    )

    from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
    hook = SlackWebhookHook(slack_webhook_conn_id='slack_webhook')
    hook.send(text=message)

@dag(
    default_args={'on_failure_callback': slack_failure_alert},
    ...
)
def my_pipeline():
    ...

For critical tasks — the ones that handle payments, feed customer-facing reports, or trigger downstream systems — consider adding a PagerDuty integration that creates an incident and pages the on-call engineer directly. Not every task failure warrants waking someone up, but some absolutely do.

SLA Monitoring

Sometimes a task does not fail — it just takes too long. SLAs (Service Level Agreements) let you define the maximum acceptable duration for a task. When a task exceeds its SLA, Airflow triggers a callback so you can investigate before it becomes a real problem.

from datetime import timedelta

@task(sla=timedelta(hours=2))
def long_running_task():
    """Must complete within 2 hours of the DAG run's execution_date."""
    ...

SLA misses are particularly valuable for catching performance degradation early. If a task that normally takes 20 minutes starts taking 90 minutes, an SLA miss notification gives you a heads-up before it grows to 4 hours and starts causing cascading delays.

Testing: The Practice Most Teams Skip (And Regret)

Most Airflow teams do not test their DAGs. The reasoning usually goes something like this: “Our DAGs are just configuration — they call operators that are already tested by the Airflow community.” This is true until it is not. A typo in a connection ID, a missing dependency, a DAG that imports fine on your laptop but fails on the server because of a missing Python package — these are all things that a simple test would catch before they reach production.

Start with DAG Import Tests

The simplest and highest-value test is the import test. It loads every DAG file and checks that none of them produce import errors. This catches syntax errors, missing modules, and broken imports before they reach your scheduler.

import pytest
from airflow.models import DagBag

@pytest.fixture
def dag_bag():
    return DagBag(dag_folder='dags/', include_examples=False)

def test_no_import_errors(dag_bag):
    assert len(dag_bag.import_errors) == 0, (
        f"DAG import errors: {dag_bag.import_errors}"
    )

This single test, running in CI on every pull request, prevents a surprising number of production incidents. If a developer accidentally introduces a syntax error or removes a module that a DAG depends on, the test fails and the PR cannot be merged.

Add Structure Tests for Critical DAGs

For your most important DAGs, add tests that verify the structure — that expected tasks exist, that dependencies are wired correctly, and that critical configuration like retries and owners are set.

def test_pipeline_structure():
    dag_bag = DagBag(dag_folder='dags/', include_examples=False)
    dag = dag_bag.get_dag('production_etl')

    task_ids = [t.task_id for t in dag.tasks]
    assert 'extract' in task_ids
    assert 'transform' in task_ids
    assert 'load' in task_ids

    # Verify dependencies
    extract = dag.get_task('extract')
    transform = dag.get_task('transform')
    assert transform.task_id in [t.task_id for t in extract.downstream_list]

Unit Test Your Business Logic Separately

The most testable DAG code is code where the business logic lives in plain Python functions that have no Airflow dependencies. Extract your transformation logic, validation rules, and data processing functions into utility modules, and test them with standard pytest. This is faster, simpler, and more reliable than trying to test tasks within the Airflow framework.

# dags/utils/transformations.py
def normalize_email(email: str) -> str:
    return email.strip().lower()

# tests/test_transformations.py
def test_normalize_email():
    assert normalize_email('  User@Example.COM  ') == 'user@example.com'

Deployment: Getting DAGs to Production Safely

Git-Sync Sidecar (Kubernetes)

The most common approach for Kubernetes-based Airflow deployments is a git-sync sidecar container that continuously pulls DAG files from a Git repository. This means your deployment process is simply “merge to main” — the sidecar picks up changes automatically within a minute or so.

# values.yaml for Helm chart
dags:
  gitSync:
    enabled: true
    repo: git@github.com:company/airflow-dags.git
    branch: main
    subPath: "dags"
    wait: 60

Docker Image Bake

An alternative is bundling DAGs directly into your Airflow Docker image at build time. This gives you immutable deployments — every version of your image contains a specific, known set of DAGs. Rolling back is as simple as deploying the previous image version.

FROM apache/airflow:2.9.0-python3.11
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY dags/ /opt/airflow/dags/

CI/CD: Automating the Safety Net

Combine your DAG tests with a CI/CD pipeline so that every change goes through import tests, structure tests, and linting before it can reach production. This is the single most effective practice for preventing “it worked on my machine” incidents.

# .github/workflows/airflow-ci.yml
name: Airflow CI
on:
  pull_request:
    paths: ['dags/**', 'tests/**']

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install apache-airflow==2.9.0 pytest
      - run: pytest tests/ -v
      - run: ruff check dags/

Airflow vs Alternatives: A Thoughtful Comparison

Airflow is not the only workflow orchestrator, and it is not the right choice for every situation. Two serious alternatives have matured in recent years — Prefect and Dagster — and understanding where each shines will help you make the right decision for your team.

Airflow is the incumbent. It has been around since 2014, has the largest community, the most provider packages (integrations with external systems), and the most battle scars. If you need a scheduler that can orchestrate complex dependencies across dozens of different systems, Airflow’s ecosystem is unmatched. Its weakness is developer experience: the gap between writing a DAG locally and running it in production can be steep, testing requires manual setup, and the programming model is more ceremony-heavy than its competitors.

Prefect (2018) took the “make orchestration feel like writing Python” approach. Your workflows are plain Python functions with decorators. There is less boilerplate, a more intuitive mental model, and strong support for dynamic and event-driven workflows. The trade-off is a smaller ecosystem of integrations compared to Airflow, though it is growing fast. If your team values developer velocity and writes a lot of ML or data science pipelines, Prefect is worth serious evaluation.

Dagster (2019) reimagined orchestration around the concept of “software-defined assets.” Instead of thinking about tasks and schedules, you define the data assets your pipeline produces and Dagster figures out what needs to run to produce them. This asset-centric model makes lineage, testing, and local development first-class experiences. It is particularly compelling for data platform teams who think in terms of data products rather than task schedules. The trade-off is a steeper initial learning curve if you are used to Airflow’s task-centric model.

Choose Airflow when you need a battle-tested orchestrator with the largest ecosystem, your primary need is scheduling and dependency management across many systems, or you are already invested in the Airflow ecosystem through a managed service like MWAA or Cloud Composer.

Consider Prefect when you want a more Pythonic experience with less boilerplate, your workflows are heavily dynamic or event-driven, or your team prioritizes developer experience and fast iteration.

Consider Dagster when your workflows are data-asset-centric and you want built-in lineage tracking, you need first-class testing and local development, or your team thinks in terms of data products rather than task execution schedules.

There is no objectively “best” orchestrator. The right choice depends on your team’s existing expertise, the complexity of your integrations, and whether you think about your work as “running tasks on a schedule” or “producing and maintaining data assets.”

Pre-Deployment Checklist

Before pushing a DAG to production, run through this checklist. Every item addresses a specific failure mode that has burned real teams:

• start_date is a fixed datetime, not a dynamic expression
• catchup=False unless you specifically intend to backfill
• All tasks are idempotent and safe to retry
• No Variable.get() or connection lookups at the module level
• Heavy imports are inside task functions, not at the top of the file
• Failure callbacks are configured for alerting
• Tasks have retries and retry_delay set
• Pools are configured for resource-constrained operations
• DAG import tests pass in CI
• Task business logic has unit tests

Next Steps

Production Airflow is less about writing clever DAGs and more about operational discipline. Start with the fundamentals — fixed start dates, no module-level queries, failure alerts, idempotent tasks — and layer in sophistication as your needs grow.

• What is Apache Airflow? — If any foundational concepts in this guide felt unfamiliar, revisit the architecture overview.
• Connections and Variables — Deep dive into managing configuration and credentials properly.
• Set up DAG import tests in CI. If you do only one thing from this guide, make it this. It takes 15 minutes to set up and prevents a disproportionate number of production incidents.
• Audit your existing DAGs against the pre-deployment checklist above. Most teams find at least a few DAGs with module-level variable lookups or missing retry configuration.