Installing Apache Airflow: pip and Docker Compose Methods

Choosing Your Installation Method

Before you type a single command, it is worth understanding the two main approaches and why you would pick one over the other. This choice will affect your daily development experience, so take a moment to consider which fits your situation.

The pip install method installs Airflow directly on your machine, just like any other Python package. It is fast to set up, lightweight, and perfect for learning. The tradeoff is that it runs in a simplified mode — SQLite database, SequentialExecutor, everything in one process. This is great for writing and testing DAGs, but it does not resemble a production environment. Think of it as learning to cook in your home kitchen: you can learn all the techniques, but it is different from running a professional restaurant.

The Docker Compose method spins up multiple containers that mimic a real Airflow deployment: a PostgreSQL database, a Redis message broker, separate containers for the Scheduler, Web Server, Workers, and Triggerer. This is more complex to set up, but the environment matches what you would see in production. Think of it as a practice kitchen that is laid out exactly like the restaurant — you learn the techniques AND the operational reality at the same time.

Here is a simple decision guide:

Method	Best for	Complexity	Production-like?
pip install	Learning, lightweight dev	Low	No (single process)
Docker Compose	Team development, testing	Medium	Yes (multi-container)

If you are brand new to Airflow and just want to see how it works, start with pip. If you are setting up a development environment for a team or want to test DAGs in conditions similar to production, go with Docker Compose. You can always switch later.

Method 1: pip Install with Constraints

Why Constraints Are Non-Negotiable

Airflow is a large project with hundreds of Python dependencies. If you run a plain pip install apache-airflow, pip will try to resolve all those dependencies on its own, and it frequently gets it wrong. You might end up with a combination of package versions that installs successfully but crashes at runtime with obscure errors.

To prevent this, the Airflow project publishes “constraint files” for every release. A constraint file is a list of exact package versions that have been tested together and are known to work. When you install with constraints, pip is forced to use those exact versions instead of guessing.

The short version: never install Airflow without a constraint file. This is the single most common source of installation headaches, and it is entirely avoidable.

Step 1: Create a Virtual Environment

A virtual environment isolates your Airflow installation from your system Python and other projects. Without one, Airflow’s hundreds of dependencies can conflict with packages you need for other work.

The commands below create a project directory, set up a virtual environment inside it, and activate it. Once activated, any Python packages you install will go into this isolated environment rather than your system-wide Python.

mkdir ~/airflow-project && cd ~/airflow-project
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip

Step 2: Set the Airflow Home Directory

Airflow needs a directory to store its configuration file, database, logs, and DAG files. By default, it uses ~/airflow, but it is better to be explicit. The AIRFLOW_HOME environment variable tells Airflow where to put everything.

Setting this explicitly keeps your Airflow files organized and separate from other projects. You will need to set this variable every time you open a new terminal session, so consider adding it to your shell profile (.bashrc or .zshrc).

export AIRFLOW_HOME=~/airflow-project/airflow_home

Step 3: Install Airflow with Constraints

The installation command looks long, but each piece has a purpose. The first two lines detect your Airflow version and Python version so the correct constraint file is downloaded. The third line constructs the URL to the constraint file hosted on GitHub. The fourth line runs the actual installation.

AIRFLOW_VERSION=2.9.3
PYTHON_VERSION="$(python3 --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

This installs only the core Airflow package. If your DAGs need to interact with external systems like PostgreSQL, AWS, or HTTP APIs, you install those providers separately. Each provider is its own pip package:

pip install "apache-airflow-providers-postgres" --constraint "${CONSTRAINT_URL}"
pip install "apache-airflow-providers-http" --constraint "${CONSTRAINT_URL}"

You do not need to install every provider up front. Add them as you need them.

Step 4: Initialize the Database

Airflow stores all of its state in a database. The db init command creates the database schema (tables for DAGs, task instances, users, etc.) and generates the default configuration file.

By default, Airflow uses SQLite, which is a file-based database stored in a single .db file. SQLite is fine for learning and local development, but it does not support concurrent access, which means you cannot run the Scheduler and Web Server as separate processes simultaneously. For anything beyond solo learning, you will want to upgrade to PostgreSQL (covered later in this article).

airflow db init

After this command completes, you will find several new items in your AIRFLOW_HOME directory: airflow.cfg (the main configuration file), airflow.db (the SQLite database), a logs/ directory (where task execution logs are stored), and a dags/ directory (where you will put your DAG files).

Step 5: Create an Admin User

Airflow’s web interface requires authentication. This command creates a user account with the Admin role, which has full access to everything in the UI.

airflow users create \
    --username admin \
    --firstname Admin \
    --lastname User \
    --role Admin \
    --email admin@example.com \
    --password admin

In a real environment you would use a strong password. For local learning, admin/admin is fine.

Step 6: Start Airflow Services

Airflow needs at least two processes running: the Web Server (which serves the UI) and the Scheduler (which triggers DAG runs and monitors tasks). In a pip installation, you start these in separate terminal windows.

This is important to understand: the Web Server and Scheduler are independent processes that communicate through the database, not through each other. If you stop the Web Server, your DAGs keep running. If you stop the Scheduler, the UI stays up but no new tasks get triggered.

Open two terminal windows. In the first, start the Web Server:

export AIRFLOW_HOME=~/airflow-project/airflow_home
source ~/airflow-project/venv/bin/activate
airflow webserver --port 8080

In the second, start the Scheduler:

export AIRFLOW_HOME=~/airflow-project/airflow_home
source ~/airflow-project/venv/bin/activate
airflow scheduler

If managing two terminals feels cumbersome, Airflow provides a shortcut that runs everything in a single process. This is great for quick testing but should not be used for any serious work:

airflow standalone

The standalone command automatically creates an admin user and prints the credentials to the terminal, so you can skip the “create admin user” step if you use this approach.

Step 7: Verify the Installation

Open your browser to http://localhost:8080. You should see the Airflow login page. Sign in with the credentials you created (or the ones printed by standalone), and you will see a list of example DAGs.

If the page does not load, check that both the Web Server and Scheduler are running without errors in their respective terminal windows. The most common issue is a port conflict — if another application is already using port 8080, you will see an error. Either stop the other application or change Airflow’s port by adding --port 9090 to the webserver command.

Method 2: Docker Compose (Recommended for Teams)

Docker Compose gives you a production-like setup with PostgreSQL, Redis, and separate containers for each Airflow component. This means you get parallel task execution, proper CeleryExecutor workers, and an environment that behaves like what you would see in staging or production.

Step 1: Download the Official Compose File

The Airflow project maintains an official docker-compose.yaml that defines all the containers you need. This file is well-tested and regularly updated.

mkdir ~/airflow-docker && cd ~/airflow-docker
curl -LfO "https://airflow.apache.org/docs/apache-airflow/2.9.3/docker-compose.yaml"

Step 2: Create Required Directories

Airflow’s Docker setup mounts local directories into the containers so that your DAG files, logs, and plugins are accessible. The .env file sets your Linux user ID, which prevents permission issues where containers create files that your host user cannot read or modify.

mkdir -p ./dags ./logs ./plugins ./config
echo -e "AIRFLOW_UID=$(id -u)" > .env

If you are on macOS, the AIRFLOW_UID step is less critical (Docker Desktop handles permissions differently), but it does no harm and keeps your setup portable.

Step 3: Understand What You Are Starting

Before running docker compose up, it helps to know what services will spin up. The official compose file defines seven services, and understanding their roles will make troubleshooting much easier:

Service	What it does	Why it exists
postgres	Metadata database	Stores all Airflow state — DAG runs, task status, connections
redis	Message broker	Passes tasks from the Scheduler to Celery workers
airflow-webserver	Serves the UI on port 8080	So you can monitor and manage your DAGs
airflow-scheduler	Parses DAGs, triggers runs, queues tasks	The brain that decides what to run and when
airflow-worker	Executes tasks received from Redis	The hands that do the actual work
airflow-triggerer	Handles deferred/async tasks	Efficient waiting without wasting worker slots
airflow-init	One-time setup	Runs database migrations and creates the admin user

The airflow-init service runs once and exits. All others stay running. This is a CeleryExecutor setup, which means you have true distributed task execution even in your local development environment.

Step 4: Initialize and Start

The initialization step runs database migrations (creating all the necessary tables in PostgreSQL) and creates a default admin user with the credentials airflow / airflow.

docker compose up airflow-init

Once initialization completes successfully, start all services in detached mode (running in the background):

docker compose up -d

Wait about 30-60 seconds for all services to become healthy. You can check the status with docker compose ps — look for all services showing “healthy” in the status column. Then open http://localhost:8080 and log in with airflow / airflow.

If a service shows “unhealthy” or keeps restarting, check its logs with docker compose logs <service-name>. The most common issues are insufficient Docker resources (Airflow needs at least 4GB of RAM allocated to Docker) and port conflicts on 8080.

Adding Python Dependencies

Your DAGs will almost certainly need Python packages that are not included in the base Airflow image. There are two approaches, and which one you choose depends on your workflow.

For quick development, the _PIP_ADDITIONAL_REQUIREMENTS environment variable installs packages when the container starts. This is convenient but slow — packages are re-installed every time a container restarts.

For anything beyond quick experiments, build a custom Docker image. Create a requirements.txt listing your packages and a Dockerfile that installs them into the Airflow image. This way, packages are baked into the image and available instantly on startup.

FROM apache/airflow:2.9.3
COPY requirements.txt /
RUN pip install --no-cache-dir -r /requirements.txt

Then update your docker-compose.yaml to use your custom image instead of the default one.

Essential Docker Compose Commands

You will use these commands daily when working with the Docker Compose setup:

# View logs for a specific service (follow mode)
docker compose logs airflow-scheduler -f

# Open a shell inside a container (useful for debugging)
docker compose exec airflow-worker bash

# Run Airflow CLI commands inside the container
docker compose exec airflow-worker airflow dags list

# Stop all services (preserves data)
docker compose down

# Full reset -- stop everything and delete all data
docker compose down --volumes --remove-orphans

The last command is your “nuclear option” — it deletes the database, all logs, and all state. Use it when you want a completely fresh start.

Configuration: How to Customize Airflow’s Behavior

Airflow’s configuration lives in airflow.cfg, but in modern deployments (especially Docker), the preferred approach is environment variables. Every setting in airflow.cfg can be overridden with an environment variable using this pattern:

AIRFLOW__<SECTION>__<KEY>=<VALUE>

The double underscores separate the prefix (AIRFLOW), the section name, and the key name. For example, to set the executor to LocalExecutor, you would use AIRFLOW__CORE__EXECUTOR=LocalExecutor.

Environment variables take precedence over airflow.cfg, which makes them ideal for Docker and Kubernetes deployments where you do not want to mount or modify configuration files, need to keep secrets out of version control, and want to easily change settings between environments (dev, staging, production).

Here are the settings you are most likely to change:

Load examples (AIRFLOW__CORE__LOAD_EXAMPLES=False) — The default Airflow installation comes with dozens of example DAGs. They are helpful for learning at first, but quickly become clutter. Disable them once you are comfortable with the basics.

Executor (AIRFLOW__CORE__EXECUTOR=LocalExecutor) — Controls how tasks are executed. See the Architecture article for a full comparison of executor types.

Parallelism (AIRFLOW__CORE__PARALLELISM=32) — The maximum number of tasks that can run simultaneously. Start with the default and increase it if you find tasks sitting in the “queued” state for too long.

Database connection (AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://...) — Points to your metadata database. For the pip installation, you will change this when upgrading from SQLite to PostgreSQL.

Verifying Everything Works

Regardless of which installation method you chose, you should verify that the full pipeline — from DAG file to task execution to log viewing — works end to end.

Create a simple test DAG. This DAG has no automatic schedule (it only runs when you trigger it manually) and contains two tasks: one that prints a message using Python, and one that prints environment information using bash. The purpose is not the tasks themselves but confirming that Airflow can find the DAG, execute tasks, and display logs.

Place this file at $AIRFLOW_HOME/dags/test_dag.py (pip method) or ./dags/test_dag.py (Docker method):

from datetime import datetime
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator

with DAG(
    dag_id="test_installation",
    start_date=datetime(2026, 1, 1),
    schedule=None,
    catchup=False,
    tags=["test"],
) as dag:

    hello = PythonOperator(
        task_id="say_hello",
        python_callable=lambda: print("Hello from Airflow!"),
    )

    check_env = BashOperator(
        task_id="check_environment",
        bash_command="python3 --version && airflow version",
    )

    hello >> check_env

After saving the file, wait a minute or two for the Scheduler to discover it (it scans the DAGs folder periodically). Then go to the UI, find “test_installation” in the DAG list, toggle it ON, and click the play button to trigger a run. Watch both tasks turn green, and click into each one to verify you can see the logs.

If the DAG does not appear after a few minutes, check that the file is in the correct directory and that it has no Python syntax errors. You can test for syntax errors by running python test_dag.py directly — if it produces no output, the syntax is fine.

Upgrading from SQLite to PostgreSQL

If you started with the pip method and want parallel task execution, you need to switch from SQLite to PostgreSQL. SQLite only supports one connection at a time, which forces the SequentialExecutor. PostgreSQL supports many concurrent connections, unlocking the LocalExecutor and CeleryExecutor.

You will need PostgreSQL running locally (install it via your system’s package manager or Homebrew on macOS). Then install the Python driver and update your configuration:

pip install psycopg2-binary
export AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@localhost:5432/airflow
export AIRFLOW__CORE__EXECUTOR=LocalExecutor

airflow db reset   # This destroys all existing data
airflow db init

The db reset command is destructive — it drops all tables and data. This is fine for a development setup, but in a real environment you would migrate rather than reset.

Common Issues and How to Fix Them

“ModuleNotFoundError” when starting Airflow — You likely forgot to activate your virtual environment. Run source venv/bin/activate before any Airflow command.

Port 8080 is already in use — Another application is using that port. Either stop it or start Airflow on a different port: airflow webserver --port 9090.

DAG does not appear in the UI — The Scheduler only checks for new DAG files every 5 minutes by default (dag_dir_list_interval). Wait a few minutes, or restart the Scheduler to force an immediate scan. Also check for Python import errors in your DAG file by running it directly with python your_dag.py.

“Database is locked” (SQLite) — SQLite cannot handle multiple connections. If you are running the Web Server and Scheduler as separate processes, switch to PostgreSQL.

Permission denied in Docker — Your containers are creating files as a different user than your host system user. Make sure your .env file contains the correct AIRFLOW_UID=$(id -u).

Scheduler is not picking up tasks — Verify the Scheduler process is actually running (docker compose ps or check the terminal window). Check the Scheduler logs for errors. A common cause is the database connection being misconfigured.

Docker containers keep restarting — Airflow needs at least 4 GB of RAM allocated to Docker. On macOS and Windows, Docker Desktop has a default memory limit that might be too low. Increase it in Docker Desktop Settings under Resources.

Next Steps

With Airflow installed and running, you are ready to write real DAGs:

• DAGs Explained — Deep dive into DAG structure, scheduling, and task dependencies.
• Architecture Explained — Understand why the components you just installed work the way they do.