Airflow Connections and Variables: Managing Configuration

The Problem: Hardcoded Credentials Are a Ticking Time Bomb

Imagine this scenario. A data engineer writes a DAG that connects to a PostgreSQL database. They put the hostname, username, and password right in the Python file because it works, it is fast, and the deadline is tomorrow. Three months later, the database password is rotated. The engineer has since moved to another team. Nobody remembers which DAG files contain the old credentials, and half the pipelines break at 2 AM on a Saturday.

This is not a hypothetical. It happens constantly. Hardcoding credentials in DAG files creates three serious problems. First, it is a security risk — anyone with access to the code repository can see your database passwords, API keys, and cloud credentials. Second, it is fragile — when credentials change, you have to hunt through every DAG file to update them. Third, it makes it impossible to run the same DAG in different environments (development, staging, production) without maintaining separate copies of the code.

Airflow solves this problem cleanly with two built-in mechanisms: connections and variables. They serve different purposes, and understanding the distinction is essential for building pipelines that are secure, flexible, and easy to maintain.

Connections: Your Address Book for External Systems

A connection in Airflow is like an entry in an address book. When you want to call someone, you do not memorize their phone number — you look them up by name. Similarly, when your DAG needs to talk to a PostgreSQL database, an S3 bucket, or a REST API, it does not hardcode the credentials. Instead, it says “use the connection called my_postgres” and Airflow looks up all the details — hostname, port, username, password, and any extra parameters.

Every connection has a unique identifier called the conn_id. This is the name you reference in your DAG code. Beyond the ID, a connection stores the connection type (postgres, aws, http, and so on), the hostname, login credentials, port, database schema, and an extra field for any additional JSON-formatted parameters the system needs.

The beauty of this design is separation of concerns. Your DAG code says what to do and which connection to use. The connection itself, stored separately, says how to authenticate. You can change the password, switch to a different database server, or move from development to production credentials without touching a single line of DAG code.

Creating Connections

There are three ways to create connections, each suited to different workflows.

Through the Airflow UI is the simplest approach for getting started. Navigate to Admin, then Connections, and click the plus button to add a new one. The UI shows type-specific forms, so when you select “Postgres” as the connection type, you get labeled fields for host, schema, login, and password. This is great for development and small teams.

Through the CLI is better for reproducibility and automation. You can script connection creation and include it in your deployment process.

airflow connections add 'my_postgres' \
    --conn-type 'postgres' \
    --conn-host 'db.example.com' \
    --conn-login 'etl_user' \
    --conn-password 'secret123' \
    --conn-port 5432 \
    --conn-schema 'analytics'

Through environment variables is the most deployment-friendly approach, especially for containerized setups. Airflow reads any environment variable prefixed with AIRFLOW_CONN_ and treats it as a connection definition. The connection ID is derived from the variable name (lowercased, with the prefix removed).

Starting with Airflow 2.3, you can use a JSON format for environment variable connections, which is much easier to read and avoids the encoding headaches that come with URI-format passwords containing special characters.

export AIRFLOW_CONN_MY_POSTGRES='{
    "conn_type": "postgres",
    "host": "db.example.com",
    "login": "etl_user",
    "password": "secret123",
    "port": 5432,
    "schema": "analytics",
    "extra": {"sslmode": "require"}
}'

Using Connections in Your DAGs

Most Airflow operators accept a conn_id parameter, which makes using connections almost invisible. You just pass the connection name and the operator handles everything else — looking up the credentials, establishing the connection, running your query, and closing it cleanly.

from airflow.providers.postgres.operators.postgres import PostgresOperator

query_task = PostgresOperator(
    task_id='run_query',
    postgres_conn_id='my_postgres',
    sql='SELECT COUNT(*) FROM orders WHERE date = {{ ds }}',
)

For custom logic where you need direct access to the connection details, use hooks. A hook is Airflow’s abstraction for interacting with an external system. The BaseHook.get_connection() method gives you the raw connection object, while system-specific hooks like PostgresHook provide higher-level methods for common operations.

from airflow.providers.postgres.hooks.postgres import PostgresHook

@task
def query_database():
    hook = PostgresHook(postgres_conn_id='my_postgres')
    records = hook.get_records("SELECT id, name FROM users LIMIT 100")
    return records

The hook approach is particularly useful because it abstracts away connection management. You do not need to write connect(), cursor(), execute(), close() boilerplate. The hook handles retries, connection pooling, and cleanup.

Variables: Settings You Can Change Without Touching Code

If connections are your address book for external systems, variables are your settings panel. They store simple key-value configuration data — things like batch sizes, feature flags, file paths, email recipient lists, or API endpoint URLs that are not tied to credentials.

The key idea is flexibility. Suppose your ETL pipeline processes records in batches of 5,000. One day, the source system slows down and you need to reduce the batch size to 1,000 to avoid timeouts. If the batch size is hardcoded in your DAG file, you need to change the code, commit it, get it reviewed, deploy it, and hope you did not introduce a typo. If it is stored as a variable, you change one value in the Airflow UI and the next run picks it up automatically. No code change, no deployment, no risk.

You can create variables through the CLI, the UI, or directly in Python code:

# Set a simple value
airflow variables set 'etl_batch_size' '5000'

# Set a JSON value for structured configuration
airflow variables set 'pipeline_config' '{"batch_size": 5000, "timeout": 300}'

The Most Important Performance Rule: Parse Time vs Runtime

This is the single most critical thing to understand about Airflow variables, and getting it wrong can bring your entire Airflow installation to its knees.

Airflow’s scheduler continuously parses every DAG file to detect changes. By default, this happens every 30 seconds. When you call Variable.get() at the top level of a DAG file — outside of any task function — that database query runs on every single parse cycle. If you have 50 DAGs, each calling Variable.get() at the top level, that is 50 database queries every 30 seconds, or 100 queries per minute, just for parsing. Scale that up to a few hundred DAGs and you have a metadata database that is spending more time serving variable lookups than actually orchestrating tasks.

Here is what the wrong way looks like. The Variable.get() call sits at the module level, so it executes every time the scheduler processes this file:

from airflow.models import Variable

# BAD: This hits the database every 30 seconds during parsing
batch_size = Variable.get('etl_batch_size')

with DAG(...) as dag:
    task = PythonOperator(
        task_id='process',
        python_callable=process_data,
        op_args=[batch_size],
    )

There are two correct alternatives. The first is Jinja templates, which Airflow evaluates only at task execution time, not at parse time:

with DAG(...) as dag:
    task = BashOperator(
        task_id='process',
        bash_command='python process.py --batch-size {{ var.value.etl_batch_size }}',
    )

The second is calling Variable.get() inside a task function, which also only runs at execution time:

@task
def process_data():
    # This is fine -- only runs when the task actually executes
    batch_size = int(Variable.get('etl_batch_size', default_var='1000'))
    config = Variable.get('pipeline_config', deserialize_json=True)
    print(f"Processing with batch size: {batch_size}")

The rule is simple: never call Variable.get() or BaseHook.get_connection() at the top level of a DAG file. Always put these calls inside task functions or use Jinja templates.

Variables vs Connections: A Quick Decision Guide

The distinction is straightforward. If the configuration involves authenticating with an external system — database credentials, API keys, cloud provider tokens — use a connection. If it is a runtime setting that controls how your pipeline behaves — batch sizes, feature flags, file paths, email lists — use a variable.

Use Case	Mechanism
Database credentials	Connection
API keys with endpoint info	Connection
Cloud provider authentication	Connection
Feature flags	Variable
Batch sizes and thresholds	Variable
File paths	Variable
Email recipient lists	Variable

Secrets Backends: Enterprise-Grade Password Management

Storing connections and variables in Airflow’s metadata database works fine for development and small deployments. But in a production environment, especially one subject to compliance requirements, it falls short. The metadata database was not designed to be a secrets manager. It lacks audit trails (who accessed which credential and when), automatic rotation, fine-grained access control, and the kind of encryption-at-rest guarantees that security teams require.

Secrets backends are Airflow’s answer to this gap. Think of them as adapters that let Airflow read connections and variables from a dedicated, enterprise-grade secrets management system — tools like HashiCorp Vault, AWS Secrets Manager, or GCP Secret Manager. These tools were specifically built to store, rotate, and audit access to sensitive credentials.

The setup is straightforward. You configure a secrets backend in your airflow.cfg file, pointing it at your secrets management system. From that point on, when Airflow needs a connection or variable, it checks the secrets backend first, then falls back to environment variables, and finally checks the metadata database. Your DAG code does not change at all — it still references conn_id='my_postgres' and Airflow transparently looks it up from Vault or Secrets Manager instead of its own database.

Here is how you would configure AWS Secrets Manager as your backend:

# airflow.cfg
[secrets]
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {
    "connections_prefix": "airflow/connections",
    "variables_prefix": "airflow/variables",
    "region_name": "us-east-1"
}

And for HashiCorp Vault:

# airflow.cfg
[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {
    "connections_path": "connections",
    "variables_path": "variables",
    "mount_point": "airflow",
    "url": "https://vault.example.com:8200"
}

The lookup order is important to understand: secrets backend first, then environment variables, then the metadata database. This means you can override any database-stored connection by adding it to your secrets backend, without modifying anything in the Airflow UI. It also means you can start with database-stored connections during development and migrate to a secrets backend for production without changing any DAG code.

Security Best Practices

Managing credentials well is not just about using the right tools. It is about habits and discipline.

Never log connection details. Airflow automatically masks passwords in logs, but avoid printing entire connection objects or their string representations. A careless print(connection) can leak credentials to log files that are stored indefinitely and accessible to a wide audience.

Use secrets backends in production. The metadata database is acceptable for development, but production workloads need the audit trails, rotation capabilities, and access controls that dedicated secrets managers provide.

Rotate credentials regularly. Secrets backends make this much easier because you update the credential in one place (Vault, Secrets Manager) and every DAG picks up the new value on its next run. No code changes, no deployments.

Mark sensitive variables carefully. Variables are not designed to store secrets. If you find yourself putting passwords or API keys in variables, you almost certainly should be using a connection instead.

Next Steps

You now understand how to keep credentials out of your DAG files and how to build pipelines that can be configured without code changes. Here is where to go from here:

• Airflow Best Practices — Learn broader production guidelines, including parse-time performance tips that build directly on the Variable.get() lesson from this article.
• Operators Guide — Understand how operators use connections under the hood, and how hooks provide the glue between your code and external systems.
• Set up a secrets backend in a development environment. Even if your production infrastructure is not ready, getting familiar with Vault or AWS Secrets Manager locally will save you significant time when the migration happens.
• Audit your existing DAGs for top-level Variable.get() calls. If you have been working with Airflow for a while, there is a good chance some of your older DAGs have this anti-pattern. Fixing them is one of the highest-impact performance improvements you can make.