Dynamic DAGs in Airflow: Patterns and Best Practices
Master dynamic DAG generation in Airflow using DAG factories, dynamic task mapping, YAML configs, expand/reduce, and avoid common pitfalls.
·9 min read · #airflow#dynamic-dags#task-mapping
#airflow
19 posts · page 1 of 1
Building Custom Operators in Apache Airflow
Learn to build custom Airflow operators from scratch with BaseOperator, hooks, templated fields, testing strategies, and packaging for reuse.
·8 min read · #airflow#custom-operators#hooks
Airflow Executors Deep Dive: From Local to Kubernetes
How Airflow executors work under the hood. SequentialExecutor, LocalExecutor, CeleryExecutor, and KubernetesExecutor compared with migration paths.
·16 min read · #airflow#executors#celery
Data-Aware Scheduling in Airflow with Datasets
Replace sensor-based waiting with Airflow Datasets. Build producer-consumer DAGs, combine time and data triggers, and design dataset URIs for production.
·12 min read · #airflow#datasets#scheduling
Real-World Airflow Patterns for Production Pipelines
Idempotent pipelines, backfilling, late data handling, error patterns, multi-environment setups, and common anti-patterns to avoid in Airflow.
·15 min read · #airflow#patterns#production
Deploying Apache Airflow to Production
Run Airflow in production with Docker Compose, Helm on Kubernetes, or managed services. Covers monitoring, logging, security, and database backends.
·12 min read · #airflow#deployment#docker
Testing Airflow DAGs: A Comprehensive Guide
Unit test tasks, validate DAG structure, mock external services, and build CI/CD pipelines for Airflow. Prevent silent data loss with systematic testing.
·13 min read · #airflow#testing#pytest
CI/CD for Data Pipelines — Ship Data with Confidence
Build CI/CD workflows for data pipelines: lint SQL, validate DAGs, run tests, deploy dbt models, and manage dev/staging/prod environments.
·10 min read · #data-engineering#cicd#devops
Airflow Architecture Explained: Components and Executors
Understand the architecture of Apache Airflow -- Scheduler, Web Server, Metadata Database, Executors, and Workers -- and learn which executor fits your workload.
·17 min read · #airflow#architecture#intermediate
Apache Airflow Best Practices for Production
Production-ready Airflow patterns covering DAG design, performance optimization, monitoring, testing, deployment strategies, and comparison with alternatives.
·16 min read · #airflow#best-practices#production
Airflow Connections and Variables: Managing Configuration
Master Airflow connections for external system credentials and variables for runtime configuration, including secrets backends for production deployments.
·11 min read · #airflow#connections#configuration
Branching and Conditional Logic in Apache Airflow
Learn how to implement conditional workflows in Airflow using BranchPythonOperator, ShortCircuitOperator, trigger rules, and the TaskFlow branch decorator.
·12 min read · #airflow#branching#intermediate
Airflow DAGs Explained: Structure, Scheduling, and Dependencies
Master Airflow DAGs -- learn the anatomy of a DAG file, scheduling with cron and presets, defining task dependencies, fan-out patterns, and task lifecycle states.
·19 min read · #airflow#dags#beginners
Airflow Operators Guide: Built-in, Provider, and Custom
Master Apache Airflow operators from BashOperator to building your own custom operators with template fields and provider packages.
·12 min read · #airflow#operators#intermediate
Airflow Sensors: Waiting for Conditions in Your Pipelines
Learn how Airflow sensors pause tasks until conditions are met, including poke vs reschedule modes, custom sensors, and deferrable operators.
·11 min read · #airflow#sensors#intermediate
Airflow XComs: Passing Data Between Tasks
Understand Airflow XComs for inter-task communication including push/pull patterns, Jinja templates, size limits, and custom backends for production use.
·11 min read · #airflow#xcoms#intermediate
Airflow TaskFlow API: The Modern Way to Write DAGs
Master the TaskFlow API with @dag and @task decorators, implicit XCom passing, dynamic task mapping, and dataset-aware scheduling in Airflow 2.0+.
·11 min read · #airflow#taskflow#intermediate
Installing Apache Airflow: pip and Docker Compose Methods
Step-by-step guide to installing Apache Airflow using pip with constraints or Docker Compose, creating an admin user, and verifying your setup works correctly.
·15 min read · #airflow#setup#beginners
What Is Apache Airflow? A Complete Introduction
Learn what Apache Airflow is, why Airbnb created it, how DAGs work, and when to use Airflow for orchestrating data pipelines, ETL workflows, and ML operations.
·14 min read · #airflow#beginners#data-engineering