Skip to content
Codeloom

Topics / Apache Airflow

Apache Airflow

The industry-standard workflow orchestrator. DAGs, operators, sensors, and production scheduling.

Browse 18 tutorials → Other topics

Why learn Apache Airflow?

  • The most widely adopted orchestration tool for data pipelines.

  • Used by Airbnb, Spotify, Twitter, and thousands of companies for production ETL.

  • Massive community, rich operator library, and native cloud integrations.

  • A must-know for any data engineering role.

What you can build with Apache Airflow

ETL/ELT pipeline orchestration Scheduled batch data processing ML model training pipelines Cross-system data synchronization Monitoring and alerting workflows

Apache Airflow tutorials

18 articles · page 1 of 1

Hand-written tutorials, ordered as a recommended learning path.

  1. 01 What Is Airflow? Learn what Apache Airflow is, why Airbnb created it, how DAGs work, and when to use Airflow for orchestrating data pipelines, ETL workflows, and ML operations.
  2. 02 Architecture Understand the architecture of Apache Airflow -- Scheduler, Web Server, Metadata Database, Executors, and Workers -- and learn which executor fits your workload.
  3. 03 Installation & Setup Step-by-step guide to installing Apache Airflow using pip with constraints or Docker Compose, creating an admin user, and verifying your setup works correctly.
  4. 04 DAGs Explained Master Airflow DAGs -- learn the anatomy of a DAG file, scheduling with cron and presets, defining task dependencies, fan-out patterns, and task lifecycle states.
  5. 05 Operators Guide Master Apache Airflow operators from BashOperator to building your own custom operators with template fields and provider packages.
  6. 06 Sensors Learn how Airflow sensors pause tasks until conditions are met, including poke vs reschedule modes, custom sensors, and deferrable operators.
  7. 07 TaskFlow API Master the TaskFlow API with @dag and @task decorators, implicit XCom passing, dynamic task mapping, and dataset-aware scheduling in Airflow 2.0+.
  8. 08 XComs Understand Airflow XComs for inter-task communication including push/pull patterns, Jinja templates, size limits, and custom backends for production use.
  9. 09 Branching & Conditionals Learn how to implement conditional workflows in Airflow using BranchPythonOperator, ShortCircuitOperator, trigger rules, and the TaskFlow branch decorator.
  10. 10 Connections & Variables Master Airflow connections for external system credentials and variables for runtime configuration, including secrets backends for production deployments.
  11. 11 Best Practices Production-ready Airflow patterns covering DAG design, performance optimization, monitoring, testing, deployment strategies, and comparison with alternatives.
  12. 12 Testing DAGs Unit test tasks, validate DAG structure, mock external services, and build CI/CD pipelines for Airflow. Prevent silent data loss with systematic testing.
  13. 13 Executors Deep Dive How Airflow executors work under the hood. SequentialExecutor, LocalExecutor, CeleryExecutor, and KubernetesExecutor compared with migration paths.
  14. 14 Data-Aware Scheduling Replace sensor-based waiting with Airflow Datasets. Build producer-consumer DAGs, combine time and data triggers, and design dataset URIs for production.
  15. 15 Production Deployment Run Airflow in production with Docker Compose, Helm on Kubernetes, or managed services. Covers monitoring, logging, security, and database backends.
  16. 16 Real-World Patterns Idempotent pipelines, backfilling, late data handling, error patterns, multi-environment setups, and common anti-patterns to avoid in Airflow.
  17. 17 Dynamic DAGs Master dynamic DAG generation in Airflow using DAG factories, dynamic task mapping, YAML configs, expand/reduce, and avoid common pitfalls.
  18. 18 Custom Operators Learn to build custom Airflow operators from scratch with BaseOperator, hooks, templated fields, testing strategies, and packaging for reuse.