Airflow Guides
NEW
Datasets and Data-Aware Scheduling in Airflow
Using datasets to implement DAG dependencies and scheduling in Airflow.
Soda Core and Airflow
Using Soda Core to implement data quality checks in Airflow DAGs.
Data quality and Airflow
Check the quality of your data using Airflow.
Orchestrating Snowflake Queries with Airflow
How to use Airflow to get enhanced observability and compute savings while orchestrating your Snowflake jobs.
Dynamic Tasks in Airflow
How to dynamically create tasks at runtime in your Airflow DAGs.
OpenLineage and Airflow
Using OpenLineage and Marquez to get lineage data from your Airflow DAGs.
Orchestrating Redshift Operations from Airflow
Setting up a connection to Redshift and using available Redshift modules.
The Astro Python SDK for ETL
Using the Astro Python SDK to implement ETL use cases in Airflow.
Introduction to Airflow Decorators
An overview of Airflow decorators and how they can improve the DAG authoring experience.
Deferrable Operators
How to implement deferrable operators to save cost and resources with Airflow.
Debugging DAGs
A beginner's guide to figuring out what's going wrong with your Airflow DAGs
Rerunning Airflow DAGs
How to use catchup, backfill, and cleared task instances in Airflow.
Scheduling and Timetables in Airflow
Everything you need to know about scheduling your Airflow DAGs.
Airflow Data Quality Checks with SQL Operators
Executing queries in Apache Airflow DAGs to ensure data quality.
Airflow Pools
Using pools to control task parallelism in Airflow.
Integrating Airflow and dbt
Running dbt models in your Airflow DAGs.
Using Airflow with SageMaker
Methods for orchestrating SageMaker machine learning pipelines with Airflow.
Executing Notebooks with Airflow
Methods for orchestrating commonly used notebooks with Airflow.
Cross-DAG Dependencies
How to implement dependencies between your Airflow DAGs.
Testing Airflow DAGs
How to apply test-driven development practices to your Airflow DAGs.
Using Task Groups in Airflow
Using Task Groups to build modular workflows in Airflow.
Custom XCom Backends
Creating a custom XCom backend with Airflow 2.0.
Passing Data Between Airflow Tasks
Methods for sharing metadata and information between tasks in your Apache Airflow DAGs, including XCom.
Deploying Kedro Pipelines to Apache Airflow
How to use the kedro-airflow plugin to change your Kedro pipelines into Apache Airflow DAGs and deploy them to a production environment.
Orchestrating Databricks Jobs with Airflow
Orchestrating Databricks Jobs from your Apache Airflow DAGs.
Executing Azure Data Factory Pipelines with Airflow
Triggering remote jobs in Azure Data Factory from your Apache Airflow DAGs.
Executing Azure Data Explorer Queries with Airflow
Executing Azure Data Explorer queries from your Apache Airflow DAGs.
Orchestrating Azure Container Instances with Airflow
Orchestrating containers with Azure Container Instances from your Apache Airflow DAGs.
Get Started with Apache Airflow 2.0
Test Apache Airflow 2.0 on your local machine with the Astro CLI.
Using Airflow to Execute SQL
Executing queries, parameterizing queries, and embedding SQL-driven ETL in Apache Airflow DAGs.
Integrating Airflow and Great Expectations
Using the Great Expectations provider natively in your Airflow DAGs.
Understanding the Airflow Metadata Database
An structural walkthrough of Apache Airflow's metadata database, with a full ERD.
Executing Talend Jobs with Airflow
Triggering remote jobs in Talend from your Apache Airflow DAGs.
Integrating Airflow and Hashicorp Vault
Pull connection information from your Hashicorp Vault to use in your Airflow DAGs.
Importing Custom Hooks & Operators
How to correctly import custom hooks and operators.
Scaling Out Airflow
How to tune your Airflow environment so it scales with your DAGs.
Airflow Executors Explained
A thorough breakdown of Apache Airflow's Executors: Celery, Local and Kubernetes.
Logging in Airflow
Demystifying Airflow's logging configuration.
Best Practices Calling AWS Lambda from Airflow
A few tips, guidelines, and best practices for calling Lambda from Airflow
Using the KubernetesPodOperator
Use the KubernetesPodOperator in Airflow to run tasks in Kubernetes Pods
Using the BashOperator
Use the BashOperator in Airflow to execute bash commands and scripts
Using SubDAGs in Airflow
Using SubDAGs to build modular workflows in Airflow.
Templating in Airflow
How to leverage the power of Jinja templating when writing your DAGs.
Branching in Airflow
Use Apache Airflow's BranchPythonOperator and ShortCircuitOperator to execute conditional branches in your workflow
Airflow's Components
Learn about the core components of Apache Airflow's infrastructure.
Running Airflow on Windows 10 & WSL
How to spin up Airflow on your Windows system.
The Airflow UI
A high-level overview of the Airflow UI
DAG Writing Best Practices in Apache Airflow
How to create effective, clean, and functional DAGs.
Managing your Connections in Apache Airflow
An overview of how connections work in the Airflow UI.
Introduction to Airflow DAGs
How to write your first DAG in Apache Airflow
Error Notifications in Airflow
Methods for managing notifications in your Airflow DAGs.
Dynamically Generating DAGs in Airflow
Using a base DAG template to create multiple DAGs.
Introduction to Apache Airflow
Everything you need to know to get started with Apache Airflow.
Managing Airflow Code
Guidelines for Working with Multiple Airflow Projects
Managing Dependencies in Apache Airflow
An overview of dependencies and triggers in Airflow.
Using Apache Airflow Plugins
A crash-course in using Airflow Plugins.
Hooks 101
An introduction to Hooks in Apache Airflow.
Sensors 101
An introduction to Sensors in Apache Airflow.
Operators 101
An introduction to Operators in Apache Airflow.