ShortCircuitOperator
Apache AirflowAllows a pipeline to continue based on the result of a python_callable.
Access Instructions
Install the Apache Airflow provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
Documentation
Allows a pipeline to continue based on the result of a python_callable.
The ShortCircuitOperator is derived from the PythonOperator and evaluates the result of a python_callable. If the returned result is False or a falsy value, the pipeline will be short-circuited. Downstream tasks will be marked with a state of “skipped” based on the short-circuiting mode configured. If the returned result is True or a truthy value, downstream tasks proceed as normal and an XCom of the returned result is pushed.
The short-circuiting can be configured to either respect or ignore the trigger_rule set for downstream tasks. If ignore_downstream_trigger_rules is set to True, the default setting, all downstream tasks are skipped without considering the trigger_rule defined for tasks. However, if this parameter is set to False, the direct downstream tasks are skipped but the specified trigger_rule for other subsequent downstream tasks are respected. In this mode, the operator assumes the direct downstream tasks were purposely meant to be skipped but perhaps not other subsequent tasks.
See also
For more information on how to use this operator, take a look at the guide: ShortCircuitOperator
Example DAGs
Example DAG demonstrating the usage of the ShortCircuitOperator.
This DAG demonstrates orchestrating multiple Azure Data Factory (ADF) pipelines using Airflow to perform classic ELT operations.
This example showcases an example of adding an operational check to ensure the dbt Cloud job is not running prior to triggering.
Demonstrates orchestrating ML model serving pipelines executed on Databricks with Airflow.