AzureDataFactoryRunPipelineOperatorAsync
Astronomer ProvidersExecutes a data factory pipeline asynchronously.
Access Instructions
Install the Astronomer Providers provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
azure_data_factory_conn_idThe connection identifier for connecting to Azure Data Factory.
pipeline_nameThe name of the pipeline to execute.
wait_for_terminationFlag to wait on a pipeline run’s termination. By default, this feature is enabled but could be disabled to perform an asynchronous wait for a long-running pipeline execution using the AzureDataFactoryPipelineRunSensor.
resource_group_nameThe resource group name. If a value is not passed in to the operator, the AzureDataFactoryHook will attempt to use the resource group name provided in the corresponding connection.
factory_nameThe data factory name. If a value is not passed in to the operator, the AzureDataFactoryHook will attempt to use the factory name name provided in the corresponding connection.
reference_pipeline_run_idThe pipeline run identifier. If this run ID is specified the parameters of the specified run will be used to create a new run.
is_recoveryRecovery mode flag. If recovery mode is set to True, the specified referenced pipeline run and the new run will be grouped under the same groupId.
start_activity_nameIn recovery mode, the rerun will start from this activity. If not specified, all activities will run.
start_from_failureIn recovery mode, if set to true, the rerun will start from failed activities. The property will be used only if start_activity_name is not specified.
parametersParameters of the pipeline run. These parameters are referenced in a pipeline via @pipeline().parameters.parameterName and will be used only if the reference_pipeline_run_id is not specified.
timeoutTime in seconds to wait for a pipeline to reach a terminal status for non-asynchronous waits. Used only if wait_for_termination is True.
check_intervalTime in seconds to check on a pipeline run’s status for non-asynchronous waits. Used only if wait_for_termination
Documentation
Executes a data factory pipeline asynchronously.