EmrJobFlowSensorAsync
Astronomer ProvidersAsync EMR Job flow sensor polls for the cluster state until it reaches any of the target states. If it fails the sensor errors, failing the task. With the default target states, sensor waits cluster to be terminated. When target_states is set to [‘RUNNING’, ‘WAITING’] sensor waits until job flow to be ready (after ‘STARTING’ and ‘BOOTSTRAPPING’ states)
Access Instructions
Install the Astronomer Providers provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
Documentation
Async EMR Job flow sensor polls for the cluster state until it reaches any of the target states. If it fails the sensor errors, failing the task. With the default target states, sensor waits cluster to be terminated. When target_states is set to [‘RUNNING’, ‘WAITING’] sensor waits until job flow to be ready (after ‘STARTING’ and ‘BOOTSTRAPPING’ states)