EmrContainerSensorAsync
Astronomer ProvidersEmrContainerSensorAsync is async version of EmrContainerSensor, Asks for the state of the job run until it reaches a failure state or success state. If the job run fails, the task will fail.
Access Instructions
Install the Astronomer Providers provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
virtual_cluster_idReference Emr cluster id
job_idjob_id to check the state
max_retriesNumber of times to poll for query state before returning the current state, defaults to None
aws_conn_idaws connection to use, defaults to aws_default
poll_intervalTime in seconds to wait between two consecutive call to check query status on athena, defaults to 10
Documentation
EmrContainerSensorAsync is async version of EmrContainerSensor, Asks for the state of the job run until it reaches a failure state or success state. If the job run fails, the task will fail.