EmrContainerSensor
AmazonAsks for the state of the job run until it reaches a failure state or success state. If the job run fails, the task will fail.
Access Instructions
Install the Amazon provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
job_idRequiredjob_id to check the state of
max_retriesNumber of times to poll for query state before returning the current state, defaults to None
aws_conn_idaws connection to use, defaults to ‘aws_default’
poll_intervalTime in seconds to wait between two consecutive call to check query status on athena, defaults to 10
Documentation
Asks for the state of the job run until it reaches a failure state or success state. If the job run fails, the task will fail.
See also
For more information on how to use this sensor, take a look at the guide: Wait on an Amazon EMR virtual cluster job