EmrAddStepsOperator
AmazonAn operator that adds steps to an existing EMR job_flow.
Access Instructions
Install the Amazon provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
job_flow_idid of the JobFlow to add steps to. (templated)
job_flow_namename of the JobFlow to add steps to. Use as an alternative to passing job_flow_id. will search for id of JobFlow with matching name in one of the states in param cluster_states. Exactly one cluster like this should exist or will fail. (templated)
cluster_statesAcceptable cluster states when searching for JobFlow id by job_flow_name. (templated)
aws_conn_idaws connection to uses
stepsboto3 style steps or reference to a steps file (must be ‘.json’) to be added to the jobflow. (templated)
wait_for_completionIf True, the operator will wait for all the steps to be completed.
execution_role_arnThe ARN of the runtime role for a step on the cluster.
do_xcom_pushif True, job_flow_id is pushed to XCom with key job_flow_id.
Documentation
An operator that adds steps to an existing EMR job_flow.
See also
For more information on how to use this operator, take a look at the guide: Add Steps to an EMR job flow