DataprocCreateMapReduceJobOperator
YandexRuns Mapreduce job in Data Proc cluster.
Access Instructions
Install the Yandex provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
main_jar_file_uriURI of jar file with job. Can be placed in HDFS or S3. Can be specified instead of main_class.
main_className of the main class of the job. Can be specified instead of main_jar_file_uri.
file_urisURIs of files used in the job. Can be placed in HDFS or S3.
archive_urisURIs of archive files used in the job. Can be placed in HDFS or S3.
jar_file_urisURIs of JAR files used in the job. Can be placed in HDFS or S3.
propertiesProperties for the job.
argsArguments to be passed to the job.
nameName of the job. Used for labeling.
cluster_idID of the cluster to run job in. Will try to take the ID from Dataproc Hook object if it’s specified. (templated)
connection_idID of the Yandex.Cloud Airflow connection.
Documentation
Runs Mapreduce job in Data Proc cluster.