LivyTrigger
Apache LivyCheck for the state of a previously submitted job with batch_id
Access Instructions
Install the Apache Livy provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
batch_idBatch job id
spark_paramsSpark parameters; for example, spark_params = {“file”: “test/pi.py”, “class_name”: “org.apache.spark.examples.SparkPi”, “args”: [“/usr/lib/spark/bin/run-example”, “SparkPi”, “10”],”jars”: “command-runner.jar”, “driver_cores”: 1, “executor_cores”: 4,”num_executors”: 1}
livy_conn_idreference to a pre-defined Livy Connection.
polling_intervaltime in seconds between polling for job completion. If poll_interval=0, in that case return the batch_id and if polling_interval > 0, poll the livy job for termination in the polling interval defined.
extra_optionsA dictionary of options, where key is string and value depends on the option that’s being modified.
extra_headersA dictionary of headers passed to the HTTP request to livy.
livy_hook_asyncLivyAsyncHook object
Documentation
Check for the state of a previously submitted job with batch_id