AzureSynapseRunSparkBatchOperator
Microsoft AzureExecutes a Spark job on Azure Synapse.
Access Instructions
Install the Microsoft Azure provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
azure_synapse_conn_idThe connection identifier for connecting to Azure Synapse.
wait_for_terminationFlag to wait on a job run’s termination.
spark_poolThe target synapse spark pool used to submit the job
payloadRequiredLivy compatible payload which represents the spark job that a user wants to submit
timeoutTime in seconds to wait for a job to reach a terminal status for non-asynchronous waits. Used only if wait_for_termination is True.
check_intervalTime in seconds to check on a job run’s status for non-asynchronous waits. Used only if wait_for_termination is True.
Documentation
Executes a Spark job on Azure Synapse.