SparkKubernetesSensor
KubernetesChecks sparkApplication object in kubernetes cluster:
Access Instructions
Install the Kubernetes provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
application_nameRequiredspark Application resource name
namespacethe kubernetes namespace where the sparkApplication reside in
container_namethe kubernetes container name where the sparkApplication reside in
kubernetes_conn_idThe kubernetes connection to Kubernetes cluster.
attach_logdetermines whether logs for driver pod should be appended to the sensor log
api_groupkubernetes api group of sparkApplication
api_versionkubernetes api version of sparkApplication
Documentation
Checks sparkApplication object in kubernetes cluster:
See also
For more detail about Spark Application Object have a look at the reference: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.1.0-2.4.5/docs/api-docs.md#sparkapplication