KubernetesPodOperator
KubernetesExecute a task in a Kubernetes Pod
Access Instructions
Install the Kubernetes provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
kubernetes_conn_idThe kubernetes connection id for the Kubernetes cluster.
namespacethe namespace to run within kubernetes.
imageDocker image you wish to launch. Defaults to hub.docker.com, but fully qualified URLS will point to custom repositories. (templated)
namename of the pod in which the task will run, will be used (plus a random suffix if random_name_suffix is True) to generate a pod id (DNS-1123 subdomain, containing only [a-z0-9.-]).
random_name_suffixif True, will generate a random suffix.
cmdsentrypoint of the container. (templated) The docker images’s entrypoint is used if this is not provided.
argumentsarguments of the entrypoint. (templated) The docker image’s CMD is used if this is not provided.
portsports for the launched pod.
volume_mountsvolumeMounts for the launched pod.
volumesvolumes for the launched pod. Includes ConfigMaps and PersistentVolumes.
env_varsEnvironment variables initialized in the container. (templated)
env_from(Optional) List of sources to populate environment variables in the container.
secretsKubernetes secrets to inject in the container. They can be exposed as environment vars or files in a volume.
in_clusterrun kubernetes client with in_cluster configuration.
cluster_contextcontext that points to kubernetes cluster. Ignored when in_cluster is True. If None, current-context is used.
reattach_on_restartif the worker dies while the pod is running, reattach and monitor during the next try. If False, always create a new pod for each try.
labelslabels to apply to the Pod. (templated)
startup_timeout_secondstimeout in seconds to startup the pod.
get_logsget the stdout of the base container as logs of the tasks.
image_pull_policySpecify a policy to cache or always pull an image.
annotationsnon-identifying metadata you can attach to the Pod. Can be a large range of data, and can include characters that are not permitted by labels.
container_resourcesresources for the launched pod. (templated)
affinityaffinity scheduling rules for the launched pod.
config_fileThe path to the Kubernetes config file. (templated) If not specified, default value is ~/.kube/config
node_selectorA dict containing a group of scheduling rules.
image_pull_secretsAny image pull secrets to be given to the pod. If more than one secret is required, provide a comma separated list: secret_a,secret_b
service_account_nameName of the service account
is_delete_operator_podWhat to do when the pod reaches its final state, or the execution is interrupted. If True (default), delete the pod; if False, leave the pod.
hostnetworkIf True enable host networking on the pod.
tolerationsA list of kubernetes tolerations.
security_contextsecurity options the pod should run with (PodSecurityContext).
container_security_contextsecurity options the container should run with.
dnspolicydnspolicy for the pod.
schedulernameSpecify a schedulername for the pod
full_pod_specThe complete podSpec
init_containersinit container for the launched Pod
log_events_on_failureLog the pod’s events if a failure occurs
do_xcom_pushIf True, the content of the file /airflow/xcom/return.json in the container will also be pushed to an XCom when the container completes.
pod_template_filepath to pod template file (templated)
priority_class_namepriority class name for the launched Pod
pod_runtime_info_envs(Optional) A list of environment variables, to be set in the container.
termination_grace_periodTermination grace period if task killed in UI, defaults to kubernetes default
configmaps(Optional) A list of names of config maps from which it collects ConfigMaps to populate the environment variables with. The contents of the target ConfigMap’s Data field will represent the key-value pairs as environment variables. Extends env_from.
base_container_nameThe name of the base container in the pod. This container’s logs will appear as part of this task’s logs if get_logs is True. Defaults to None. If None, will consult the class variable BASE_CONTAINER_NAME (which defaults to “base”) for the base container name to use.
deferrableRun operator in the deferrable mode.
poll_intervalPolling period in seconds to check for the status. Used only in deferrable mode.
Documentation
Execute a task in a Kubernetes Pod
See also
For more information on how to use this operator, take a look at the guide: KubernetesPodOperator
Note
If you use Google Kubernetes Engine and Airflow is not running in the same cluster, consider using GKEStartPodOperator, which simplifies the authorization process.