Execute a command inside a docker container.

View on GitHub

Last Updated: Apr. 14, 2023

Access Instructions

Install the Docker provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.


imageRequiredDocker image from which to create the container. If image tag is omitted, “latest” will be used. (templated)
api_versionRemote API version. Set to auto to automatically detect the server’s version.
commandCommand to be run in the container. (templated)
container_nameName of the container. Optional (templated)
cpusNumber of CPUs to assign to the container. This value gets multiplied with 1024. See
docker_urlURL of the host running the docker daemon. Default is unix://var/run/docker.sock
environmentEnvironment variables to set in the container. (templated)
private_environmentPrivate environment variables to set in the container. These are not templated, and hidden from the website.
env_fileRelative path to the .env file with environment variables to set in the container. Overridden by variables in the environment parameter. (templated)
force_pullPull the docker image on every run. Default is False.
mem_limitMaximum amount of memory the container can use. Either a float value, which represents the limit in bytes, or a string like 128m or 1g.
host_tmp_dirSpecify the location of the temporary directory on the host which will be mapped to tmp_dir. If not provided defaults to using the standard system temp directory.
network_modeNetwork mode for the container. It can be one of the following: bridge - Create new network stack for the container with default docker bridge network None - No networking for this container container: - Use the network stack of another container specified via host - Use the host network stack. Incompatible with port_bindings ‘|’ - Connects the container to user created network (using docker network create command)
tls_ca_certPath to a PEM-encoded certificate authority to secure the docker connection.
tls_client_certPath to the PEM-encoded certificate used to authenticate docker client.
tls_client_keyPath to the PEM-encoded key used to authenticate docker client.
tls_hostnameHostname to match against the docker server certificate or False to disable the check.
tls_ssl_versionVersion of SSL to use when communicating with docker daemon.
mount_tmp_dirSpecify whether the temporary directory should be bind-mounted from the host to the container. Defaults to True
tmp_dirMount point inside the container to a temporary directory created on the host by the operator. The path is also made available via the environment variable AIRFLOW_TMP_DIR inside the container.
userDefault user inside the docker container.
mountsList of volumes to mount into the container. Each item should be a docker.types.Mount instance.
entrypointOverwrite the default ENTRYPOINT of the image
working_dirWorking directory to set on the container (equivalent to the -w switch the docker client)
xcom_allPush all the stdout or just the last line. The default is False (last line).
docker_conn_idThe Docker connection id
dnsDocker custom DNS servers
dns_searchDocker custom DNS search domain
auto_removeAuto-removal of the container on daemon side when the container’s process exits. The default is never.
shm_sizeSize of /dev/shm in bytes. The size must be greater than 0. If omitted uses system default.
ttyAllocate pseudo-TTY to the container This needs to be set see logs of the Docker container.
hostnameOptional hostname for the container.
privilegedGive extended privileges to this container.
cap_addInclude container capabilities
retrieve_outputShould this docker image consistently attempt to pull from and output file before manually shutting down the image. Useful for cases where users want a pickle serialized output that is not posted to logs
retrieve_output_pathpath for output file that will be retrieved and passed to xcom
device_requestsExpose host resources such as GPUs to the container.
log_opts_max_sizeThe maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (k, m, or g). Eg: 10m or 1g Defaults to -1 (unlimited).
log_opts_max_fileThe maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. Only effective when max-size is also set. A positive integer. Defaults to 1.
ipc_modeSet the IPC mode for the container.
skip_exit_codeIf task exits with this exit code, leave the task in skipped state (default: None). If set to None, any non-zero exit code will be treated as a failure.


Execute a command inside a docker container.

By default, a temporary directory is created on the host and mounted into a container to allow storing files that together exceed the default disk size of 10GB in a container. In this case The path to the mounted directory can be accessed via the environment variable AIRFLOW_TMP_DIR.

If the volume cannot be mounted, warning is printed and an attempt is made to execute the docker command without the temporary folder mounted. This is to make it works by default with remote docker engine or when you run docker-in-docker solution and temporary directory is not shared with the docker engine. Warning is printed in logs in this case.

If you know you run DockerOperator with remote engine or via docker-in-docker you should set mount_tmp_dir parameter to False. In this case, you can still use mounts parameter to mount already existing named volumes in your Docker Engine to achieve similar capability where you can store files exceeding default disk size of the container,

If a login to a private registry is required prior to pulling the image, a Docker connection needs to be configured in Airflow and the connection ID be provided with the parameter docker_conn_id.

Was this page helpful?