SecretsManagerBackend

Amazon

Retrieves Connection or Variables from AWS Secrets Manager

View on GitHub

Last Updated: Mar. 1, 2023

Access Instructions

Install the Amazon provider package into your Airflow environment.

Update your environment config per the instructions in the docs below.

Parameters

Documentation

Connections

Parameters

connections_prefixSpecifies the prefix of the secret to read to get Connections. If set to None (null value in the configuration), requests for connections will not be sent to AWS Secrets Manager. If you don’t want a connections_prefix, set it as an empty string

connections_lookup_patternSpecifies a pattern the connection ID needs to match to be looked up in AWS Secrets Manager. Applies only if connections_prefix is not None. If set to None (null value in the configuration), all connections will be looked up first in AWS Secrets Manager.

variables_prefixSpecifies the prefix of the secret to read to get Variables. If set to None (null value in the configuration), requests for variables will not be sent to AWS Secrets Manager. If you don’t want a variables_prefix, set it as an empty string

variables_lookup_patternSpecifies a pattern the variable key needs to match to be looked up in AWS Secrets Manager. Applies only if variables_prefix is not None. If set to None (null value in the configuration), all variables will be looked up first in AWS Secrets Manager.

config_prefixSpecifies the prefix of the secret to read to get Configurations. If set to None (null value in the configuration), requests for configurations will not be sent to AWS Secrets Manager. If you don’t want a config_prefix, set it as an empty string

config_lookup_patternSpecifies a pattern the config key needs to match to be looked up in AWS Secrets Manager. Applies only if config_prefix is not None. If set to None (null value in the configuration), all config keys will be looked up first in AWS Secrets Manager.

sepseparator used to concatenate secret_prefix and secret_id. Default: “/”

extra_conn_wordsfor using just when you set full_url_mode as false and store the secrets in different fields of secrets manager. You can add more words for each connection part beyond the default ones. The extra words to be searched should be passed as a dict of lists, each list corresponding to a connection part. The optional keys of the dict must be: user, password, host, schema, conn_type.

Documentation

Retrieves Connection or Variables from AWS Secrets Manager

Configurable via airflow.cfg like so:

[secrets]
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {"connections_prefix": "airflow/connections"}

For example, when {"connections_prefix": "airflow/connections"} is set, if a secret is defined with the path airflow/connections/smtp_default, the connection with conn_id smtp_default would be accessible.

When {"variables_prefix": "airflow/variables"} is set, if a secret is defined with the path airflow/variables/hello, the variable with the name hello would be accessible.

When {"config_prefix": "airflow/config"} set, if a secret is defined with the path airflow/config/sql_alchemy_conn, the config with they sql_alchemy_conn would be accessible.

You can also pass additional keyword arguments listed in AWS Connection Extra config to this class, and they would be used for establishing a connection and passed on to Boto3 client.

[secrets]
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {"connections_prefix": "airflow/connections", "region_name": "eu-west-1"}

There are two ways of storing secrets in Secret Manager for using them with this operator: storing them as a conn URI in one field, or taking advantage of native approach of Secrets Manager and storing them in multiple fields. There are certain words that will be searched in the name of fields for trying to retrieve a connection part. Those words are:

possible_words_for_conn_fields = {
    "login": ["login", "user", "username", "user_name"],
    "password": ["password", "pass", "key"],
    "host": ["host", "remote_host", "server"],
    "port": ["port"],
    "schema": ["database", "schema"],
    "conn_type": ["conn_type", "conn_id", "connection_type", "engine"],
}

However, these lists can be extended using the configuration parameter extra_conn_words. Also, you can have a field named extra for extra parameters for the conn. Please note that this extra field must be a valid JSON.