PubSubCreateSubscriptionOperator

Google

Create a PubSub subscription.

View on GitHub

Last Updated: Feb. 25, 2023

Access Instructions

Install the Google provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

Documentation

Connections

Parameters

project_idOptional, the Google Cloud project ID where the topic exists. If set to None or missing, the default project_id from the Google Cloud connection is used.

topicRequiredthe topic to create. Do not include the full topic path. In other words, instead of projects/{project}/topics/{topic}, provide only {topic}. (templated)

subscriptionthe Pub/Sub subscription name. If empty, a random name will be generated using the uuid module

subscription_project_idthe Google Cloud project ID where the subscription will be created. If empty, topic_project will be used.

ack_deadline_secsNumber of seconds that a subscriber has to acknowledge each message pulled from the subscription

gcp_conn_idThe connection ID to use connecting to Google Cloud.

delegate_toThe account to impersonate using domain-wide delegation of authority, if any. For this to work, the service account making the request must have domain-wide delegation enabled.

push_configIf push delivery is used with this subscription, this field is used to configure it. An empty pushConfig signifies that the subscriber will pull and ack messages using API methods.

retain_acked_messagesIndicates whether to retain acknowledged messages. If true, then messages are not expunged from the subscription’s backlog, even if they are acknowledged, until they fall out of the message_retention_duration window. This must be true if you would like to Seek to a timestamp.

message_retention_durationHow long to retain unacknowledged messages in the subscription’s backlog, from the moment a message is published. If retain_acked_messages is true, then this also configures the retention of acknowledged messages, and thus configures how far back in time a Seek can be done. Defaults to 7 days. Cannot be more than 7 days or less than 10 minutes.

labelsClient-assigned labels; see https://cloud.google.com/pubsub/docs/labels

enable_message_orderingIf true, messages published with the same ordering_key in PubsubMessage will be delivered to the subscribers in the order in which they are received by the Pub/Sub system. Otherwise, they may be delivered in any order.

expiration_policyA policy that specifies the conditions for this subscription’s expiration. A subscription is considered active as long as any connected subscriber is successfully consuming messages from the subscription or is issuing operations on the subscription. If expiration_policy is not set, a default policy with ttl of 31 days will be used. The minimum allowed value for expiration_policy.ttl is 1 day.

filter_An expression written in the Cloud Pub/Sub filter language. If non-empty, then only PubsubMessages whose attributes field matches the filter are delivered on this subscription. If empty, then no messages are filtered out.

dead_letter_policyA policy that specifies the conditions for dead lettering messages in this subscription. If dead_letter_policy is not set, dead lettering is disabled.

retry_policyA policy that specifies how Pub/Sub retries message delivery for this subscription. If not set, the default retry policy is applied. This generally implies that messages will be retried as soon as possible for healthy subscribers. RetryPolicy will be triggered on NACKs or acknowledgement deadline exceeded events for a given message.

retry(Optional) A retry object used to retry requests. If None is specified, requests will not be retried.

timeout(Optional) The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

metadata(Optional) Additional metadata that is provided to the method.

impersonation_chainOptional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

Documentation

Create a PubSub subscription.

with DAG('successful DAG') as dag:
    (
        PubSubCreateSubscriptionOperator(
            project_id='my-project',
            topic='my-topic',
            subscription='my-subscription'
        )
        >> PubSubCreateSubscriptionOperator(
            project_id='my-project',
            topic='my-topic',
            subscription='my-subscription',
        )
    )

The operator can be configured to fail if the subscription already exists.

with DAG('failing DAG') as dag:
    (
        PubSubCreateSubscriptionOperator(
            project_id='my-project',
            topic='my-topic',
            subscription='my-subscription',
        )
        >> PubSubCreateSubscriptionOperator(
            project_id='my-project',
            topic='my-topic',
            subscription='my-subscription',
            fail_if_exists=True,
        )
    )

Finally, subscription is not required. If not passed, the operator will generated a universally unique identifier for the subscription’s name.

with DAG('DAG') as dag:
    PubSubCreateSubscriptionOperator(project_id='my-project', topic='my-topic')

project_id, topic, subscription, subscription_project_id and impersonation_chain are templated so you can use Jinja templating in their values.