GlueCatalogPartitionSensor
AmazonWaits for a partition to show up in AWS Glue Catalog.
Access Instructions
Install the Amazon provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
table_nameRequiredThe name of the table to wait for, supports the dot notation (my_database.my_table)
expressionThe partition clause to wait for. This is passed as is to the AWS Glue Catalog API’s get_partitions function, and supports SQL like notation as in ds='2015-01-01' AND type='value' and comparison operators as in "ds>=2015-01-01". See https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html #aws-glue-api-catalog-partitions-GetPartitions
aws_conn_idID of the Airflow connection where credentials and extra configuration are stored
region_nameOptional aws region name (example: us-east-1). Uses region from connection if not specified.
database_nameThe name of the catalog database where the partitions reside.
poke_intervalTime in seconds that the job should wait in between each tries
Documentation
Waits for a partition to show up in AWS Glue Catalog.