GlueCrawlerOperator
AmazonCreates, updates and triggers an AWS Glue Crawler. AWS Glue Crawler is a serverless service that manages a catalog of metadata tables that contain the inferred schema, format and data types of data stores within the AWS cloud.
Access Instructions
Install the Amazon provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
configConfigurations for the AWS Glue crawler
aws_conn_idaws connection to use
poll_intervalTime (in seconds) to wait between two consecutive calls to check crawler status
wait_for_completionWhether or not wait for crawl execution completion. (default: True)
Documentation
Creates, updates and triggers an AWS Glue Crawler. AWS Glue Crawler is a serverless service that manages a catalog of metadata tables that contain the inferred schema, format and data types of data stores within the AWS cloud.
See also
For more information on how to use this operator, take a look at the guide: Create an AWS Glue crawler