Access Instructions
Install the Amazon provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
s3_bucketS3 bucket where logs and local etl script will be uploaded
job_nameunique job name per AWS account
descjob description
concurrent_run_limitThe maximum number of concurrent runs allowed for a job
script_locationpath to etl script on s3
retry_limitMaximum number of times to retry this job if it fails
num_of_dpusNumber of AWS Glue DPUs to allocate to this Job
region_nameaws region name (example: us-east-1)
iam_role_nameAWS IAM Role for Glue Job Execution
create_job_kwargsExtra arguments for Glue Job Creation
Documentation
Interact with AWS Glue. Provide thick wrapper around boto3.client("glue").
Additional arguments (such as aws_conn_id) may be specified and are passed down to the underlying AwsBaseHook.