S3ToRedshiftOperator

Amazon

Executes an COPY command to load files from s3 to Redshift

View on GitHub

Last Updated: Feb. 20, 2023

Access Instructions

Install the Amazon provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

schemaRequiredreference to a specific schema in redshift database

tableRequiredreference to a specific table in redshift database

s3_bucketRequiredreference to a specific S3 bucket

s3_keyRequiredkey prefix that selects single or multiple objects from S3

redshift_conn_idreference to a specific redshift database OR a redshift data-api connection

aws_conn_idreference to a specific S3 connection If the AWS connection contains ‘aws_iam_role’ in extras the operator will use AWS STS credentials with a token https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-authorization.html#copy-credentials

verifyWhether or not to verify SSL certificates for S3 connection. By default SSL certificates are verified. You can provide the following values: False: do not validate SSL certificates. SSL will still be used(unless use_ssl is False), but SSL certificates will not be verified. path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.You can specify this argument if you want to use a different CA cert bundle than the one used by botocore.

column_listlist of column names to load

copy_optionsreference to a list of COPY options

methodAction to be performed on execution. Available APPEND, UPSERT and REPLACE.

upsert_keysList of fields to use as key on upsert action

redshift_data_api_kwargsIf using the Redshift Data API instead of the SQL-based connection, dict of arguments for the hook’s execute_query method. Cannot include any of these kwargs: {'sql', 'parameters'}

Documentation

Executes an COPY command to load files from s3 to Redshift

Example DAGs

Executing Predictions with AWS SageMaker

This DAG shows an example implementation of executing predictions from a machine learning model using AWS SageMaker.

AI + Machine LearningData Science

Perform Data Quality Checks in Redshift with Airflow

An example pipeline demonstrating how to perform data quality checks in Redshift using SQL Check Operators.

Data Management & GovernanceETL/ELTDatabases

Great Expectations Redshift Example

Example DAG showcasing loading and data quality checking with Redshift and Great Expectations.

Data QualityETL/ELT

Perform Data Integrity Checks from S3 to Redshift

This is the second in a series of DAGs showing an EL pipeline with data integrity checking of data in S3 as well as Redshift.

Data Management & GovernanceETL/ELTDatabases

Advanced Data Integrity Checks from S3 to Redshift

This is the third in a series of DAGs showing an EL pipeline with data integrity and data quality checking for data in S3 and Redshift using ETag verification and row-based data quality checks where t…

Data Management & GovernanceETL/ELTDatabases

Tutorial Redshift DAG

Run an ETL pipeline that resumes a Redshift cluster, extracts data from S3, transforms data, loads data back to S3, then pauses the cluster.

ETL/ELTDatabases

Datasets ML Example Consume

This DAG shows an example implementation of executing predictions from a machine learning model using AWS SageMaker.

Airflow FundamentalsETL/ELTAI + Machine Learning