LocalFilesystemToS3Operator

Amazon

Uploads a file from a local filesystem to Amazon S3.

View on GitHub

Last Updated: Oct. 23, 2022

Access Instructions

Install the Amazon provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

filenameRequiredPath to the local file. Path can be either absolute (e.g. /path/to/file.ext) or relative (e.g. ../../foo//.csv). (templated)

dest_keyRequiredThe key of the object to copy to. (templated) It can be either full s3:// style url or relative path from root level. When it’s specified as a full s3:// url, please omit dest_bucket.

dest_bucketName of the S3 bucket to where the object is copied. (templated)

aws_conn_idConnection id of the S3 connection to use

verifyWhether or not to verify SSL certificates for S3 connection. By default SSL certificates are verified. You can provide the following values: False: do not validate SSL certificates. SSL will still be used,but SSL certificates will not be verified. path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.You can specify this argument if you want to use a different CA cert bundle than the one used by botocore.

replaceA flag to decide whether or not to overwrite the key if it already exists. If replace is False and the key exists, an error will be raised.

encryptIf True, the file will be encrypted on the server-side by S3 and will be stored in an encrypted form while at rest in S3.

gzipIf True, the file will be compressed locally

acl_policyString specifying the canned ACL policy for the file being uploaded to the S3 bucket.

Documentation

Uploads a file from a local filesystem to Amazon S3.

Example DAGs

Perform Data Quality Checks in Redshift with Airflow

An example pipeline demonstrating how to perform data quality checks in Redshift using SQL Check Operators.

Data Management & GovernanceETL/ELTDatabases

Write-Audit-Publish Great Expectations Example With Snowflake

Example DAG showcasing a write-audit-publish data quality pattern with Snowflake and Great Expectations.

DatabasesData Management & GovernanceData QualityETL/ELT

Great Expectations Redshift Example

Example DAG showcasing loading and data quality checking with Redshift and Great Expectations.

Data QualityETL/ELT

Great Expectations Snowflake Example

Example DAG showcasing loading and data quality checking with Snowflake and Great Expectations.

Data QualityETL/ELT

Taxi Data Quality Example With Snowflake

An example DAG showing the use of SnowflakeCheckOperators.

ETL/ELTData QualityDatabases

Crate Data Quality Checks DAG

Imports local files to S3, then to CrateDB and checks several data quality properties

ETL/ELTData QualityDatabases