SqlToS3Operator
AmazonSaves data from a specific SQL query into a file in S3.
Access Instructions
Install the Amazon provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
queryRequiredthe sql query to be executed. If you want to execute a file, place the absolute path of it, ending with .sql extension. (templated)
s3_bucketRequiredbucket where the data will be stored. (templated)
s3_keyRequireddesired key for the file. It includes the name of the file. (templated)
replacewhether or not to replace the file in S3 if it previously existed
sql_conn_idRequiredreference to a specific database.
parameters(optional) the parameters to render the SQL query with.
aws_conn_idreference to a specific S3 connection
verifyWhether or not to verify SSL certificates for S3 connection. By default SSL certificates are verified. You can provide the following values: False: do not validate SSL certificates. SSL will still be used(unless use_ssl is False), but SSL certificates will not be verified. path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.You can specify this argument if you want to use a different CA cert bundle than the one used by botocore.
file_formatthe destination file format, only string ‘csv’, ‘json’ or ‘parquet’ is accepted.
pd_kwargsarguments to include in DataFrame .to_parquet(), .to_json() or .to_csv().
Documentation
Saves data from a specific SQL query into a file in S3.
See also
For more information on how to use this operator, take a look at the guide: MySQL to Amazon S3 transfer operator