S3ListOperator
AmazonList all objects from the bucket with the given string prefix in name.
Access Instructions
Install the Amazon provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
Documentation
List all objects from the bucket with the given string prefix in name.
This operator returns a python list with the name of objects which can be used by xcom in the downstream task.
See also
For more information on how to use this operator, take a look at the guide: List Amazon S3 objects
- Example:
The following operator would list all the files (excluding subfolders) from the S3
customers/2018/04/key in thedatabucket.s3_file = S3ListOperator(task_id='list_3s_files',bucket='data',prefix='customers/2018/04/',delimiter='/',aws_conn_id='aws_customers_conn')
Example DAGs
This DAG shows an example implementation of sorting files in an S3 bucket into two different buckets based on logic involving the content of the files using dynamic task mapping with the expand_kwargs…
This DAG shows an example implementation of comparing data between two S3 buckets and a table in Snowflake using the new .zip() method of the XComArg object.
