S3KeySensor
AmazonWaits for one or multiple keys (a file-like instance on S3) to be present in a S3 bucket. The path is just a key/value pointer to a resource for the given S3 path. Note: S3 does not support folders directly, and only provides key/value pairs.
Access Instructions
Install the Amazon provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
bucket_keyRequiredThe key(s) being waited on. Supports full s3:// style url or relative path from root level. When it’s specified as a full s3:// url, please leave bucket_name as None
bucket_nameName of the S3 bucket. Only needed when bucket_key is not provided as a full s3:// url. When specified, all the keys passed to bucket_key refers to this bucket
wildcard_matchwhether the bucket_key should be interpreted as a Unix wildcard pattern
check_fnFunction that receives the list of the S3 objects, and returns a boolean: - True: the criteria is met - False: the criteria isn’t met Example: Wait for any S3 object size more than 1 megabyte def check_fn(files: List) -> bool: return any(f.get('Size', 0) > 1048576 for f in files)
aws_conn_ida reference to the s3 connection
verifyWhether or not to verify SSL certificates for S3 connection. By default SSL certificates are verified. You can provide the following values: False: do not validate SSL certificates. SSL will still be used(unless use_ssl is False), but SSL certificates will not be verified. path/to/cert/bundle.pem: A filename of the CA cert bundle to uses.You can specify this argument if you want to use a different CA cert bundle than the one used by botocore.
Documentation
Waits for one or multiple keys (a file-like instance on S3) to be present in a S3 bucket. The path is just a key/value pointer to a resource for the given S3 path. Note: S3 does not support folders directly, and only provides key/value pairs.
See also
For more information on how to use this sensor, take a look at the guide: Wait on an Amazon S3 key