FTPToS3Operator
AmazonThis operator enables the transfer of files from a FTP server to S3. It can be used to transfer one or multiple files.
Access Instructions
Install the Amazon provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
ftp_pathRequiredThe ftp remote path. For one file it is mandatory to include the file as well. For multiple files, it is the route where the files will be found.
s3_bucketRequiredThe targeted s3 bucket in which to upload the file(s).
s3_keyRequiredThe targeted s3 key. For one file it must include the file path. For several, it must end with “/”.
ftp_filenamesOnly used if you want to move multiple files. You can pass a list with exact filenames present in the ftp path, or a prefix that all files must meet. It can also be the string ‘*’ for moving all the files within the ftp path.
s3_filenamesOnly used if you want to move multiple files and name them different from the originals from the ftp. It can be a list of filenames or file prefix (that will replace the ftp prefix).
ftp_conn_idThe ftp connection id. The name or identifier for establishing a connection to the FTP server.
aws_conn_idThe s3 connection id. The name or identifier for establishing a connection to S3.
replaceA flag to decide whether or not to overwrite the key if it already exists. If replace is False and the key exists, an error will be raised.
encryptIf True, the file will be encrypted on the server-side by S3 and will be stored in an encrypted form while at rest in S3.
gzipIf True, the file will be compressed locally
acl_policyString specifying the canned ACL policy for the file being uploaded to the S3 bucket.
Documentation
This operator enables the transfer of files from a FTP server to S3. It can be used to transfer one or multiple files.
See also
For more information on how to use this operator, take a look at the guide: FTP to Amazon S3 transfer operator