HdfsSensor
Apache HDFSWaits for a file or folder to land in HDFS
Access Instructions
Install the Apache HDFS provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
filepathRequiredThe route to a stored file.
hdfs_conn_idThe Airflow connection used for HDFS credentials.
ignored_extThis is the list of ignored extensions.
ignore_copyingShall we ignore?
file_sizeThis is the size of the file.
Documentation
Waits for a file or folder to land in HDFS
See also
For more information on how to use this operator, take a look at the guide: Waits for a file or folder to land in HDFS