HdfsRegexSensor
Apache HDFSWaits for matching files by matching on regex
Access Instructions
Install the Apache HDFS provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
filepathThe route to a stored file.
hdfs_conn_idThe Airflow connection used for HDFS credentials.
ignored_extThis is the list of ignored extensions.
ignore_copyingShall we ignore?
file_sizeThis is the size of the file.
Documentation
Waits for matching files by matching on regex
See also
For more information on how to use this operator, take a look at the guide: HdfsRegexSensor