ADLSListOperator

Microsoft Azure

List all files from the specified path

Last Updated: Oct. 23, 2022

Install the Microsoft Azure provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

pathRequiredThe Azure Data Lake path to find the objects. Supports glob strings (templated)

azure_data_lake_conn_idReference to the Azure Data Lake connection.

List all files from the specified path

param path: The Azure Data Lake path to find the objects. Supports glob strings (templated)
param azure_data_lake_conn_id: Reference to the Azure Data Lake connection.

Example:

The following Operator would list all the Parquet files from folder/output/ folder in the specified ADLS account

adls_files = ADLSListOperator(
    task_id='adls_files',
    path='folder/output/*.parquet',
    azure_data_lake_conn_id='azure_data_lake_default'
)