BeamRunPythonPipelineOperator

Apache Beam

Launching Apache Beam pipelines written in Python. Note that both default_pipeline_options and pipeline_options will be merged to specify pipeline execution parameter, and default_pipeline_options is expected to save high-level options, for instances, project and zone information, which apply to all beam operators in the DAG.

View on GitHub

Last Updated: Jan. 23, 2023

Access Instructions

Install the Apache Beam provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

Documentation

Example DAGs

Parameters

py_fileRequiredReference to the python Apache Beam pipeline file.py, e.g., /some/local/file/path/to/your/python/pipeline/file. (templated)

py_optionsAdditional python options, e.g., [“-m”, “-v”].

py_interpreterPython version of the beam pipeline. If None, this defaults to the python3. To track python versions supported by beam and related issues check: https://issues.apache.org/jira/browse/BEAM-1251

py_requirementsAdditional python package(s) to install. If a value is passed to this parameter, a new virtual environment has been created with additional packages installed. You could also install the apache_beam package if it is not installed on your system or you want to use a different version.

py_system_site_packagesWhether to include system_site_packages in your virtualenv. See virtualenv documentation for more information. This option is only relevant if the py_requirements parameter is not None.

Documentation

Example DAGs

dataflow

Example Airflow DAG for Google Cloud Dataflow service

Big Data & AnalyticsData ProcessingETL/ELT