BeamRunGoPipelineOperator

Apache Beam

Launching Apache Beam pipelines written in Go. Note that both default_pipeline_options and pipeline_options will be merged to specify pipeline execution parameter, and default_pipeline_options is expected to save high-level options, for instances, project and zone information, which apply to all beam operators in the DAG.

View on GitHub

Last Updated: Jan. 23, 2023

Access Instructions

Install the Apache Beam provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

Documentation

Parameters

go_fileReference to the Apache Beam pipeline Go source file, e.g. /local/path/to/main.go or gs://bucket/path/to/main.go. Exactly one of go_file and launcher_binary must be provided.

launcher_binaryReference to the Apache Beam pipeline Go binary compiled for the launching platform, e.g. /local/path/to/launcher-main or gs://bucket/path/to/launcher-main. Exactly one of go_file and launcher_binary must be provided.

worker_binaryReference to the Apache Beam pipeline Go binary compiled for the worker platform, e.g. /local/path/to/worker-main or gs://bucket/path/to/worker-main. Needed if the OS or architecture of the workers running the pipeline is different from that of the platform launching the pipeline. For more information, see the Apache Beam documentation for Go cross compilation: https://beam.apache.org/documentation/sdks/go-cross-compilation/. If launcher_binary is not set, providing a worker_binary will have no effect. If launcher_binary is set and worker_binary is not, worker_binary will default to the value of launcher_binary.