PinotAdminHook

Apache Pinot

This hook is a wrapper around the pinot-admin.sh script. For now, only small subset of its subcommands are implemented, which are required to ingest offline data into Apache Pinot (i.e., AddSchema, AddTable, CreateSegment, and UploadSegment). Their command options are based on Pinot v0.1.0.

View on GitHub

Last Updated: Nov. 13, 2022

Access Instructions

Install the Apache Pinot provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

Documentation

Parameters

conn_idThe name of the connection to use.

cmd_pathDo not modify the parameter. It used to be the filepath to the pinot-admin.sh executable but in version 4.0.0 of apache-pinot provider, value of this parameter must remain the default value: pinot-admin.sh. It is left here to not accidentally override the pinot_admin_system_exit in case positional parameters were used to initialize the hook.

pinot_admin_system_exitIf true, the result is evaluated based on the status code. Otherwise, the result is evaluated as a failure if “Error” or “Exception” is in the output message.

Documentation

Unfortunately, as of v0.1.0, pinot-admin.sh always exits with status code 0. To address this behavior, users can use the pinot_admin_system_exit flag. If its value is set to false, this hook evaluates the result based on the output message instead of the status code. This Pinot’s behavior is supposed to be improved in the next release, which will include the following PR: https://github.com/apache/incubator-pinot/pull/4110

PinotAdminHook

Access Instructions

Parameters

Documentation

Was this page helpful?

Product

Resources

Company