QuboleValueCheckOperator

Qubole

Performs a simple value check using Qubole command. By default, each value on the first row of this Qubole command is compared with a pre-defined value. The check fails and errors out if the output of the command is not within the permissible limit of expected value.

View on GitHub

Last Updated: Oct. 23, 2022

Access Instructions

Install the Qubole provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

Documentation

Parameters

qubole_conn_idConnection id which consists of qds auth_token

pass_valueRequiredExpected value of the query results.

toleranceDefines the permissible pass_value range, for example if tolerance is 2, the Qubole command output can be anything between -2*pass_value and 2*pass_value, without the operator erring out.

results_parser_callableThis is an optional parameter to extend the flexibility of parsing the results of Qubole command to the users. This is a Python callable which can hold the logic to parse list of rows returned by Qubole command. By default, only the values on first row are used for performing checks. This callable should return a list of records on which the checks have to be performed.

Documentation

kwargs:: Arguments specific to Qubole command can be referred from QuboleOperator docs.

Note

All fields in common with template fields of QuboleOperator and SQLValueCheckOperator are template-supported.