SQLTableCheckOperator
Common SQLPerforms one or more of the checks provided in the checks dictionary. Checks should be written to return a boolean result.
Access Instructions
Install the Common SQL provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
Documentation
Performs one or more of the checks provided in the checks dictionary. Checks should be written to return a boolean result.
{"row_count_check": {"check_statement": "COUNT(*) = 1000"},"column_sum_check": {"check_statement": "col_a + col_b < col_c"},"third_check": {"check_statement": "MIN(col) = 1", "partition_clause": "col IS NOT NULL"},}
- param partition_clause
a partial SQL statement that is added to a WHERE clause in the query built by the operator that creates partition_clauses for the checks to run on, e.g.
"date = '1970-01-01'"
- param conn_id
the connection ID used to connect to the database
- param database
name of database which overwrite the defined one in connection
See also
For more information on how to use this operator, take a look at the guide: Check SQL Table Values
Example DAGs
Example DAG showcasing loading, transforming, and data quality checking with multiple datasets in Snowflake.
Imports local files to S3, then to CrateDB and checks several data quality properties
