GreatExpectationsOperator

Great Expectations

An operator to leverage Great Expectations as a task in your Airflow DAG.

View on GitHub

Access Instructions

Install the Great Expectations provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

Documentation

Example DAGs

Parameters

run_nameOptional[str]Identifies the validation run (defaults to timestamp if not specified)

conn_idOptional[str]The name of a connection in Airflow

execution_engineOptional[str]The execution engine to use when running Great Expectations

expectation_suite_nameOptional[str]Name of the expectation suite to run if using a default Checkpoint

data_asset_nameOptional[str]The name of the table or dataframe that the default Data Context will load and default Checkpoint will run over

data_context_root_dirOptional[str]Path of the great_expectations directory

data_context_configOptional[DataContextConfig]A great_expectations DataContextConfig object

dataframe_to_validateOptional[str]A pandas dataframe to validate

query_to_validateOptional[str]A SQL query to validate

checkpoint_nameOptional[str]A Checkpoint name to use for validation

checkpoint_configOptional[CheckpointConfig]A great_expectations CheckpointConfig object to use for validation

checkpoint_kwargsOptional[Dict]A dictionary whose keys match the parameters of CheckpointConfig which can be used to update and populate the Operator’s Checkpoint at runtime

validation_failure_callbackCallable[[CheckpointResult], None]Called when the Great Expectations validation fails

fail_task_on_validation_failureboolFail the Airflow task if the Great Expectation validation fails

return_json_dictboolIf True, returns a json-serializable dictionary instead of a CheckpointResult object

use_open_lineageboolIf True (default), creates an OpenLineage action if an OpenLineage environment is found

schemaOptional[str]If provided, overwrites the default schema provided by the connection

Documentation

An operator to leverage Great Expectations as a task in your Airflow DAG.

Current list of expectations types: https://docs.greatexpectations.io/en/latest/reference/glossary_of_expectations.html

How to create expectations files: https://docs.greatexpectations.io/en/latest/guides/tutorials/how_to_create_expectations.html

Example DAGs

Integrate Great Expectations with Kedro Pipelines

Example DAG that integrates Great Expectations checks into a Kedro pipeline.

Data ProcessingData Management & GovernanceData Science

great_expectations_dag

A DAG that demonstrates use of the operators in this provider package.

Data Management & Governance

Write-Audit-Publish Great Expectations Example With Snowflake

Example DAG showcasing a write-audit-publish data quality pattern with Snowflake and Great Expectations.

DatabasesData Management & GovernanceData QualityETL/ELT

Great Expectations BigQuery Example

Example DAG showcasing loading and data quality checking with BigQuery and Great Expectations.

Data QualityETL/ELT

Great Expectations Redshift Example

Example DAG showcasing loading and data quality checking with Redshift and Great Expectations.

Data QualityETL/ELT

Great Expectations Snowflake Example

Example DAG showcasing loading and data quality checking with Snowflake and Great Expectations.

Data QualityETL/ELT