GreatExpectationsOperator

Great Expectations

An operator to leverage Great Expectations as a task in your Airflow DAG.

Access Instructions

Install the Great Expectations provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

run_nameOptional[str]Identifies the validation run (defaults to timestamp if not specified)
conn_idOptional[str]The name of a connection in Airflow
execution_engineOptional[str]The execution engine to use when running Great Expectations
expectation_suite_nameOptional[str]Name of the expectation suite to run if using a default Checkpoint
data_asset_nameOptional[str]The name of the table or dataframe that the default Data Context will load and default Checkpoint will run over
data_context_root_dirOptional[str]Path of the great_expectations directory
data_context_configOptional[DataContextConfig]A great_expectations DataContextConfig object
dataframe_to_validateOptional[str]A pandas dataframe to validate
query_to_validateOptional[str]A SQL query to validate
checkpoint_nameOptional[str]A Checkpoint name to use for validation
checkpoint_configOptional[CheckpointConfig]A great_expectations CheckpointConfig object to use for validation
checkpoint_kwargsOptional[Dict]A dictionary whose keys match the parameters of CheckpointConfig which can be used to update and populate the Operator’s Checkpoint at runtime
validation_failure_callbackCallable[[CheckpointResult], None]Called when the Great Expectations validation fails
fail_task_on_validation_failureboolFail the Airflow task if the Great Expectation validation fails
return_json_dictboolIf True, returns a json-serializable dictionary instead of a CheckpointResult object
use_open_lineageboolIf True (default), creates an OpenLineage action if an OpenLineage environment is found
schemaOptional[str]If provided, overwrites the default schema provided by the connection

Documentation

An operator to leverage Great Expectations as a task in your Airflow DAG.

Current list of expectations types: https://docs.greatexpectations.io/en/latest/reference/glossary_of_expectations.html

How to create expectations files: https://docs.greatexpectations.io/en/latest/guides/tutorials/how_to_create_expectations.html

Was this page helpful?