DatabricksSqlOperator
DatabricksExecutes SQL code in a Databricks SQL endpoint or a Databricks cluster
Access Instructions
Install the Databricks provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
databricks_conn_idReference to Databricks connection id (templated)
http_pathOptional string specifying HTTP path of Databricks SQL Endpoint or cluster. If not specified, it should be either specified in the Databricks connection’s extra parameters, or sql_endpoint_name must be specified.
sql_endpoint_nameOptional name of Databricks SQL Endpoint. If not specified, http_path must be provided as described above.
sqlthe SQL code to be executed as a single string, or a list of str (sql statements), or a reference to a template file. (templated) Template references are recognized by str ending in ‘.sql’
parameters(optional) the parameters to render the SQL query with.
session_configurationAn optional dictionary of Spark session parameters. Defaults to None. If not specified, it could be specified in the Databricks connection’s extra parameters.
client_parametersAdditional parameters internal to Databricks SQL Connector parameters
http_headersAn optional list of (k, v) pairs that will be set as HTTP headers on every request. (templated)
catalogAn optional initial catalog to use. Requires DBR version 9.0+ (templated)
schemaAn optional initial schema to use. Requires DBR version 9.0+ (templated)
output_pathoptional string specifying the file to which write selected data. (templated)
output_formatformat of output data if output_path` is specified. Possible values are ``csv, json, jsonl. Default is csv.
csv_paramsparameters that will be passed to the csv.DictWriter class used to write CSV data.
Documentation
Executes SQL code in a Databricks SQL endpoint or a Databricks cluster
See also
For more information on how to use this operator, take a look at the guide: DatabricksSqlOperator