dataframe
Astro SDKThis decorator will allow users to write python functions while treating SQL tables as dataframes.
Access Instructions
Install the Astro SDK provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
python_callableCallableThis parameter is filled in automatically when you use the dataframe function as a decorator. This is where the python function gets passed to the wrapping function
multiple_outputsboolIf set to True, the decorated function's return value will be unrolled to multiple XCom values. Dict will unroll to XCom values with its keys as XCom keys. Defaults to False.
conn_idstrConnection ID for the database you want to connect to. If you do not pass in a value for this object we can infer the connection ID from the first table passed into the python_callable function. (required if there are no table arguments)
databasestrDatabase within the SQL instance you want to access. If left blank we will default to the table.metatadata.database in the first Table passed to the function (required if there are no table arguments)
schemastrSchema within the SQL instance you want to access. If left blank we will default to the table.metatadata.schema in the first Table passed to the function (required if there are no table arguments)
columns_names_capitalizationLiteral['upper', 'lower', 'original']determines whether to convert all columns to lowercase/uppercase in the resulting dataframe
kwargsDictAny keyword arguments supported by the BaseOperator is supported (e.g queue, owner)
Documentation
This decorator allows a user to run python functions in Airflow but with the huge benefit that SQL tables will automatically be turned into dataframes and resulting dataframes can automatically used in astro.sql functions.