PostgresHook
PostgresInteract with Postgres.
Access Instructions
Install the Postgres provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
Documentation
Interact with Postgres.
You can specify ssl parameters in the extra field of your connection as {"sslmode": "require", "sslcert": "/path/to/cert.pem", etc}. Also you can choose cursor as {"cursor": "dictcursor"}. Refer to the psycopg2.extras for more details.
Note: For Redshift, use keepalives_idle in the extra connection parameters and set it to less than 300 seconds.
Note: For AWS IAM authentication, use iam in the extra connection parameters and set it to true. Leave the password field empty. This will use the “aws_default” connection to get the temporary token unless you override in extras. extras example: {"iam":true, "aws_conn_id":"my_aws_conn"} For Redshift, also use redshift in the extra connection parameters and set it to true. The cluster-identifier is extracted from the beginning of the host field, so is optional. It can however be overridden in the extra field. extras example: {"iam":true, "redshift":true, "cluster-identifier": "my_cluster_id"}
Example DAGs
This is the third in a series of DAGs showing an EL pipeline with data integrity and data quality checking for data in S3 and Redshift using ETag verification and row-based data quality checks where t…
Example DAG that queries the Airflow Metadata Database and moves data through GCS to Postgres.
