HivePartitionSensorAsync

Astronomer ProvidersCertified

Waits for a given partition to show up in Hive table asynchronously.

View on GitHub

Access Instructions

Install the Astronomer Providers provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

Documentation

Parameters

tablethe table where the partition is present.

partitionThe partition clause to wait for. This is passed as notation as in “ds=’2015-01-01’”

schemadatabase which needs to be connected in hive. By default, it is ‘default’

metastore_conn_idconnection string to connect to hive.

polling_intervalThe interval in seconds to wait between checks for partition.

Documentation

Waits for a given partition to show up in Hive table asynchronously.

Note

HivePartitionSensorAsync uses impyla library instead of PyHive. The sync version of this sensor uses PyHive <https://github.com/dropbox/PyHive>.

Since we use impyla library, please set the connection to use the port 10000 instead of 9083. For auth_mechanism='GSSAPI' the ticket renewal happens through command airflow kerberos in worker/trigger.

You may also need to allow traffic from Airflow worker/Triggerer to the Hive instance, depending on where they are running. For example, you might consider adding an entry in the etc/hosts file present in the Airflow worker/Triggerer, which maps the EMR Master node Public IP Address to its Private DNS Name to allow the network traffic.

The library version of hive and hadoop in Dockerfile should match the remote cluster where they are running.