NamedHivePartitionSensorAsync

Astronomer ProvidersCertified

Waits asynchronously for a set of partitions to show up in Hive.

View on GitHub

Access Instructions

Install the Astronomer Providers provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

Documentation

Parameters

partition_namesList of fully qualified names of the partitions to wait for. A fully qualified name is of the form schema.table/pk1=pv1/pk2=pv2, for example, default.users/ds=2016-01-01.

metastore_conn_idMetastore thrift service connection id.

Documentation

Waits asynchronously for a set of partitions to show up in Hive.

Note

HivePartitionSensorAsync uses impyla library instead of PyHive. The sync version of this sensor uses PyHive <https://github.com/dropbox/PyHive>.

Since we use impyla library, please set the connection to use the port 10000 instead of 9083. For auth_mechanism='GSSAPI' the ticket renewal happens through command airflow kerberos in worker/trigger.

You may also need to allow traffic from Airflow worker/Triggerer to the Hive instance, depending on where they are running. For example, you might consider adding an entry in the etc/hosts file present in the Airflow worker/Triggerer, which maps the EMR Master node Public IP Address to its Private DNS Name to allow the network traffic.

The library version of hive and hadoop in Dockerfile should match the remote cluster where they are running.