GCSTaskHandler

Google

GCSTaskHandler is a python log handler that handles and reads task instance logs. It extends airflow FileTaskHandler and uploads to and reads from GCS remote storage. Upon log reading failure, it reads from host machine’s local disk.

View on GitHub

Last Updated: Mar. 7, 2023

Access Instructions

Install the Google provider package into your Airflow environment.

Import the module into your DAG file and instantiate it with your desired params.

Parameters

base_log_folderRequiredBase log folder to place logs.
gcs_log_folderRequiredPath to a remote location where logs will be saved. It must have the prefix gs://. For example: gs://bucket/remote/log/location
filename_templatetemplate filename string
gcp_key_pathPath to Google Cloud Service Account file (JSON). Mutually exclusive with gcp_keyfile_dict. If omitted, authorization based on the Application Default Credentials will be used.
gcp_keyfile_dictDictionary of keyfile parameters. Mutually exclusive with gcp_key_path.
gcp_scopesComma-separated string containing OAuth2 scopes
project_idProject ID to read the secrets from. If not passed, the project ID from credentials will be used.

Documentation

GCSTaskHandler is a python log handler that handles and reads task instance logs. It extends airflow FileTaskHandler and uploads to and reads from GCS remote storage. Upon log reading failure, it reads from host machine’s local disk.

Was this page helpful?