Astronomer Providers

OfficialData Storage Databases Containers Public Cloud Web Service

A library of Apache Airflow providers containing Deferrable Operators & Sensors maintained by Astronomer.

Version

1.15.2

Downloads

17,904/month

Last Published

Mar. 27, 2023

Quick Install


<title>Astronomer Providers &#8212; Astronomer Providers 1.18.4 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=4f649999" />
<link rel="stylesheet" type="text/css" href="_static/alabaster.css?v=039e1c02" />
<link rel="stylesheet" type="text/css" href="_static/graphviz.css?v=eafc0fe6" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" />
<link rel="stylesheet" type="text/css" href="/_/static/css/badge_only.css" />
<script src="_static/documentation_options.js?v=d91df534"></script>
<script src="_static/doctools.js?v=888ff710"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/clipboard.min.js?v=a7894cd8"></script>
<script src="_static/copybutton.js?v=f281be69"></script>
<script async="async" src="/_/static/javascript/readthedocs-doc-embed.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="API Reference" href="_api/index.html" />

{"ad_free": false, "api_host": "https://readthedocs.org", "builder": "sphinx", "canonical_url": null, "docroot": "/docs/", "features": {"docsearch_disabled": false}, "global_analytics_code": null, "language": "en", "page": "index", "programming_language": "py", "project": "astronomer-providers", "proxied_api_host": "/_", "source_suffix": ".rst", "subprojects": {}, "theme": "alabaster", "user_analytics_code": "", "version": "stable"} READTHEDOCS_DATA = JSON.parse(document.getElementById('READTHEDOCS_DATA').innerHTML); -->


<div class="document">
  <div class="documentwrapper">
    <div class="bodywrapper">
      

      <div class="body" role="main">

Astronomer Providers¶ Apache Airflow Providers containing Deferrable Operators & Sensors from Astronomer. Installation¶ Install and update using pip: pip install astronomer-providers This only installs dependencies for core provider. To install all dependencies, run: pip install 'astronomer-providers[all]' To only install the dependencies for a specific provider, specify the integration name as extra argument, example to install Kubernetes provider dependencies, run: pip install 'astronomer-providers[cncf.kubernetes]' Extras¶ Extra Name Installation Command Dependencies all pip install 'astronomer-providers[all]' All amazon pip install 'astronomer-providers[amazon]' Amazon apache.hive pip install 'astronomer-providers[apache.hive]' Apache Hive apache.livy pip install 'astronomer-providers[apache.livy]' Apache Livy cncf.kubernetes pip install 'astronomer-providers[cncf.kubernetes]' Cncf Kubernetes databricks pip install 'astronomer-providers[databricks]' Databricks dbt.cloud pip install 'astronomer-providers[dbt.cloud]' Dbt Cloud google pip install 'astronomer-providers[google]' Google http pip install 'astronomer-providers[http]' Http microsoft.azure pip install 'astronomer-providers[microsoft.azure]' Microsoft Azure openlineage pip install 'astronomer-providers[openlineage]' Openlineage sftp pip install 'astronomer-providers[sftp]' Sftp snowflake pip install 'astronomer-providers[snowflake]' Snowflake Example Usage¶ This repo is structured same as the Apache Airflow’s source code, so for example if you want to import Async operators, you can import it as follows: from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync as S3KeySensor

waiting_for_s3_key = S3KeySensor( task_id="waiting_for_s3_key", bucket_key="sample_key.txt", wildcard_match=False, bucket_name="sample-bucket", )

Example DAGs for each provider is within the respective provider’s folder. For example, the Kubernetes provider’s DAGs are within the astronomer/providers/cncf/kubernetes/example_dags folder. Principle¶ We will only create Async operators for the “sync-version” of operators that do some level of polling (take more than a few seconds to complete). For example, we won’t create an async Operator for a BigQueryCreateEmptyTableOperator but will create one for BigQueryInsertJobOperator that actually runs queries and can take hours in the worst case for task completion. To create async operators, we need to inherit from the corresponding airflow sync operators. If sync version isn’t available, then inherit from airflow BaseOperator. To create async sensors, we need to inherit from the corresponding sync sensors. If sync version isn’t available, then inherit from airflow BaseSensorOperator. Changelog¶ We follow Semantic Versioning for releases. Check CHANGELOG.rst for the latest changes. Contributing Guide¶ All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome. A detailed overview on how to contribute can be found in the Contributing Guide. As contributors and maintainers to this project, you are expected to abide by the Contributor Code of Conduct. Goals for the project¶ Our focus is on the speed of iteration and development in this stage of the project and so we want to be able to quickly iterate with our community members and customers and cut releases as necessary Airflow Providers are separate packages from the core apache-airflow package and we would like to avoid further bloating the Airflow repo We want users and the community to be able to easily track features and the roadmap for individual providers that we develop We would love to see the Airflow community members create, maintain and share their providers to build an Ecosystem of Providers. Limitations¶ In Airflow sensors have a param mode which can be poke and reschedule. In async sensors, this param has no usage since tasks gets deferred to Triggerer. License¶ Apache License 2.0 Operators and Sensors¶ Example¶ Changelog¶ For a list of all releases: Changelog 1.18.4 (2023-12-07) 1.18.3 (2023-11-29) 1.18.2 (2023-11-11) 1.18.1 (2023-11-01) 1.18.0 (2023-09-25) 1.17.3 (2023-08-07) 1.17.2 (2023-07-26) 1.17.1 (2023-06-22) 1.17.0 (2023-06-21) 1.16.0 (2023-05-19) 1.15.5 (2023-04-24) 1.15.4 (2023-04-19) 1.15.3 (2023-04-17) 1.15.2 (2023-03-27) 1.15.1 (2023-03-09) 1.15.0 (2023-02-15) 1.14.0 (2023-01-09) 1.13.0 (2022-12-16) 1.12.0 (2022-12-02) 1.11.2 (2022-11-19) 1.11.1 (2022-10-28) 1.11.0 (2022-10-21) 1.10.0 (2022-09-30) 1.9.0 (2022-09-13) 1.8.1 (2022-09-01) 1.8.0 (2022-08-16) 1.7.1 (2022-07-25) 1.7.0 (2022-07-19) 1.6.0 (2022-06-28) 1.5.0 (2022-06-15) 1.4.0 (2022-05-25) 1.3.1 (2022-05-22) 1.3.0 (2022-05-09) 1.2.0 (2022-04-12) 1.1.0 (2022-03-23) 1.0.0 (2022-03-01) Indices and tables¶ Index Module Index Search Page


      </div>
      
    </div>
  </div>
  <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
    <div class="sphinxsidebarwrapper">

Astronomer ProvidersAirflow Providers containing Deferrable Operators & Sensors from Astronomer Navigation Home Python API Reference Operators and Sensors Async Example DAG Changelog Related Topics Documentation overview Next: API Reference Quick search document.getElementById('searchbox').style.display = "block"


    </div>
  </div>
  <div class="clearer"></div>
</div>
<div class="footer">
  &copy;.
  
  |
  Powered by <a href="http://sphinx-doc.org/">Sphinx 7.2.6</a>
  &amp; <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.13</a>
  
  |
  <a href="_sources/index.rst.txt"
      rel="nofollow">Page source</a>
</div>

Astronomer Providers

Version

Downloads

Last Published

Quick Install

Helpful Links

Product

Resources

Company