ProduceToTopicOperator
Apache KafkaProduceToTopicOperator An operator that produces messages to a Kafka topic
Access Instructions
Install the Apache Kafka provider package into your Airflow environment.
Import the module into your DAG file and instantiate it with your desired params.
Parameters
topicstr, optionalThe topic the producer should produce to, defaults to None
producer_functionUnion[str, Callable[…, Any]], optionalThe function that generates key/value pairs as messages for production, defaults to None
producer_function_argsOptional[Sequence[Any]], optionalAdditional arguments to be applied to the producer callable, defaults to None
producer_function_kwargsOptional[Dict[Any, Any]], optionalAdditional keyword arguments to be applied to the producer callable, defaults to None
delivery_callbackOptional[str], optionalThe callback to apply after delivery(or failure) of a message, defaults to None
kafka_conn_idOptional[str], optionalThe airflow connection to get brokers address from, defaults to None
synchronousOptional[bool], optionalIf writes to kafka should be fully synchronous, defaults to True
kafka_configOptional[Dict[Any, Any]], optionalthe config dictionary for the kafka client (additional information available on the confluent-python-kafka documentation), defaults to None
poll_timeoutfloat, optionalHow long of a delay should be applied when calling poll after production to kafka, defaults to 0
AirflowException
Documentation
ProduceToTopicOperator An operator that produces messages to a Kafka topic
Registers a producer to a kafka topic and publishes messages to the log.