Kafka Component

Overview

Semarchy xDI allows to work with Kafka to produce fully customized Data Flows.

Install the Kafka Component

The Kafka Component is installed from Semarchy xDI Designer using the component installation feature.

Supported Features

Feature Description

Metadata

Kafka producers and consumers can be designed in Metadata.

Consume messages

Reading messages from Kafka consumers is supported.

The following common features are supported:

  • Reading messages keys

  • Reading messages values

  • Reading messages headers

The following advanced features are supported:

  • Reading messages properties such as the topic it is coming from, the corresponding partition, offset, …​

  • Reading messages from multiple topics at the same time

  • Reading multiple messages in parallel

  • Transactional mode to read messages in a transaction

When reading messages, you can write the received messages in any database, file, …​

Produce messages

Writing data into Kafka through Kafka Producers is supported.

The following common features are supported when producing messages:

  • Defining message key

  • Defining message value

  • Defining message headers

The following advanced features are supported when producing messages:

  • Defining message partition

  • Producing messages in multiple topics at the same time

  • Producing multiple messages in parallel

  • Transactional mode to produce messages in a transaction

You can produce messages from any database, file, …​

Data consultation

Consulting and consuming messages directly from Semarchy xDI Designer is supported.

Kafka versions

Supported Kafka versions: 2.6 to 3.2