Kafka component

Overview

The Kafka component lets you create Kafka integration flows in Semarchy xDI.

Install the Kafka component

If you did not install it yet, install the Kafka component in Designer by following the component installation process.

Supported features

Feature Description

Metadata

Kafka producers and consumers can be designed in Metadata.

Consume messages

Reading messages from Kafka consumers is supported.

The following common features are supported:

  • Reading messages keys

  • Reading messages values

  • Reading messages headers

The following advanced features are supported:

  • Reading messages properties such as the topic it is coming from, the corresponding partition, offset, …​

  • Reading messages from multiple topics at the same time

  • Reading multiple messages in parallel

  • Transactional mode to read messages in a transaction

When reading messages, you can write the received messages in any database, file, …​

Produce messages

Writing data into Kafka through Kafka Producers is supported.

The following common features are supported when producing messages:

  • Defining message key

  • Defining message value

  • Defining message headers

The following advanced features are supported when producing messages:

  • Defining message partition

  • Producing messages in multiple topics at the same time

  • Producing multiple messages in parallel

  • Transactional mode to produce messages in a transaction

You can produce messages from any database, file, …​

Data consultation

Consulting and consuming messages directly from xDI Designer is supported.

Kafka versions

Supported Kafka versions: 2.6 to 3.2

Notes

In the Modules Management window, a Kafka module may show a duplicate library warning. You can ignore this warning, as the module needs all copies of the libraries.

Kafka modules management page with warning showing