This is documentation for Semarchy xDI 2024.4, which is no longer actively maintained.

For more information, see our Global Support and Maintenance Policy.

Kafka component

Overview

The Kafka component lets you create Kafka integration flows in Semarchy xDI.

Install the Kafka component

If you did not install it yet, install the Kafka component in Designer by following the component installation process.

Supported features

Feature

Description

Metadata

Kafka producers and consumers can be designed in Metadata.

Consume messages

Reading messages from Kafka consumers is supported.

The following common features are supported:

Reading messages keys
Reading messages values
Reading messages headers

The following advanced features are supported:

Reading messages properties such as the topic it is coming from, the corresponding partition, offset, …
Reading messages from multiple topics at the same time
Reading multiple messages in parallel
Transactional mode to read messages in a transaction

When reading messages, you can write the received messages in any database, file, …

Produce messages

Writing data into Kafka through Kafka Producers is supported.

The following common features are supported when producing messages:

Defining message key
Defining message value
Defining message headers

The following advanced features are supported when producing messages:

Defining message partition
Producing messages in multiple topics at the same time
Producing multiple messages in parallel
Transactional mode to produce messages in a transaction

You can produce messages from any database, file, …

Data consultation

Consulting and consuming messages directly from xDI Designer is supported.

Kafka versions

Supported Kafka versions: 2.6 to 3.2

Notes

In the Modules Management window, a Kafka module may show a duplicate library warning. You can ignore this warning, as the module needs all copies of the libraries.

Kafka modules management page with warning showing