Getting started with Apache Kafka Structured metadata

Overview

This article describes how to read and write structured data with Apache Kafka. The Semarchy xDI structured metadata allows exchanging data in a specific format (for example, AVRO or JSON).

To exchange data with no specific format, use the Semarchy xDI raw metadata. Refer to Getting started with Kafka Raw metadata for more information.

Create metadata

To create the metadata:

Right-click the project folder, then select New > Metadata.
In the New Metadata window, select Kafka Server - structured and click Next.
Name the data model and click Next.
Select the installed Apache Kafka module and click Finish.
The metadata is created.

Define a host

Select the root node of the metadata to define the server properties listed below:

Table 1. Apache Kafka Server Properties
Property	Description
Name	A label/alias for this metadata.
Module	Apache Kafka module to use.
Connection Properties	Properties to be used for connection to kafka server, defined with the format: `<key>=<value>` and separated with line returns. The `boostrap.servers` property is mandatory. Specify it in the format `bootstrap.servers = <host>:<port>`.

The following image illustrates an example of a host:
getting started kafka structured host

Create topics

Topics are used to store data in Apache Kafka in the form of messages.
To create a topic:

Right-click the host node, then click New > Topic.
Enter a name and a physical name for the topic.
Right-click the topic, click New > Key.

Do not create more than one key on a topic.
In the Standard tab of the key properties, in the Schema field, drag and drop the JSON or the AVRO schema that the topic should represent. You can also use the browse button to browse through the available schemas.

Drag and drop or select only the root node of the JSON or AVRO schema.
Right-click the topic again, click New > Value.

Do not create more than one value on a topic.
In the Standard tab of the value properties, in the Schema field, drag and drop the JSON or the AVRO schema that the topic should represent. You can also use the browse button to browse through the available schemas.

Drag and drop or select only the root node of the JSON or AVRO schema.

getting started kafka structured value

Create consumers

Consumers subscribe to topics to read messages.
To create a consumer:

Right-click the host, then click New > Topic Consumer.
Enter a name and a group name for the consumer.
From the Topic drop-down list, select the topic to which the consumer should subscribe.

Create mappings

Mappings can be created for sending (produce) and receiving (consume) data.

Produce mapping

The following example illustrates a produce mapping in which a database table is mapped to a topic:
getting started kafka structured mapping produce

The topic node must be mapped to a field.

Consume mapping

The following example illustrates a consume mapping in which a consumer is mapped to a database table:
getting started kafka structured mapping consume

Development tips

When you create a consumer group, you can set the Auto Offset Reset to determine the read offset for the consumer group. However, this option only applies at the moment the group is created, which is when it is first used. It has no effect at other times.

If you want to reset a read offset for any consumer group, use the dedicated Kafka Reset Offsets process action in a process.