Kafka component release notes

This page lists the main features added to the Kafka component.

Changelog

Version 2024.4.0

Feature improvements

  • DI-10248: Upgraded component libraries to the latest versions.

Bug fixes

  • DI-10138: Updated ClassGraph third-party libraries.

  • DI-10586: Fixed an issue that blocked collecting header data from a Kafka topic when the header contains hyphens.

Version 2024.3.0

Breaking changes

Previous versions of the Apache Kafka connector had a bug that incorrectly added an extra schema or root node to consumed or produced JSON structures.

The fixed connector no longer adds the incorrect extra node. However, any applications that implemented a workaround for the wrong behavior should remove their workaround.

Feature improvements

  • DI-7970: Added the ability to reset the offset related to a consumer group.

  • DI-9595: Updated main component libraries to the latest versions.

Bug fixes

  • DI-8662: Template Load Kafka Structured to Rdbms: fixed an issue that added an extra root or schema node to consumed and produced JSON messages. The fixed connector consumes and produces messages without adding an extra node.

  • DI-9664: Updated Apache ZooKeeper third-party libraries.

  • DI-9936: Fixed an issue that prevented xDI from consuming Avro structured messages.

  • DI-10187: Fixed missing datatypes for Kafka timestamp fields when loading them to an RDBMS stage.

Version 2024.2.0

Bug fixes

  • DI-8304: Updated netty-handler third-party libraries.

  • DI-8440: Cleaned up XML parser declarations in templates.

  • DI-8619: Updated netty-handler third-party libraries.

  • DI-8824: Updated Apache ZooKeeper third-party libraries.

Version 2024.1.10

Bug fixes

  • DI-9718: Updated Netty third-party libraries.

  • DI-10220: Updated jose4j third-party libraries.

2024.1.9

Breaking changes

Previous versions of the Apache Kafka connector had a bug that incorrectly added an extra schema or root node to consumed or produced JSON structures.

The fixed connector no longer adds the incorrect extra node. However, any applications that implemented a workaround for the wrong behavior should remove their workaround.

Bug fixes

  • DI-8662: Template Load Kafka Structured to Rdbms: fixed an issue that added an extra root or schema node to consumed and produced JSON messages. The fixed connector consumes and produces messages without adding an extra node.

  • DI-10187: Fixed missing datatypes for Kafka timestamp fields when loading them to an RDBMS stage.

Version 2024.1.8

Bug fixes

  • DI-9664: Updated Apache ZooKeeper third-party libraries.

Version 2024.1.7

Bug fixes

  • DI-9936: Fixed an issue that prevented xDI from consuming Avro structured messages.

Version 2024.1.1

Bug fixes

  • DI-8824: Update Apache ZooKeeper third-party libraries.

Version 2024.1.0

Feature improvements

  • DI-6169: Support External Value Resolver in Kafka raw and Kafka structured.

  • DI-8589: Update Kafka backend to the latest version.

Bug fixes

  • DI-8759: Update Apache Avro third-party libraries.

  • DI-8830: Update JSON third-party libraries.

  • DI-8861: Temporary tables are not cleaned up correctly at the end of a process.

Version 2023.4.0

Feature improvements

  • DI-8190: Third-party library upgrade.

Version 2023.2.0

Bug fixes

  • DI-6324: When using Kafka structured as source, the Load Kafka Structure to RDBMS template does not create the Load_Temporal table and an insert error occurs at the integration step.

Version 2023.1.18

Bug fixes

  • DI-9718: Updated Netty third-party libraries.

  • DI-10220: Updated jose4j third-party libraries.

Version 2023.1.17

Breaking changes

Previous versions of the Apache Kafka connector had a bug that incorrectly added an extra schema or root node to consumed or produced JSON structures.

The fixed connector no longer adds the incorrect extra node. However, any applications that implemented a workaround for the wrong behavior should remove their workaround.

Bug fixes

  • DI-8662: Template Load Kafka Structured to Rdbms: fixed an issue that added an extra root or schema node to consumed and produced JSON messages. The fixed connector consumes and produces messages without adding an extra node.

  • DI-10187: Fixed missing datatypes for Kafka timestamp fields when loading them to an RDBMS stage.

Version 2023.1.16

Bug fixes

  • DI-9664: Updated Apache ZooKeeper third-party libraries.

Version 2023.1.13

Feature improvements

  • DI-9620: Added Semarchy Data Intelligence harvesting features.

Version 2023.1.9

Bug fixes

  • DI-8824: Update Apache ZooKeeper third-party libraries.

Version 2023.1.8

Bug fixes

  • DI-8759: Update Apache Avro third-party libraries.

  • DI-8830: Update JSON third-party libraries.

  • DI-8861: Temporary tables are not cleaned up correctly at the end of a process.

Version 2023.1.5 (component pack)

Bug fixes

  • DI-8220: Third-party library upgrade.

  • DI-7989: Third-party library upgrade.

  • DI-7908: Third-party library upgrade.

Version 2023.1.2 (component pack)

Bug fixes

  • DI-6324: When using Kafka structured as source, the Load Kafka Structure to RDBMS template does not create the Load_Temporal table and an insert error occurs at the integration step.

Version 2023.1.0 (component pack)

Feature improvements

  • DI-4661: The Create indexes on Target Table parameter has been added to the INTEGRATION Kafka to Rdbms template.

  • DI-6300: The Kafka 3.2 is now supported.

Bug fixes

  • DI-6298: Multiple third-party libraries upgrade.

  • DI-6316: SnakeYAML - Third-party library upgrade.

  • DI-6520: Jackson - Third-party library upgrade.

Version 5.3.8 (Component pack)

Bug fixes

  • DI-6304: When using Kafka structured as source and an RDBMS table as stage target, the load template fails to load data into the stage as well as into the temporary load tables.

Version 3.0.0 (component pack)

Feature improvements

  • DI-3701: Allow Components to contribute to Designer monitored statistics

  • DI-4053: Query Editor menu renamed to "Launch Query Editor"

  • DI-4508: Update Components and Designer to take into account dedicated license permissions

  • DI-4813: Rebranding: Drivers classes and URLs

  • DI-4962: Improved component dependencies and requirements management

Version 2.0.2 (Kafka component)

Feature improvements

  • DI-1298: INTEGRATION Kafka to File Template - Addition of the "Batch Size" parameter to define how many messages must be read from the topic before writing to the file

  • DI-1940: Support synchronous mode when sending messages (new attribute and parameter in Metadata and Templates are available to define the write mode)

  • DI-1909: Templates updated - New Parameters 'Unlock Cdc Table' and 'Lock Cdc Table' to configure the behaviour of CDC tables locking

Bug fixes

  • DI-1913: An error was thrown when trying to connect to a secure Kafka server, such as "KafkaException: javax.security.auth.login.LoginException: unable to find LoginModule class: org.apache.kafka.common.security.plain.PlainLoginModule"

  • DI-2162: The Component was not working on Topics having a dot in the name

  • DI-2658: Kafka Metadata - 'connect to database' context menu was not working

  • DI-1908: Templates updated - The 'Cdc Subscriber' parameter was ignored in some Templates on Lock / Unlock CDC steps

  • DI-1907: Templates updated - The 'Cdc Subscriber' parameter was ignored in some Templates when querying the source data

Version 2.1.0 (Kafka component)

Feature improvements

  • DI-3713: Internal change on how some libraries are built to ease maintenance