Configure data notifications

This page describes how to configure data notifications with Semarchy xDM.

Prepare the named query

Data notification messages are structured after named queries. Data propagated by a data notification is filtered using the related named query’s filter and named query parameters.

To filter records created or updated within a range of batch IDs, the named query must include the following elements:
- A filter having two parameters—minimum batch ID and maximum batch ID—to define the range of batch IDs. A typical named query filter for data notifications looks as follows:
  BatchID > :QUERY_PARAM_PREVIOUS_BATCH and BatchID <= :QUERY_PARAM_CURRENT_BATCH
- Two named query parameters for minimum batch ID and maximum batch ID used in the filter:
  - CURRENT_BATCH (number) using the binding name QUERY_PARAM_CURRENT_BATCH.
  - PREVIOUS_BATCH (number), using the binding name QUERY_PARAM_PREVIOUS_BATCH.
To filter data depending on the date and time when integration batches ran, the following named query parameters can be added to the named query and used in its filter:
- CURRENT_BATCH_TIMESTAMP (timestamp) using the binding name QUERY_PARAM_CURRENT_BATCH_TIMESTAMP.
- PREVIOUS_BATCH_TIMESTAMP (timestamp), using the binding name QUERY_PARAM_PREVIOUS_BATCH_TIMESTAMP.
To filter data depending on the date and time when the data notification is triggered, the following named query parameter can be added to the named query and used in its filter:
- NOTIFICATION_TIMESTAMP (timestamp) using the binding name QUERY_PARAM_NOTIFICATION_TIMESTAMP.

Create a data notification

Data notifications are configured within data locations. To create a data notification:

In the Management view, expand the Data Locations node, and then expand the data location for which you want to configure a data notification.
Right-click the Data Notifications node and click New Data Notification.
The Create New Data Notification wizard opens.
Enter the following information:
- Name: internal name of the data notification.
- Named Query: define a named query using the following steps:
  1. Click Edit to open the Select Named Query picker.
  2. Select the named query that will define the notification dataset content.
  3. Click OK to close the picker.
    
    To make a new named query visible in the Select Named Query picker, refresh your application (on a development instance) or redeploy the model (on a production instance).
- Target Type: select the type of channel meant to receive data notifications.
  There are three types of targets:
  - JMS (Java Message Service) queue or topic.
  - Kafka topic.
  - REST API endpoint.
Click Finish.
The Data Notification editor opens.
(Optional) Enter a description.
Configure the content of data notifications in the Dataset section of the editor:
1. (Optional) Set the Run as User property with the user required to resolve model variables from the named query.
2. Select the View Type to define the type of data sent in the notification. For more information, see View types.
  
  Not all view types are available for data notifications. For example, views bound to loads and batches are not listed.
3. Set the root records limit to restrain the number of records sent in a notification. If more records are retrieved from the named query, the notification will automatically fail.

Configure the named query parameters.
Parameters from the selected named query are listed in the Named Query Parameters table.

(Optional) Click Refresh to update the list of parameters if the named query was updated.

(Optional) Click Delete to remove parameters that you do not want to map.
For each parameter, set the following properties:

Parameter Type: type mapped to the named query parameter, which can be either:

Literal: the query parameter value is a literal text value.

Variable: the query parameter value is the evaluation of the provided variable. The following variables are available:

Variable name Definition

Variable name	Definition
`HIGH_BATCH_ID`	ID of the current batch running the integration job.
`HIGH_BATCH_TIMESTAMP`	Timestamp when the batch with `HIGH_BATCH_ID` was submitted.
`LOW_BATCH_ID`	ID of the previous batch that ran the integration job.
`LOW_BATCH_TIMESTAMP`	Timestamp when the batch with `LOW_BATCH_ID` was submitted.
`NOTIFICATION_TRIGGER_TIMESTAMP`	Timestamp when the data notification is triggered.

HIGH_BATCH_ID

ID of the current batch running the integration job.

HIGH_BATCH_TIMESTAMP

Timestamp when the batch with HIGH_BATCH_ID was submitted.

LOW_BATCH_ID

ID of the previous batch that ran the integration job.

LOW_BATCH_TIMESTAMP

Timestamp when the batch with LOW_BATCH_ID was submitted.

NOTIFICATION_TRIGGER_TIMESTAMP

Timestamp when the data notification is triggered.

Parameter Value:
- If Parameter Type is set to Literal: populate with the literal value to use.
- If Parameter Type is set to Variable: populate with the name of the variable.

To only return records created or updated within a range of batch IDs, a configuration needs to map the following named query parameters with built-in variables:

CURRENT_BATCH: set with the HIGH_BATCH_ID built-in variable that contains the highest batch ID in the data location.
PREVIOUS_BATCH: set with the LOW_BATCH_ID variable that contains the current value of the low batch ID.

The CURRENT_BATCH and PREVIOUS_BATCH named query parameters are named following the named query prerequisites.

To filter a named query depending on batch execution timestamps, the following built-in variables are available and can be mapped with named query parameters (see named query prerequisites):

PREVIOUS_BATCH_TIMESTAMP: contains the timestamp of the previous recent batch (identified by the low batch ID).
CURRENT_BATCH_TIMESTAMP: contains the timestamp of the most recent batch (identified by the high batch ID).

To filter a named query depending on the data notification firing timestamp, the built-in NOTIFICATION_TRIGGER_TIMESTAMP variable is available and can be mapped with a named query parameter. It provides the timestamp when the current data notification instance was triggered.

Configure the Trigger.
The following data notification triggers are available:
- On schedule: activates data notifications at predefined times using a cron expression.
  To configure the on-schedule trigger:
  1. Set the Trigger on property to Schedule.
  2. Click Edit next to the Cron Expression property to open the Cron Schedule dialog.
  3. Select a predefined frequency or select Cron Expression to enter a custom expression, and then click OK.
- On batch completion: activates data notifications immediately after an integration job completes.
  To configure the on-batch-completion trigger:
  1. Set the Trigger on property to Batch complete.
  2. (Optional) Click Edit to open the Watched Entities selector. The data notification will be triggered only if the batch processes the selected entities. Leave this property empty to watch all entities.

Configure the notification messages in the Message section:
- Records Per Message: number of records per message. If the total number of records returned by the named query is higher than this value, the query results will be split over multiple messages. An empty value means that all records will be sent in one message (default: empty).
- Notify On Empty Dataset: defines whether a notification should be sent if the named query returns no result (default: unselected).
  
  Selecting this option is highly recommended when testing connectivity so that a notification is sent even for an empty dataset.
- Max Retries: maximum number of retries in case of error when sending messages to the target. After all retries, the notification is considered failed and future notifications are automatically suspended (default: 1).
- Retry Interval: Duration in seconds between two retries (default: 60).

Set Target parameters.
Depending on the selected target, the following properties are editable:
- For JMS
  - Connection Factory: name of the object used by the JMS client to create a connection with a JNDI provider. The default value is ConnectionFactory`.
  - JMS Destination: name of the JMS queue or topic the notification messages must be sent to. Prefix the name with queue/ or topic/ depending on the destination type. If no prefix is provided, queue/ is assumed.
  - JMS Username: name of the user required to connect to the JMS destination. Leave empty if no authentication is required by the broker (default: empty).
  - JMS Password: password required to connect to the JMS destination. Leave empty if no authentication is required by the broker (default: empty).
- For Kafka
  - Kafka Topic: name of the Kafka topic to which notification messages must be sent.
  - Kafka Message Key: key to use for Kafka messages (if empty, the name of the data notification is used).
    
    The URL of the Kafka broker and other connection parameters are set in the Target Properties section.
- For REST
  - REST Request URL: URL of the REST endpoint. The value must include HTTP/S.
  - REST Timeout: timeout for the HTTP request, in seconds. After this timeout, the notification is considered failed and future notifications are automatically suspended.

Configure target properties: to cover all technical requirements of the middleware or downstream applications to which the notification messages are sent, additional properties can be provided in the Target Properties section using key-value pairs.

For a REST target, these properties are passed as the HTTP headers of the request.
Kafka and JMS targets use these properties to configure the connection.

To add a property:

Click the Add Target Property button.
A new line is added to the Target Properties table.
Enter the Property Name (name of the JMS/Kafka property or HTTP header) or click the Select Property button to select the property within a list of predefined keys per target type.

Enter the Property Value. If required (e.g., in the case of a password), this value can be transformed into a secret by clicking the Make Secret button.

If the target of the notification is a Kafka topic, you must configure the broker’s URL by adding a property with the predefined key bootstrap.servers. For more information, see Configure data notifications to a Kafka target.

Validate and save.
The configuration of a data notification is validated every time it is saved or when the user clicks the Validate button in the editor. A data notification cannot be saved until all errors are fixed.
Test and prepare the data notification for execution.
Once the data notification is saved, the following actions are available:
- Send Manual Notification: sends a data notification on a defined range of batch IDs, for test or replay purposes.
- Set Low Batch ID: allows to set the value of the LOW_BATCH_ID variable.
- Activate: activates the data notification trigger.

Configure data notifications

Configure data notifications to a JMS target

The instructions provided in this section are tailored for Tomcat. If you are using a different application server such as WildFly or Jetty, follow these guidelines:

For WildFly, the target directory for external libraries is <wildfly_home>/standalone/deployments. Copy the necessary JAR files into this directory instead of tomcat/lib.
For Jetty, the target directory for external libraries is <jetty_home>/lib/ext. Copy the necessary JAR files into this directory instead of tomcat/lib.

If you are using a different application server, refer to its installation and configuration documentation for the specific location to place external libraries and adjust the steps accordingly.

Using ActiveMQ

Install and configure ActiveMQ Server

Download ActiveMQ 5.18.3 from the Apache ActiveMQ website.
Install your ActiveMQ distribution. For detailed instructions, see the official ActiveMQ documentation.
Copy the following JAR files from the /lib folder of your ActiveMQ distribution and paste them into your tomcat/lib folder:
- hawtbuf-1.11.jar
- slf4j-api-2.0.9.jar
Download the Jakarta client bundle file.
Copy the activemq-client-jakarta-5.18.3.jar file and paste it into your tomcat/lib folder.
Copy the jakarta.jms-api-3.1.0.jar file from the additional-libraries folder of your Semarchy xDM installation package and paste it into your tomcat/lib folder.
Restart Tomcat (or your application server) to ensure the copied libraries are properly loaded.

xDM versions 2023.4 and later use a Jakarta client version 5.18.X.
For an xDM version earlier than 2023.4, follow steps 1 and 2 described above, and proceed as follows:

Copy the following JAR files from the /lib folder of your ActiveMQ distribution and paste them into your tomcat/lib folder:
- hawtbuf-1.11.jar
- slf4j-api-2.0.9.jar
- activemq-client-5.18.3.jar
Download Jakarta 2.0.
Copy the jakarta.jms-api-2.0.3.jar file and paste it into your tomcat/lib folder.
Restart Tomcat (or your application server) to ensure the copied libraries are properly loaded.

For more detailed, version-specific instructions involving a different ActiveMQ distribution, contact our Technical Support team.

Set up a data notification

Once you have your ActiveMQ client installed and configured, you can set up a data notification.

In xDM, create a data notification using JMS.
In your JMS configuration, modify the target property JMS Destination to <your_consumer_queue_name>.
Add the following target properties:
- java.naming.factory.initial = org.apache.activemq.jndi.ActiveMQInitialContextFactory
- java.naming.provider.url = failover://tcp://localhost:61616

Save and send a manual notification.
You should see the notification in your ActiveMQ consumer console. If not, expand your user account dropdown menu in the Application Builder and select Error Log to check for a potential error.

Ensure driver compatibility and be aware of potential breaking changes with Jakarta 3.X, especially for external applications such as Tibco.

Using Azure Service Bus

Install and configure Azure Service Bus

Copy the jakarta.jms-api-3.1.0.jar file from the additional-libraries folder of your Semarchy xDM installation package and paste it into your tomcat/lib folder.
From the Maven Repository website, download and install the following JAR files in your tomcat/lib folder:
- netty-buffer-4.1.100.Final.jar
- netty-codec-4.1.100.Final.jar
- netty-codec-http-4.1.100.Final.jar
- netty-common-4.1.100.Final.jar
- netty-handler-4.1.100.Final.jar
- netty-resolver-4.1.100.Final.jar
- netty-transport-4.1.100.Final.jar
- netty-transport-classes-epoll-4.1.100.Final.jar
- netty-transport-classes-kqueue-4.1.100.Final.jar
- netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar
- netty-transport-native-kqueue-4.1.100.Final-osx-x86_64.jar
- netty-transport-native-unix-common-4.1.100.Final.jar
- proton-j-0.34.1.jar
- qpid-jms-client-2.5.0.jar
Restart Tomcat (or your application server) to ensure the copied libraries are properly loaded.
Sign in to the Microsoft Azure portal.
Create a Service Bus queue. For detailed instructions, see the official Azure documentation.
Be sure to note your shared access policy name and its primary key. You will need these two variables later.

Set up a data notification

Once you have your Service Bus client set up, you can configure a data notification.

In xDM, create a data notification using JMS.
In your JMS configuration, modify JMS Destination to <your_queue_name>.
In the Target Properties table, add the following properties:
- connectionfactory.ConnectionFactory = amqps://<your_namespace>.servicebus.windows.net
- java.naming.factory.initial = org.apache.qpid.jms.jndi.JmsInitialContextFactory
- java.naming.provider.url = amqps://<your_namespace>.servicebus.windows.net
- Username = <your_policy_name>
- Password = <your_primary_key>

While not essential for establishing a connection, the java.naming.provider.url property is crucial for ensuring a valid JMS configuration.

Save and send a manual notification.
Perform a peek operation on your Service Bus queue to view your message. For more information on browsing or peeking messages, see the official Azure documentation.

Configure data notifications to a Kafka target

Install and configure Kafka

Download the latest Kafka release from the official Apache Kafka website.
Install your Kafka distribution and create a topic to store your events. For detailed instructions, see the official Kafka documentation.

Set up a data notification

Once you have your Kafka client set up, you can configure a data notification.

In xDM, create a data notification using Kafka.
In the Kafka Topic field, enter the name of the topic to be consumed by xDM.
In the Target Properties table, add the following mandatory property:

bootstrap.servers=<broker>:<port>

Replace <broker> and <port> with the hostname (or IP address) and port number of your Kafka broker (e.g., localhost:9092).

(Optional) Configure authentication.
Kafka supports multiple authentication mechanisms, such as SSL, SASL/PLAIN, SASL/SCRAM, and more. To configure the appropriate authentication properties, see your Kafka broker’s documentation.
Example configuration with SASL/PLAIN authentication
The following configuration demonstrates how to authenticate a Kafka client with SASL/PLAIN, commonly used in Kafka deployments, such as Confluent Kafka.

In the Target Properties table, add the following properties:
- sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='<API_key>' password='<API_secret>'
  
  Replace <API_key> and <API_secret> with your actual Kafka credentials.
- sasl.mechanism=PLAIN
- security.protocol=SASL_SSL
Save your configuration.

Configure data notifications using the REST API

Endpoints are available on the Semarchy xDM REST API to consult and configure data notifications.

For more details, see the REST API documentation.