Pre-upgrade actions

Before proceeding with the upgrade procedure, specific actions may be required based on the version of Semarchy xDM from which you are upgrading.

Carefully examine the changes and actions listed on this page, as they may have a significant impact on your configuration.
Make sure to read all the sub-sections on this page until you reach your current Semarchy xDM product version.

For all versions

Execution engine queue and load management

Failure to clear an execution engine queue prior to an upgrade can cause conflicts, interruptions, and performance degradation that may hinder the upgrade process and prolong downtime for users.

Required action

Platform administrators must ensure that all execution engine queues are empty and no loads are scheduled, pending, or running before initiating the upgrade process.

Browser support

Semarchy xDM no longer supports old web browsers. For the current list of supported web browsers, see System requirements.

Recommended action

Administrators should inform all Semarchy xDM users of the new client requirements.

Upgrade from versions before 2024.2.0

This section applies to all installations upgrading from a version prior to 2024.2.0.

Integration job parameter renaming

To clarify its scope of application, the integration job parameter previously called PARAM_RECYCLE_ERRORS—which activates pre-consolidation error recycling when enabled—​has been renamed to PARAM_RECYCLE_PRE_CONSOLIDATION_ERRORS. Existing jobs are automatically upgraded to reflect this renaming.

For more information on this parameter, see Job parameters reference.

Custom translation files backup

This version introduces additional localizable files into the custom translation archive, which has been reorganized accordingly. As part of this adjustment, some localization strings (mostly located in the former /commons and /commons2 folders) are moved to a new /shared-utilities folder.

Recommended action

To facilitate the re-translation of any string value that might be lost in the migration process, platform administrators should export their existing custom translation files before upgrading to version 2024.2.

Execution engine optimization and index handling

As a part of optimizing the execution engine’s performance by reducing the number of database queries needed to identify running loads in the continuous load process, an index is automatically created on MTA_INTEG_LOAD(LOADID) in version 2024.2.0.

Recommended action

To avoid potential conflicts, platform administrators who manually added a custom database index on MTA_INTEG_LOAD(LOADID) should remove it before upgrading to version 2024.2.0.

Default context root for Docker images

The default context root for Semarchy xDM images has been updated from /semarchy to /.

Recommended action

To prevent potential risks such as compromising SAML and OpenID authentication, disrupting REST API integrations, and breaking dashboard and end-user browser links, platform administrators should appropriately define the customizable CONTEXT_PATH parameter in the Docker image configuration file if they prefer accessing xDM via /semarchy or any other path.

Upgrade from versions before 2024.1.6

This section applies to all installations upgrading from a version prior to 2024.1.6.

Model cache loading at startup

Starting with version 2024.1.6, the model cache used by the probe API endpoints is loaded at startup to prevent the first call from returning a 503 error. This change may cause longer startup times for applications with large models.

Platform administrators can revert to the previous behavior by setting the system property xdm.api.rest.model.unsecured.cacheEagerLoad.enable to false, delaying the model cache loading until the first call to the probe API endpoints.

Upgrade from versions before 2024.1.3

This section applies to all installations upgrading from a version prior to 2024.1.3.

Default behavior of the allowXForwardedHeaders system property

For enhanced control and security, the default (false) behavior of the allowXForwardedHeaders system property, which previously ignored X-Forwarded headers, now removes them from HTTP requests.

In scenarios where intricate network configurations require retaining X-Forwarded headers for seamless communication between components, platform administrators are invited to reach out to our Technical Support team.

Execution engine performance optimization

The execution engine’s performance has been optimized by reducing the number of database queries required to identify running loads in the continuous load process.

Recommended action

For enhanced performance, platform administrators should create a custom database index on MTA_INTEG_LOAD(LOADID) to optimize queries that involve filtering or sorting by load ID.

Upgrade from versions before 2024.1.0

This section applies to all installations upgrading from a version prior to 2024.1.0.

Authentication mechanism

For security purposes, xDM enforces a temporary block on the user’s IP address for 24 hours following five consecutive failed login attempts. This security measure applies solely to users connecting to xDM through the built-in identity provider (i.e., internal IDP).

Platform administrators can either modify the maximum number of attempts by adjusting the value of the xdm.idm.maxloginattempts property, or deactivate this safeguard by setting the xdm.idm.maxloginattempts.enable property to false using the appropriate startup configuration method.

When accessed via a reverse proxy, xDM considers the IP provided in the X-Forwarded-For HTTP header for counting connection attempts.

Groovy script security update

This version introduces an allowlist mechanism for Groovy scripts and conditions, which limits the scope of permissible classes. A default allowlist ensures that only specified classes or methods can be utilized in the execution of Groovy scripts or the evaluation of conditions when configuring custom job notifications.

To modify the default allowlist without compromising security, platform designers are invited to contact our Technical Support team.

Duplicate index handling

This version adds an ix_integ_load_loadid native index on the loadid column in the MTA_INTEG_LOAD repository table to address contention and prevent deadlocks in SQL Server.

On Oracle databases, the xDM upgrade process attempts to drop any pre-existing user-created indexes and fails otherwise. However, the creation of the new index will not hinder the repository upgrade on other supported host databases—​namely SQL Server and PostgreSQL—​where duplicate indexes are permitted.

Recommended action

Due to complexities in index structure verification and the risk of unintentionally removing useful indexes with slightly different definitions, we recommend SQL Server and PostgreSQL database administrators check for true duplicates on the loadid column and execute the drop operation manually, rather than automatically dropping the index.

Upgrade from versions before 2023.4.0

This section applies to all installations upgrading from a version prior to 2023.4.0.

Infrastructure and system requirements

Starting from this version, PostgreSQL version 11 is no longer supported (end of life).

In line with Azure’s announcement of extended support for PostgreSQL 11 servers, Semarchy xDM will maintain its support for Azure Database for PostgreSQL 11 in Single Server until November 9, 2024.

External library update

This version includes updates to several third-party libraries.

Recommended action

Platform administrators should check the current versions of libraries used on their Tomcat server and replace, when appropriate, the ones that are outdated with the latest versions provided in the additional-libraries file of the xDM version 2023.4.0 server installation archive.

Extended maximum string size support

Starting from version 2023.4, designers can increase the maximum length of string entity attributes from 4,000 to 32,767 bytes. To enable this feature:

  • Platform administrators must set the com.semarchy.mdm.supportExtendedMaxStringSize system property to true.

    Once set to true, the com.semarchy.mdm.supportExtendedMaxStringSize property cannot be reverted to false.

  • Oracle database administrators should additionally set the MAX_STRING_SIZE parameter to EXTENDED in the Oracle database settings.

    Once set to EXTENDED, the MAX_STRING_SIZE parameter cannot be set back to STANDARD.

Upgrade from versions before 2023.3.0

This section applies to all installations upgrading from a version prior to 2023.3.0.

Infrastructure and system requirements

  • Starting from this version, Semarchy xDM does not support Java 11 anymore. Instances running on this version must upgrade to a supported version of Java Runtime Environment (JRE) or Java Development Kit (JDK) (see System requirements).

  • Starting from this version, the following application servers are no longer supported:

    • Tomcat 8.5

    • WildFly application server versions before 27.x

    • IBM Websphere Liberty (all versions)

    • Glassfish (all versions)

    • Oracle WebLogic (all versions)

  • Starting from this version, Semarchy xDM does not support Oracle Database 18c.

If your current infrastructure does not meet the new requirements, upgrade your infrastructure before upgrading Semarchy xDM.

Upgrade from versions before 2023.1.8

This section applies to all installations upgrading from a version prior to 2023.1.8.

Groovy script security update

This version introduces an allowlist mechanism for Groovy scripts and conditions, which limits the scope of permissible classes. A default allowlist ensures that only specified classes or methods can be utilized in the execution of Groovy scripts or the evaluation of conditions when configuring custom job notifications.

To modify the default allowlist without compromising security, platform administrators are invited to contact our Technical Support team.

Upgrade from versions before 2023.1.6

This section applies to all installations upgrading from a version prior to 2023.1.6.

Infrastructure and system requirements

Starting from this version, PostgreSQL version 11 is no longer supported (end of life).

In line with Azure’s announcement of extended support for PostgreSQL 11 servers, Semarchy xDM is scheduled to maintain its support for Azure Database for PostgreSQL 11 in Single Server until November 9, 2024.

Upgrade from versions before 2023.1.3

This section applies to all installations upgrading from a version prior to 2023.1.3.

X-Forwarded-For, Forwarded, and X-Real-IP HTTP headers usage

Starting from version 2023.1.3, the X-Forwarded-For, Forwarded, and X-Real-IP HTTP headers are ignored by default by Semarchy xDM to enforce security.

Required action

For infrastructures requiring to use one or several of these headers, administrators must set the allowXForwardedHeaders system property to true.

This configuration should be activated only when the xDM instance is exclusively accessible through a reverse proxy.

Upgrade from versions before 2023.1.0

This section applies to all installations upgrading from a version prior to 2023.1.0.

Infrastructure and system requirements

Starting from this version, Semarchy xDM does not support Java 8 anymore. Instances running on this version must upgrade to a supported version of Java Runtime Environment (JRE) or Java Development Kit (JDK) (see System requirements).

The following application servers are no longer supported as they do not support Java 11.

  • Glassfish versions before 6.x (6.1 is the minimum required version).

  • JBoss versions before 7.1 (WildFly 15 is the minimum required version).

If your current infrastructure does not meet the new requirements, upgrade your infrastructure before upgrading Semarchy xDM.

Upgrade from versions before 5.3.9

This section applies to all installations upgrading from a version prior to 5.3.9.

Logging

Starting from this version, the logging in Semarchy xDM relies on Log4j 2 and requires to migrate the existing startup and platform logging configurations to Log4j 2 format.

Required action

Before starting the upgrade, administrators must create a backup of the current platform logging configuration by opening Configuration  Logging Configuration and copying the entire content to a file. This backup will be used to create or amend the new Log4j 2 configuration.

Upgrade from versions before 5.3.4

This section applies to all installations upgrading from a version prior to 5.3.4.

Javax.json library for Tomcat

This release no longer includes the javax.json library to avoid possible conflicts with other libraries (JMS drivers) or application server-provided implementations. Javax.json is part of JEE 7, and application servers like JBoss, Glassfish, or Oracle WebLogic provide a built-in implementation of this library.

However, Tomcat does not provide a built-in implementation of javax.json. When deploying Semarchy xDM in an existing Tomcat server, make sure to copy the org.glassfish.jakarta.json_*.jar file from the mdm-server/additional-libraries/ folder to the Tomcat lib/ folder.

Upgrade from versions before 5.3.0

This section applies to all installations upgrading from a version prior to 5.3.0.

Single deployment per application server

This release introduces a new mechanism to bootstrap a startup configuration for Semarchy xDM, using environment properties and variables. Due to this startup configuration mechanism, it is no longer possible, starting with this version, to perform multiple deployments of Semarchy xDM in the same application server.

Recommended actions

Before starting the upgrade, administrators should review their infrastructure for possible Semarchy xDM deployments in the same application server instance, in different deployment contexts.

For instances having multiple deployments in the same application server, it is recommended to perform an out-of-place upgrade. The replicated environment should be configured to take into account this limitation.

Integrated datasources

This release introduces new built-in datasources to connect the backend databases, such as those hosting the repository and the data locations.

With this new mechanism, datasources are no longer configured at the application server level (in the semarchy.xml configuration file). They are configured in Semarchy xDM:

  • The repository connection (formerly SEMARCHY_REPOSITORY datasource) is now configured as two datasources, set in the startup configuration, via a configuration file or environment variables:

    • The Repository Datasource, which reproduces the SEMARCHY_REPOSITORY, with credentials allowing to read from and write into the repository schema.

    • A new Repository Read-Only Datasource, with credentials allowing to read a subset of the tables stored in the repository schema. This datasource uses a new dedicated user with read-only privileges to the repository.

  • Other datasources are now configured as platform datasource in the Semarchy xDM Configuration.

    • Administrators must re-configure these datasources after the repository upgrade based on their current application server configuration.

    • Components using datasources, including data locations, xDM Dashboard, xDM Discovery, the variable value providers, and the enrichment or validation plugins will refer to these datasources.

Required actions

Before starting the upgrade, administrators must:

The startup configuration is required to start the Semarchy xDM platform during the upgrade process.

Administrators should also gather connection information for all datasources used by Semarchy xDM to facilitate reconfiguring these datasources during the post-upgrade actions.

For a Tomcat installation, the original datasources are defined in the semarchy.xml file.

Integrated authentication

Semarchy xDM introduces in this release a new system to manage user authentication. With this new system:

  • Locally defined users authenticate with a built-in internal identity provider (IDP), which stores users, passwords, and their assigned roles in the repository.

    This IDP replaces users stored at the application server level, such as the UserDatabaseRealm (tomcat-users.xml file) and DataSourceRealm (users stored in a database) for Tomcat.

  • Third-party IDPs are configured for single sign-on (SSO)—such as Google, Active Directory, Microsoft Entra ID (formerly known as Azure Active Directory), etc.—in a new Identity Management built-in feature of the Semarchy xDM platform, which supports user authentication, authorizations, and profile synchronization.

    This feature replaces the IDP configuration performed at the application server level. For example, the Realms and Valves configured in semarchy.xml for Tomcat.

  • A new built-in Role Lookup feature replaces Tomcat’s JDBC Database Realm to assign roles to users based on the content of a database table.

With these features, administrators no longer need to edit files on the server to configure the authentication and authorization for the Semarchy xDM platform.

When upgrading the repository:

  • You will log in using a setup token. The setup token is a string of your choice that you set in a SEMARCHY_SETUP_TOKEN environment variable. For more information, see Startup configuration.

  • After connecting with the setup token, you will provide an administrator username and password to seed a first administrator (SemAdmin) user.

After the upgrade:

  • The Semarchy xDM platform will be configured with the internal IDP only and this administrator user.

  • Existing user profiles will be converted to users, with no password, with internal authentication disabled. These users may still be able to connect from a third-party IDP. Administrators will need to enable internal authentication, reset the passwords, and grant roles for the users to log in with the internal IDP.

Using first administrator credentials, administrators will be able to:

  • Reconfigure the users previously stored at the application server level.

  • Configure identity management, including:

    • The third-party authentication (SSO) systems as IDPs with, optionally, their role mapping.

    • If applicable, the Role Lookup feature, to assign roles to users based on database table contents.

Required action

Before starting the upgrade, administrators should plan the post-upgrade actions to restore the authentication configuration. This includes the configuration of the third-party IDPs as well as the migration of their users stored in databases or files to the internal IDP.

For more information, see the following documentation sections:

For a Tomcat installation, the identity providers (SSO) configuration is contained in the semarchy.xml configuration file, and the local users used for form authentication are in the tomcat-users.xml file.

Platform privileges

This release introduces changes to the platform privileges that you can assign to roles:

  • Two new privileges named User Management and Role Management are introduced to manage the users and roles. The SemAdmin role is required to configure identity management.

  • A new privilege named Dashboard Design grants limited access to the Dashboard Builder to create charts and dashboards only. This role complements the existing Dashboard Management role.

Recommended action

Administrators should review the new platform privileges and plan to apply relevant changes to the existing roles.

Secrets management

Starting with this version, you can configure how secrets—that is, sensitive data that need to be encrypted and optionally decrypted, such as passwords, tokens, or keys—are managed in Semarchy xDM. By default, these secrets are stored with default encryption.

The storage of the secrets can be configured to use more secure mechanisms:

  • A key management service (KMS) defines how secrets stored in the repository are encrypted using, for example, a key local to the Semarchy server or a third-party system such as AWS Key Manager Service or Azure Key Vault.

  • Secrets manager configuration allows storing secrets in an external system (for example AWS Secrets Manager or Azure Key Vault).

After the upgrade, Semarchy xDM is configured to use a default KMS for encrypting secrets, labeled Insecure. This KMS provides encryption capabilities at installation time using a default built-in key.

Recommended actions

Administrators should review the new secrets management mechanisms and consider configuring stronger encryption or an external secrets manager after the upgrade.

Stored procedures for PostgreSQL

This release supports stored procedures for PostgreSQL version 11 and later. Stepper triggers now use procedures by default instead of functions.

Recommended actions

When upgrading from PostgreSQL version 10 to 11 or later, administrators should review the stepper triggers definition and update legacy functions to procedures. Alternatively, they can set the usePostgreSQLFunctionsAsProceduresInStepperTriggers=true system property to continue using functions as procedures.

Upgrade from versions before 5.2.0

This section applies to all installations upgrading from a version prior to 5.2.0.

Infrastructure and system requirements

The following changes in the system requirements may impact your setup:

  • Oracle Database versions 11gR1, 11gR2, and 12gR1 are no longer supported.

  • PostgreSQL version 9.5.x is no longer supported.

  • Eclipse Jetty versions before 9.4 are no longer supported.

  • WildFly versions before 17.x are no longer supported.

Recommended actions

  • Review the system requirements.

  • If your infrastructure uses a Java machine, database, or application server version that is no longer supported, make sure to plan for an upgrade of your infrastructure before upgrading to this release.

  • Make sure to upgrade older driver versions to more recent ones.

Upgrade from versions before 5.1.0

This section applies to all installations upgrading from a version prior to 5.1.0.

New license management

Semarchy xDM introduces in this release a new system to manage licenses. This new license system simplifies the management of licenses and xDM instances at an enterprise level. With the license system, new or upgraded xDM instances run during a grace period, during which they must be declared and activated.

The new license management applies to newly created xDM 5.1 instances, as well as instances upgraded to version 5.1.

Recommended actions

Before starting the upgrade, administrators should plan the upgrade of their various environments with the following facts in mind:

  • The repository upgrade process erases the existing license key in xDM instances when upgrading from a version prior to 5.1.0.

  • Such instances have a seven-day grace period during which they work normally. After that period, they will stop working.

  • Before the grace period is over, administrators should activate the instance, using the instructions described in Manage the license.

Our global Technical Support team (support@semarchy.com) is available for assistance during the upgrade process and to help plan the license change.

Oracle JDBC datasource configuration

Starting with this version, the JDBC connections to an Oracle Database hosting a repository or data location must be made with the oracle.jdbc.J2EE13Compliant property set to true.

When this option is not set, errors such as the following one will be raised in the application log.

com.semarchy.mdm.runtime.data.InvalidDataAccessResourceUsageException: java.lang.RuntimeException: Unexpected DB value .... (Class oracle.sql.TIMESTAMP for logicalType TIMESTAMP)
Required action

Administrators should review their Oracle datasources configuration and make sure to have them configured with the oracle.jdbc.J2EE13Compliant property set to true.

New indexes on MD tables

The data location upgrade in version 5.1.0 creates new performance indexes for all the MD_xxx tables, using the B_SUGGMERGEID column.

For performance optimization, database administrators may have already created manually such indexes, which may conflict with the new ones that have been created during the data location upgrade.

Required actions

Administrators should check all MD_xxx tables in data locations for customized indexes manually created on the B_SUGGMERGEID column. If such indexes exist, they should be dropped before the data location upgrade.

Upgrade from versions before 5.0.0

This section applies to all installations upgrading from a version prior to 5.0.0.

Infrastructure and system requirements

The following changes in the system requirements may impact your setup:

  • Apache Tomcat 7.x-8.0.x and Oracle WebLogic 12c Release 1 (12.1.x) application servers are no longer supported

  • Oracle Database version 10g is no longer supported for the repository and data locations.

  • Oracle JDBC driver version is updated to version 12.2.x (ojdbc8.jar).

  • PostgreSQL JDBC driver version is updated to version 42.2.5.

Other changes to the Semarchy xDM web application:

  • The web application uses JEE 7.

  • Support for Java 11.

Recommended actions

  • Review the system requirements.

  • If your infrastructure uses a Java machine, database, or application server version that is no longer supported, make sure to plan for an upgrade of your infrastructure before upgrading to this release.

  • Make sure to upgrade older driver versions to the newer ones.

  • Review the installation instructions to review possible changes required to switch to Java 11.

Upgrade from versions before 4.0.0

This section applies to all installations upgrading from a version prior to 4.0.0.

Infrastructure and system requirements

The following changes in the system requirements may impact your setup:

  • Semarchy xDM now requires Java 8. Java Runtime Environment (JRE) or Java Development Kit (JDK) version 1.8.0 or above is now required.

  • The following application servers are no longer supported as they do not support Java 8:

    • Glassfish 3.x (4.x is the minimum required version).

    • JBoss 7.x (WildFly 8 is the minimum required version).

Required action

If your current infrastructure does not meet the new requirements, upgrade your infrastructure before upgrading Semarchy xDM.

Browser support

Semarchy xDM no longer supports old web browsers. For the current list of supported web browsers, see System requirements.

Recommended action

Administrators should inform all Semarchy xDM users of the new client requirements.

SOAP web service decommission

SOAP web services available in previous versions of Semarchy xDM are decommissioned in this release. A new REST API is made available for new users, providing similar integration capabilities for querying data in the hub and publishing data into the MDM hub. Note that certain features available in the SOAP Web Services are not available in the REST API, such as workflow management and administrative services.

Recommended action

Existing users relying on the SOAP web services, and willing to continue using them temporarily before moving to the new REST API should contact our global Technical Support team (support@semarchy.com).

Pulse dashboard not unavailable

This version does not include the Pulse dashboards in the generated application. Pulse is replaced by Semarchy xDM Discovery for profiling and Semarchy xDM Dashboard for metrics.

For more information about these new components, users currently using Pulse can contact our global Technical Support team (support@semarchy.com).

Data edition decommission

The data edition feature is removed in this release. A data location now hosts one deployed model edition at a time and provides access to the current state of the data in the hub, based on the deployed model edition.

The upgrade process prunes the data in closed data editions from the data locations. Only data from the latest data edition is preserved in each data location.

Recommended action

Existing users willing to archive the contents from previous data editions should export this information from the data location tables using scripting or data integration flows, or keep a backup of the data location contents performed before the upgrade.

Error storage

Error storage has been changed in this release. The golden errors and source error tables no longer store the entire history of errors, but only the latest error instances.

The data location upgrade process prunes the history of errors on golden and source records, preserving only the latest error instances.

Recommended action

Existing users willing to preserve the history of errors should export this information from the data location tables using scripting or data integration flows, or keep a backup of the data location contents performed before the upgrade.