Pre-upgrade actions
Before proceeding with the upgrade procedure, specific actions may be required based on the version of Semarchy xDM from which you are upgrading.
| Carefully examine the changes and actions listed on this page, as they may have a significant impact on your configuration. |
| Make sure to read all the sub-sections on this page until you reach your current Semarchy xDM product version. |
For all versions
Execution engine queue and load management
Failure to clear an execution engine queue prior to an upgrade can cause conflicts, interruptions, and performance degradation that may hinder the upgrade process and prolong downtime for users.
Platform administrators must ensure that all execution engine queues are empty and no loads are scheduled, pending, or running before initiating the upgrade process.
Browser support
Semarchy xDM no longer supports old web browsers. For the current list of supported web browsers, see System requirements.
Administrators should inform all Semarchy xDM users of the new client requirements.
Upgrade from versions before 2023.1.8
This section applies to all installations upgrading from a version prior to 2023.1.8.
Groovy script security update
This version introduces an allowlist mechanism for Groovy scripts and conditions, which limits the scope of permissible classes. A default allowlist ensures that only specified classes or methods can be utilized in the execution of Groovy scripts or the evaluation of conditions when configuring custom job notifications.
To modify the default allowlist without compromising security, platform administrators are invited to contact our Technical Support team.
Upgrade from versions before 2023.1.6
This section applies to all installations upgrading from a version prior to 2023.1.6.
Infrastructure and system requirements
Starting from this version, PostgreSQL version 11 is no longer supported (end of life).
| In line with Azure’s announcement of extended support for PostgreSQL 11 servers, Semarchy xDM is scheduled to maintain its support for Azure Database for PostgreSQL 11 in Single Server until November 9, 2024. |
Upgrade from versions before 2023.1.3
This section applies to all installations upgrading from a version prior to 2023.1.3.
X-Forwarded-For, Forwarded, and X-Real-IP HTTP headers usage
Starting from version 2023.1.3, the X-Forwarded-For, Forwarded, and X-Real-IP HTTP headers are ignored by default by Semarchy xDM to enforce security.
For infrastructures requiring to use one or several of these headers, administrators must set the allowXForwardedHeaders system property to true.
| This configuration should be activated only when the xDM instance is exclusively accessible through a reverse proxy. |
Upgrade from versions before 2023.1.0
This section applies to all installations upgrading from a version prior to 2023.1.0.
Infrastructure and system requirements
Starting from this version, Semarchy xDM does not support Java 8 anymore. Instances running on this version must upgrade to a supported version of Java Runtime Environment (JRE) or Java Development Kit (JDK) (see System requirements).
The following application servers are no longer supported as they do not support Java 11.
-
Glassfish versions before 6.x (6.1 is the minimum required version).
-
JBoss versions before 7.1 (WildFly 15 is the minimum required version).
| If your current infrastructure does not meet the new requirements, upgrade your infrastructure before upgrading Semarchy xDM. |
Upgrade from versions before 5.3.9
This section applies to all installations upgrading from a version prior to 5.3.9.
Logging
Starting from this version, the logging in Semarchy xDM relies on Log4j 2 and requires to migrate the existing startup and platform logging configurations to Log4j 2 format.
Before starting the upgrade, administrators must create a backup of the current platform logging configuration by opening menu:Configuration[Logging Configuration] and copying the entire content to a file. This backup will be used to create or amend the new Log4j 2 configuration.
Upgrade from versions before 5.3.4
This section applies to all installations upgrading from a version prior to 5.3.4.
Javax.json library for Tomcat
This release no longer includes the javax.json library to avoid possible conflicts with other libraries (JMS drivers) or application server-provided implementations.
Javax.json is part of JEE 7, and application servers like JBoss, Glassfish, or Oracle WebLogic provide a built-in implementation of this library.
However, Tomcat does not provide a built-in implementation of javax.json. When deploying Semarchy xDM in an existing Tomcat server, make sure to copy the org.glassfish.jakarta.json_*.jar file from the mdm-server/additional-libraries/ folder to the Tomcat lib/ folder.
Upgrade from versions before 5.3.0
This section applies to all installations upgrading from a version prior to 5.3.0.
Single deployment per application server
This release introduces a new mechanism to bootstrap a startup configuration for Semarchy xDM, using environment properties and variables. Due to this startup configuration mechanism, it is no longer possible, starting with this version, to perform multiple deployments of Semarchy xDM in the same application server.
Before starting the upgrade, administrators should review their infrastructure for possible Semarchy xDM deployments in the same application server instance, in different deployment contexts.
For instances having multiple deployments in the same application server, it is recommended to perform an out-of-place upgrade. The replicated environment should be configured to take into account this limitation.
Integrated datasources
This release introduces new built-in datasources to connect the backend databases, such as those hosting the repository and the data locations.
With this new mechanism, datasources are no longer configured at the application server level (in the semarchy.xml configuration file). They are configured in Semarchy xDM:
-
The repository connection (formerly
SEMARCHY_REPOSITORYdatasource) is now configured as two datasources, set in the startup configuration, via a configuration file or environment variables:-
The Repository Datasource, which reproduces the
SEMARCHY_REPOSITORY, with credentials allowing to read from and write into the repository schema. -
A new Repository Read-Only Datasource, with credentials allowing to read a subset of the tables stored in the repository schema. This datasource uses a new dedicated user with read-only privileges to the repository.
-
-
Other datasources are now configured as platform datasource in the Semarchy xDM Configuration.
-
Administrators must re-configure these datasources after the repository upgrade based on their current application server configuration.
-
Components using datasources, including data locations, xDM Dashboard, xDM Discovery, the variable value providers, and the enrichment or validation plugins will refer to these datasources.
-
Before starting the upgrade, administrators must:
-
Create the repository read-only user.
-
Prepare the startup configuration required to start the application. This configuration should include the connection information for the repository and repository read-only datasources.
| The startup configuration is required to start the Semarchy xDM platform during the upgrade process. |
Administrators should also gather connection information for all datasources used by Semarchy xDM to facilitate reconfiguring these datasources during the post-upgrade actions.
For a Tomcat installation, the original datasources are defined in the semarchy.xml file.
|
Integrated authentication
Semarchy xDM introduces in this release a new system to manage user authentication. With this new system:
-
Locally defined users authenticate with a built-in internal identity provider (IDP), which stores users, passwords, and their assigned roles in the repository.
This IDP replaces users stored at the application server level, such as the UserDatabaseRealm ( tomcat-users.xmlfile) and DataSourceRealm (users stored in a database) for Tomcat. -
Third-party IDPs are configured for single sign-on (SSO)—such as Google, Active Directory, Microsoft Entra ID (formerly known as Azure Active Directory), etc.—in a new Identity Management built-in feature of the Semarchy xDM platform, which supports user authentication, authorizations, and profile synchronization.
This feature replaces the IDP configuration performed at the application server level. For example, the Realms and Valves configured in semarchy.xmlfor Tomcat. -
A new built-in Role Lookup feature replaces Tomcat’s JDBC Database Realm to assign roles to users based on the content of a database table.
With these features, administrators no longer need to edit files on the server to configure the authentication and authorization for the Semarchy xDM platform.
When upgrading the repository:
-
You will log in using a setup token. The setup token is a string of your choice that you set in a
SEMARCHY_SETUP_TOKENenvironment variable. For more information, see Startup configuration. -
After connecting with the setup token, you will provide an administrator username and password to seed a first administrator (SemAdmin) user.
After the upgrade:
-
The Semarchy xDM platform will be configured with the internal IDP only and this administrator user.
-
Existing user profiles will be converted to users, with no password, with internal authentication disabled. These users may still be able to connect from a third-party IDP. Administrators will need to enable internal authentication, reset the passwords, and grant roles for the users to log in with the internal IDP.
Using first administrator credentials, administrators will be able to:
-
Reconfigure the users previously stored at the application server level.
-
Configure identity management, including:
-
The third-party authentication (SSO) systems as IDPs with, optionally, their role mapping.
-
If applicable, the Role Lookup feature, to assign roles to users based on database table contents.
-
Before starting the upgrade, administrators should plan the post-upgrade actions to restore the authentication configuration. This includes the configuration of the third-party IDPs as well as the migration of their users stored in databases or files to the internal IDP.
For more information, see the following documentation sections:
For a Tomcat installation, the identity providers (SSO) configuration is contained in the semarchy.xml configuration file, and the local users used for form authentication are in the tomcat-users.xml file.
|
Platform privileges
This release introduces changes to the platform privileges that you can assign to roles:
-
Two new privileges named User Management and Role Management are introduced to manage the users and roles. The SemAdmin role is required to configure identity management.
-
A new privilege named Dashboard Design grants limited access to the Dashboard Builder to create charts and dashboards only. This role complements the existing Dashboard Management role.
Administrators should review the new platform privileges and plan to apply relevant changes to the existing roles.
Secrets management
Starting with this version, you can configure how secrets—that is, sensitive data that need to be encrypted and optionally decrypted, such as passwords, tokens, or keys—are managed in Semarchy xDM. By default, these secrets are stored with default encryption.
The storage of the secrets can be configured to use more secure mechanisms:
-
A key management service (KMS) defines how secrets stored in the repository are encrypted using, for example, a key local to the Semarchy server or a third-party system such as AWS Key Manager Service or Azure Key Vault.
-
Secrets manager configuration allows storing secrets in an external system (for example AWS Secrets Manager or Azure Key Vault).
After the upgrade, Semarchy xDM is configured to use a default KMS for encrypting secrets, labelled Insecure. This KMS provides encryption capabilities at installation time using a default built-in key.
Administrators should review the new secrets management mechanisms and consider configuring stronger encryption or an external secrets manager after the upgrade.
Stored procedures for PostgreSQL
This release supports stored procedures for PostgreSQL version 11 and later. Stepper triggers now use procedures by default instead of functions.
When upgrading from PostgreSQL version 10 to 11 or later, administrators should review the stepper triggers definition and update legacy functions to procedures. Alternatively, they can set the usePostgreSQLFunctionsAsProceduresInStepperTriggers=true system property to continue using functions as procedures.
Upgrade from versions before 5.2.0
This section applies to all installations upgrading from a version prior to 5.2.0.
Infrastructure and system requirements
The following changes in the system requirements may impact your setup:
-
Oracle Database versions 11gR1, 11gR2, and 12gR1 are no longer supported.
-
PostgreSQL version 9.5.x is no longer supported.
-
Eclipse Jetty versions before 9.4 are no longer supported.
-
WildFly versions before 17.x are no longer supported.
-
Review the system requirements.
-
If your infrastructure uses a java machine, database, or application server version that is no longer supported, make sure to plan for an upgrade of your infrastructure before upgrading to this release.
-
Make sure to upgrade older driver versions to the newer ones.
Upgrade from versions before 5.1.0
This section applies to all installations upgrading from a version prior to 5.1.0.
New license management
Semarchy xDM introduces in this release a new system to manage licenses. This new license system simplifies the management of licenses and xDM instances at an enterprise level. With the license system, new or upgraded xDM instances run during a grace period, during which they must be declared and activated.
The new license management applies to newly created xDM 5.1 instances, as well as instances upgraded to version 5.1.
Before starting the upgrade, administrators should plan the upgrade of their various environments with the following facts in mind:
-
The repository upgrade process erases the existing license key in xDM instances when upgrading from a version prior to 5.1.0.
-
Such instances have a seven-day grace period during which they work normally. After that period, they will stop working.
-
Before the grace period is over, administrators should activate the instance, using the instructions described in Manage the license.
Our global Technical Support team (support@semarchy.com) is available for assistance during the upgrade process and to help plan the license change.
Oracle JDBC datasource configuration
Starting with this version, the JDBC connections to an Oracle Database hosting a repository or data location must be made with the oracle.jdbc.J2EE13Compliant property set to true.
|
When this option is not set, errors such as the following one will be raised in the application log. com.semarchy.mdm.runtime.data.InvalidDataAccessResourceUsageException: java.lang.RuntimeException: Unexpected DB value .... (Class oracle.sql.TIMESTAMP for logicalType TIMESTAMP) |
Administrators should review their Oracle datasources configuration and make sure to have them configured with the oracle.jdbc.J2EE13Compliant property set to true.
New indexes on MD tables
The data location upgrade in version 5.1.0 creates new performance indexes for all the MD_xxx tables, using the B_SUGGMERGEID column.
For performance optimization, database administrators may have already created manually such indexes, which may enter in conflict with the new ones, created during the data location upgrade.
Administrators should check all MD_xxx tables in data locations for customized indexes manually created on the B_SUGGMERGEID column.
If such indexes exist, they should be dropped before the data location upgrade.
Upgrade from versions before 5.0.0
This section applies to all installations upgrading from a version prior to 5.0.0.
Infrastructure and system requirements
The following changes in the system requirements may impact your setup:
-
Apache Tomcat 7.x-8.5.x and Oracle WebLogic 12c Release 1 (12.1.x) application servers are no longer supported
-
Oracle Database version 10g is no longer supported for the repository and data locations.
-
Oracle JDBC driver version is updated to version 12.2.x (ojdbc8.jar).
-
PostgreSQL JDBC driver version is updated to version 42.2.5.
Other changes to the Semarchy xDM web application:
-
The web application uses JEE 7
-
Support for Java 11.
-
Review the system requirements.
-
If your infrastructure uses a java machine, database, or application server version that is no longer supported, make sure to plan for an upgrade of your infrastructure before upgrading to this release.
-
Make sure to upgrade older driver versions to the newer ones.
-
Review the installation instructions to review possible changes required to switch to Java 11.
Upgrade from versions before 4.0.0
This section applies to all installations upgrading from a version prior to 4.0.0.
Infrastructure and system requirements
The following changes in the system requirements may impact your setup:
-
Semarchy xDM now requires Java 8. A Java Runtime Environment (JRE) or Development Kit (JDK) version 1.8.0 or above is now required.
-
The following application servers are no longer supported as they do not support Java 8.
-
Glassfish 3.x (4.x is the minimum required version)
-
JBoss 7.x (WildFly 8 is the minimum required version)
-
|
Required action
If your current infrastructure does not meet the new requirements, upgrade your infrastructure before upgrading Semarchy xDM. |
Browser support
Semarchy xDM no longer supports old web browsers. For the current list of supported web browsers, see System requirements.
Administrators should inform all Semarchy xDM users of the new client requirements.
SOAP web service decommission
SOAP web services available in previous versions of Semarchy xDM are decommissioned in this release. A new REST API is made available for new users, providing similar integration capabilities for querying data in the hub and publishing data into the MDM hub. Note that certain features available in the SOAP Web Services are not available in the REST API, such as workflow management and administrative services.
Existing users relying on the SOAP web services, and willing to continue using them temporarily before moving to the new REST API should contact our global Technical Support team (support@semarchy.com).
Pulse dashboard not unavailable
This version does not include the Pulse dashboards in the generated application. Pulse is replaced by Semarchy xDM Discovery for profiling and Semarchy xDM Dashboard for metrics.
For more information about these new components, users currently using Pulse can contact our global Technical Support team (support@semarchy.com).
Data edition decommission
The data edition feature is removed in this release. A data location now hosts one deployed model edition at a time and provides access to the current state of the data in the hub, based on the deployed model edition.
The upgrade process prunes the data in closed data editions from the data locations. Only data from the latest data edition is preserved in each data location.
Existing users willing to archive the contents from previous data editions should export this information from the data location tables using scripting or data integration flows, or keep a backup of the data location contents performed before the upgrade.
Error storage
Error storage has been changed in this release. The golden errors and source error tables no longer store the entire history of errors, but only the latest error instances.
The data location upgrade process prunes the history of errors on golden and source records, preserving only the latest error instances.
Existing users willing to preserve the history of errors should export this information from the data location tables using scripting or data integration flows, or keep a backup of the data location contents performed before the upgrade.