Post-upgrade actions

Bug fix MDM-16161 has been introduced in this version to resolve an issue where, during golden-record authoring for a fuzzy-matched child entity, updating the parent reference using the parent’s FDN attribute failed to update the parent reference (FID or FDN attributes) during the certification process.

The Microsoft JDBC Driver for SQL Server has been updated to ensure compatibility with JDK 17.

Following the introduction of a new PARAM_RECYCLE_POST_CONSOLIDATION_ERRORS integration job parameter to complement the PARAM_RECYCLE_PRE_CONSOLIDATION_ERRORS parameter (formerly PARAM_RECYCLE_ERRORS), model designers should assess the relevance of this new feature to their current error handling procedures and adjust the configuration of existing integration jobs accordingly.

For more information on integration job parameters, see Job parameters reference.

Additional localizable files are included in the custom translation archive to enable custom translation of all user-facing components. Custom translation files have been reorganized into the following extended set of folders:

As a result of this reorganization, custom translation archives from previous versions can no longer be imported. Platform administrators should re-export from an upgraded instance and align with the new archive structure to ensure a successful import of the translated files back into xDM.

Additionally, the migration of some localization strings (mostly located in the former /commons and /commons2 folders) to the new /shared-utilities folder might result in a loss of translated content after upgrading to version 2024.2. Platform administrators who may notice missing translations should export the new custom translation archive and translate the affected strings again.

To ensure that SemQL expressions align with the expected execution behavior of user task steps within a workflow definition, the UserTaskInstance and UserTaskInstanceID attributes have been deprecated in the Assignment Condition view of the SemQL editor.

When defining an assignment condition, workflow designers should replace these attributes with StepInstance and StepInstanceID, respectively.

Bug fix MDM-16161 has been introduced in this version to resolve an issue where, during golden-record authoring for a fuzzy-matched child entity, updating the parent reference using the parent’s FDN attribute failed to update the parent reference (FID or FDN attributes) during the certification process.

After upgrading to version 2024.1.8, validation rules using square brackets in LIKE statements may fail in SQL Server environments, potentially resulting in valid data being incorrectly rejected. This issue is associated with the fix for MDM-15633, introduced in version 2024.1.7 and disabled by default for SQL Server in version 2024.1.9.

To bypass any potential browser limitations that might prevent downloading or accessing files exceeding 2 MB in workflows, all attachments over 2 MB are automatically downloaded on all browsers when users attempt to access them.

Microsoft JDBC Driver for SQL Server has been updated to ensure compatibility with JDK 17.

To avoid duplicate user entries due to case variations in usernames when using LDAP authentication in the platform’s identity management system, all LDAP usernames are converted to lowercase during login.

To prevent notifications from being indefinitely delayed due to connectivity issues with remote servers, a default timeout duration of 30 seconds is applied to SMTP and SMTPS notification servers when platform administrators do not specify any value.

The Read-Only property available on form fields is replaced with Authoring Mode that allows model designers to define if each field is displayed using its editable, read-only, or browsing variant (see Form field properties).

When migrating to 2023.1.0 or above, the Authoring Mode property of existing form fields is automatically initiated according to the following rules:

Additionally, the Read-Only property available on stepper form steps is replaced in version 2023.1.0 with Form Fields Mode. This property allows model designers to override the Authoring Mode for all the fields of a given step (see Form steps flow).

When migrating to 2023.1.0 or above, the Form Fields Mode property of existing stepper form steps is initiated according to the following rules:

Semarchy xDM version 5.3.24 includes a fix on the certification process (MDM-14406) to prevent golden references to fuzzy-matched entities on basic entity golden records from being temporarily reset to null during integration job execution. To upgrade integration jobs, this fix requires redeploying all models.

Starting with this version, the logging in Semarchy xDM relies on Log4j 2, which is configured using an XML configuration format, in replacement of the key-value property pairs previously used. It is therefore required to migrate the existing startup and platform logging configurations to that new Log4j 2 format.

A new security framework in version 5.3 introduces changes to the SAML identity provider (IDP) and assertion consumer service (ACS) URLs, necessitating updates to single sign-on (SSO) provider configuration.

As mentioned in the pre-upgrade actions, after upgrading to version 5.3.0, the Semarchy xDM platform is configured with the internal identity provider only and the administrator user that was created during the upgrade process.

As mentioned in the pre-upgrade actions, the datasources used by the data locations, xDM Dashboard, xDM Discovery, the variable value providers and the enrichment or validation plugins are no longer configured at the application server level and must be defined in the Semarchy xDM platform as platform datasources.

The upgrade process automatically creates empty datasources for the components that refer to application server datasources. However, it does not recover the datasource configurations.

The referenced JNDI datasources are upgraded into empty platform datasources with a normalized name. The normalized name is derived from the JNDI name by performing the following operations:

For example, in a data location, a reference to a JNDI datasource named java:comp/env/jdbc/DATA_LOCATION_1 is upgraded to a platform datasource named DATA_LOCATION_1. That datasource is empty and must be configured by the administrator.

The following changes have been made to the REST API authentication mechanism:

The Enable Import, Enable Import-Update, and Restrict Import to Update properties available on workflow transitions, steppers, and actions are replaced with two properties:

The Batch Update Size properties of enricher plugins now automatically take the value 1,000 when left empty. This value provides better performance in most situations.

The master records’ OldMatchGroupID attribute (B_OLDMATCHGRP column in the MI table) is deprecated.

With the new Integrated authentication feature, the user and role tables (MTA_USER, MTA_USER_ROLES) stored in the repository have been moved and their structure has been changed. The new tables storing users and roles are named IDM_USER and IDM_USER_ROLE.

The following changes have occurred in the built-in variable:

This release changes how case sensitivity is handled for database functions declared in the model.

Starting with this release, you can declare the function names and schemas in any case. However, a model validation prevents having two functions with the same qualified name and a different case.

PostgreSQL and Oracle are case-insensitive for all function calls. PostgreSQL assumes that all object names are in lowercase, and Oracle assumes that all object names are in uppercase. The product behavior does not change for models based on these two databases.

An SQL Server database may be configured to handle object names in a case-sensitive manner. Starting with this release, Semarchy xDM assumes, for SQL Server, case sensitivity for all function calls, including the SemQL built-in functions. Function calls performed in SemQL with a function name that does not match exactly the case of the built-in function definition (as it appears in the SemQL editor) now raise warnings in the model validation, since they may fail at run-time, depending on the database server configuration.

A new Melissa plugin for Semarchy xDM provides enrichers to fix and complete contact data for the US and Canada using the Personator service and to validate international addresses in 240 countries using the Global Address Verification service.

Existing enrichers using the legacy Personator enricher are automatically upgraded to use the new enricher as part of the automated model upgrade process.

The upgrade automatically configures the enricher to use the new plugin capabilities, as described below:

The Matchcode Global plugin for Semarchy xDM, using GBGRoup Matchcode Global to provide an enricher for international postal addresses is removed in this release.

in xDM Discovery, the Row Sampling table property is deprecated and replaced with the Maximum Distinct Values Stored and Maximum Patterns Stored properties with default values of 50. The value set in Row Sampling is automatically copied to these new properties by the upgrade process.

Semarchy xDM introduces in this release a new system to manage licenses.

The repository upgrade process erases the existing license key in xDM instances when upgrading from a version prior to 5.1.0. Such instances have a seven-day grace period during which they work normally. After that period, they will stop working.

Semarchy xDM Discovery enables data architects and business users to gather metrics and profile any source data to prepare a data management initiative.

When connecting the welcome page, users with the appropriate privileges will see a new xDM Discovery link to access the user interface for Semarchy xDM Discovery.

in previous releases, only the Delete, Confirm Duplicates, Review and Confirm Duplicates and Review Duplicates Suggestions actions had Enable All properties to execute the action on all records when no selection was made.

Starting with this release, this implicit select all is replaced with an explicit select all capability that applies to all actions. With this feature, users can explicitly select all the filtered records in a collection, possibly deselect some of them and then start an action with the selection.

The number of records that can be processed at the same time may be limited at design time for each action using a new optional Items Limit property, available in the Action Configuration properties. A warning is raised to the user when he triggers the action with a selection larger than this limit.

Actions are automatically upgraded to use this new feature:

A new Split Duplicate Propagation property is added to the references configuration to define how splitting records in the referenced entity affects the referencing records. This property applies when both the referenced and referencing entities are fuzzy-matched.

Setting this property to Reset Matching forces rematching child records when their parent is split in a duplicate manager.

The child records are re-matched, ignoring their previous match groups and user decisions. The resulting golden records are redistributed under their respective parent records. When re-matched, the referencing records are all re-merged into new golden records with new Golden IDs and lose all their value overrides.

This release introduces full support for SQL Server, including production certification. The configuration of the SQL Server databases used for the repository and data locations need to be reviewed to make sure they comply with the requirements described in database-specific considerations.

The QUOTED_IDENTIFIER and READ_COMMITTED_SNAPSHOT should be set to ON for these databases.

in previous releases, a child record referencing a parent record detected as erroneous during the pre-consolidation validation passed the certification process and was published with a null reference.

Starting with this release, when a child record references a parent record detected as erroneous during the pre-consolidation validation, then this child record is also rejected in the pre-consolidation validation by the reference validation. If a previous valid version of the parent record exists in the hub, then the child record is attached to that version instead. During post-consolidation validation, a reference is always considered valid if the referenced record exists (flagged as erroneous or not).

Semarchy xDM Dashboard is a new component for designing visualizations and dashboards that create insight by combining your business data and the data stored and managed in the Semarchy xDM intelligent data hubs.

When connecting the welcome page, users with the appropriate privileges will see a new Dashboard Builder link to access the design-time user interface for dashboards.

The user interfaces available in Semarchy xDM have been reorganized:

The Workbench is now split into two user interfaces:

A new Dashboard Builder user interface is available for designing dashboard applications.

in previous releases of Semarchy xDM, roles with a model privilege grant with the Grant full access to the model selected had also access to the REST API, as if the Grant access to integration web services was selected too.

Starting with this release, Grant full access to the model does not automatically grant access to the REST API, and the Grant access to integration web services must be explicitly selected in the model privilege grant.
The upgrade process automatically takes care of checking this option for all model privilege grants with Grant full access to the model.

The Semarchy Lookup enricher supports multiple outputs, which changes its configuration. The configuration of plugin enrichers using this component is automatically upgraded.

in previous releases, publishing with the REST API a record update for a fuzzy- and ID-matched entity incorrectly set to null the attributes not present in the payload.

This behavior has been aligned (in bug fix MDM-7819) with the basic entity behavior. Only the attributes present in the payload are taken into account to update the existing record. Attributes absent from the payload keep the value of the original record.

To enable large file imports, a streaming data import feature has been implemented for CSV and XLSX file formats. The XLS format for import is deprecated since it does not support streaming. XLS files are processed as CSV files.

This release introduces a memory leak fix (MDM-8086) to the REST API and makes calls stateless to reduce resource consumption and increase security.

This fix requires Tomcat instances to remove references to semarchy.commons.web.configuration_*.jar in the tomcat.util.scan.StandardJarScanFilter.jarsToSkip property found in the Catalina/catalina.properties file.

in this version, business views support searching and filtering every collection in the business view. In previous releases, filtering was only supported on the root collection of the business view. This new feature implies the following changes:

in this version, a new Date datatype replaces DateTime to store dates with no time or timestamp. The Timestamp datatype remains to store dates and times with timestamp information. The upgrade process automatically converts DateTime attributes to the new Date data type in the model and data locations. As a consequence, data columns using the old DateTime type are automatically trimmed to store a Date.

The DateTime type is deprecated for plugin enrichers and validators, and the new Date type is added.

in this version, the data purge is modified with the following changes:

Besides:

A new option is available for matching entities on duplicate managers and form steps. Enable Master Value Picking enables users to override the value of consolidated golden records to select values from the master records in addition to being able to enter custom values.

When upgrading to this version, this option is automatically activated for duplicate managers, but not for form steps.

This version introduces a new visualization for business views. Hierarchies views provide a user-friendly way for business users to navigate hierarchies (of products, organization, cost centers, employees) with an intuitive user interface.

Hierarchies are not configured by default during the upgrade.

in this version, it is possible to enable automated historization to keep track of all changes on master and golden data for each entity. Applications can be configured to allow browsing the history of the records.

The upgrade does not automatically enable historization for the entities.

This version introduces the capability to perform soft and hard deletes for records in all entity types.

This option is not automatically enabled by the upgrade.

Starting with version 4.3.0, ID- and fuzzy-matched entities support authoring (creating, editing, importing) source or erroneous records on behalf of publishers. This option is enabled when configuring authoring actions for ID- and fuzzy-matched entities.

Existing action sets are not modified by the upgrade process to support this feature.

in this release, the following changes take place in the integration interfaces and should be reviewed by the integration specialists:

After upgrading from a version before 4.3.0, any form or collection for an entity with a Primary Key attribute of type UUID, and using this UUID in the display card, fails to load with the following exception:

Display cards created after version 4.3.0 do not have this issue since the UUID is automatically wrapped in a SEM_UUID_TO_CHAR function call.

After a model upgrade, the data entry workflows that existed in a version prior to 4.0.0 are not automatically upgraded. They are preserved in a hidden form in the model.

The data authoring experience in version 4 supports two methods:

Workflows and steppers are triggered using actions defined in action sets.

For more information about steppers, workflows, and action sets, see Applications.

After a model upgrade, the duplicate management workflows that existed in a version prior to 4.0.0 are not automatically upgraded. They are preserved in a hidden form in the model.

The duplicate management experience now uses a dedicated design-time artifact, the Duplicate Manager, duplicate management actions defined in action sets to review, confirm, merge, or split duplicates and suggestions.

For more information on duplicate managers and action sets, see Applications.

ID- and fuzzy-matched entities (also known as UDPK and SDPK entities) now support overriding data values on golden records. When data authoring operations are performed in these entities, the data authored by the users (i.e., authoring data) is separated from the data loaded from external publishers (i.e., source data).

The consolidators have evolved into Survivorship rules, and support two phases:

Survivorship rules allow implementing simply the pattern into which users fix consolidated values or directly create golden records in matching entities, without having to rely on matching and consolidation rules. They are now used only for the data loaded from external publishers (i.e., source data).

in this version, a Documentation property is added to entity attributes, complex types’ definition attributes, references, matching, and survivorship rules. The documentation property supports the Markdown syntax for rich text. It provides detailed information to end users using the generated application.

The Description property is not only used for describing/documenting model artifacts for model designers. Descriptions are no longer visible to end users.

The documentation of an attribute/reference is used by default in all form fields based on this attribute/reference. It is possible to override this inherited documentation in the form field.

The upgrade process automatically copies the content of the Description properties (which was visible to end users before) to the Documentation property.

The data edition feature is removed in this release. A data location now hosts one deployed model edition at a time and provides access to the current state of the data in the hub, based on the deployed model edition.

Post-consolidation validations, running on consolidated data, are no longer removing golden records. They only flag these records as erroneous, yet keep them as golden records.

The storage of errors has changed in this release:

The structure and the content of the applications and application components have considerably changed between versions 3.x and 4.x. This section details how the repository upgrade converts original artifacts into new ones.

Users who have upgraded to version 3.2 before version 3.2.4 may experience an issue with singleton golden records (for fuzzy-matching entities) already in the hub not being merged with incoming records that match them. This issue is caused by a defect in the upgrade mechanism.

New capabilities added to the SemQL language introduce the following reserved keywords:

The integration job changes from a “by-entity” to a “by-phase” execution scheme. This means that instead of processing entirely each entity (e.g., Customers then Contact), it now processes entirely each phase (enrichment for all entities, then validation for all entities, etc.). This change does not impact the outcome of the integration job.

In versions before 3.2, a matcher includes a single matching condition.
Starting with version 3.2, a matcher is composed of several matching rules and supports scoring for automated merge and confirmation. To reflect this new matcher and provide performance enhancements, the data certification job was redesigned and the data location structures have been modified.

in versions before 3.2, all the default built-in search methods (full text, advanced, by form, and SemQL) were available in the business object views. Starting with version 3.2, it is possible to create customized forms and define which of the search methods or customized forms are available in applications.

Starting with version 3.2, it is possible to declare user-defined PL/SQL functions. By declaring these functions, designers make them available in the SemQL editor and avoid SemQL warnings raised by these functions.

Before version 3.2, the attributes’ physical column names could take any value. Starting with version 3.2.0, it is no longer possible to define physical column names starting with B_ as they may collide with reserved column names.

Models with physical column names violating this rule no longer pass model validation and each column will raise a validation error such as Invalid physical name: <column_name> matches the reserved pattern B_*.

in versions before 3.1, the SemAdmin role had hardcoded full access to the data on all models.
Starting with version 3.1 the privilege grant for this role is no longer hardcoded and can be modified.

When creating a new model, or upgrading an existing one, a model-level privilege grant is automatically created for the SemAdmin role with Grant full access to the model selected to keep the same level of privileges.

in versions before 3.1, accessing the integration services required a role having platform-level read/write privileges to the data location. Starting with version 3.1, a specific option called Grant access to integration services appears in the model privilege grants to define whether a role can access this model’s data via the services.

This privilege applies to the SOAP and REST service APIs.

When upgrading, roles that could access the integration services thanks to their privileges on the data locations will automatically have this option checked for the models deployed in the data location.