Base component release notes

DI-9266: Added new template: LOAD JSON to BigQuery

DI-9611: Added new template: INTEGRATION BigQuery to File

DI-5962: Process action TOOL Ldap Extractor(search): fixed an issue that inserted a blank line after the header, when header generation and table truncation were both switched on.

DI-8546: Added an option to retrieve all LDAP operational attributes in an integration flow.

DI-9443: Fixed a missing column error in the replicator template on the BigQuery Part for Load action.

DI-9545: Fixed an issue with mappings where stages with multiple sets did not generate the necessary SQL clauses.

DI-9638: Template LOAD File: added the ability to ignore empty lines at the end of a file.

DI-9877: Fixed an issue generating table names with a not exists clause.

DI-6630: Added two XML serializers, Rbdms Xml as String and Hsql Xml as String. Use them to parse XML data to RDBMS string datatypes.

DI-8440: Cleaned up XML parser declarations in templates.

DI-9411: Fixed incorrect naming of Store elements extracted from Semarchy xDG.

DI-9737: Changed how RDBMS column are displayed so they are listed by their true position, rather than the order they were added.

DI-10269: Template Integration BigQuery: fixed an issue that truncated fields in the group by clause.

DI-10536: Fixed an issue preventing HSQL check constraints from being reversed correctly.

DI-5962: Process action TOOL Ldap Extractor(search): fixed an issue that inserted a blank line after the header, when header generation and table truncation were both switched on.

DI-8546: Added an option to retrieve all LDAP operational attributes in an integration flow.

DI-9443: Fixed a missing column error in the replicator template on the BigQuery Part for Load action.

DI-9640: Fixed an issue with WSDL metadata that generated an incorrect fault segment name.

DI-9877: Fixed an issue generating table names with a not exists clause.

DI-9308: Fixed an issue which showed an error when creating an OAuth2 entry for login.microsoftonline.com.

DI-9411: Fixed incorrect naming of Store elements extracted from Semarchy xDG.

DI-3419: Add support for field expressions that reference target columns when the execution location is set to Target.

DI-6546: Add new serializer template SERIALIZER JSON as RDBMS String to parse JSON data to an RDBMS string data type.

DI-6562: Templates INTEGRATION HSQL and INTEGRATION RDBMS: add support for field expressions that reference target columns when the execution location is set to Target.

DI-7621: Implement serializer functionality in Designer. Use the serializer in mappings to parse structured data from a source and flatten it to string data.

DI-8540: Add support for using values retrieved from external value resolvers in Oauth2 parameters.

DI-8921: Centralize OAuth2 code for better cross-component use.

DI-7370: Remove AUTONOMOUS SQL transaction type, as it has no effect.

DI-7649: Fix SQL Statement parsing error when integrating a hierarchical file to a target CSV or Positional file while using the INTEGRATION File template.

DI-7973: Fix the Metadata Process Palette so the Process Reference parameter selector displays processes instead of metadata only.

DI-8427: Disable connection pooling for the file driver.

DI-8451: Remove unused metadata parameter from the file driver JDBC URL.

DI-8516: The XPath function tech:rejectedBy() in Reject Templates generates an incorrect expression.

DI-8786: Template INTEGRATION File (FileExport): missing column name in template.

DI-9030: Fix mapping build failure when using a WSDL metadata as target.

DI-8378: Move H2 to a dedicated component, and the JDBC library as an optional third-party pack.

DI-5930: Null values are unexpectedly loaded as "<|null|>" inside the target table when reading Salesforce data.

DI-8087: The "Enable Certificate Validation" and "Enable Hostname Validation" metadata properties are ignored by the WSDL metadata wizard.

DI-8272: The REST wizard mismatches the input and output nodes when performing a reverse engineering with WSDL technology.

DI-8604: Template Replicator RDBMS: the templates unexpectedly depend on the Amazon component.

DI-8658: The automatic data type conversion mechanism returns an incorrect data type for the JSON Root Value level.

DI-6631: Add DESERIALIZER RDBMS String as XML template.

DI-6919: Make XML root nodes available as a Deserializer output.

DI-7623: Update templates using S3 features to use the existing AWS S3 TOOL instead of dedicated code.

DI-7906: Add the sourceSet() XPath function to the map.tech technology.

DI-7907: Add ref:parentElement() XPath function to the XML technology.

DI-7821: Review the HSQL datatypes and their releated corresponding super types for consistency during datatype conversions.

DI-8043: Load XML to RDBMS: when using a technology with object size limitation, the table name is unexpectedly truncated if it contains a variable or scripting expression.

DI-9737: Changed how RDBMS column are displayed so they are listed by their true position, rather than the order they were added.

DI-10536: Fixed an issue preventing HSQL check constraints from being reversed correctly.

DI-5962: Process action TOOL Ldap Extractor(search): fixed an issue that inserted a blank line after the header, when header generation and table truncation were both switched on.

DI-8546: Added an option to retrieve all LDAP operational attributes in an integration flow.

DI-9443: Fixed a missing column error in the replicator template on the BigQuery Part for Load action.

DI-9640: Fixed an issue with WSDL metadata that generated an incorrect fault segment name.

DI-9877: Fixed an issue generating table names with a not exists clause.

DI-9620: Added Semarchy Data Intelligence harvesting features.

DI-8516: The XPath function tech:rejectedBy() in Reject Templates generates an incorrect expression.

DI-8786: Template INTEGRATION File (FileExport): missing column name in template.

DI-8272: The REST wizard mismatches the input and output nodes when performing a reverse engineering with WSDL technology.

DI-8427: Disable connection pooling for the file driver.

DI-8451: Remove unused metadata parameter from the file driver JDBC URL.

DI-8412: Improve the management of the reversed words supported by the parser.

DI-8248: Incorrect queries are generated when a mapping datastore is named "OFF".

DI-7233: Add Enable Certificate Validation and Enable Hostname Validation parameters in the WSDL Metadata.

DI:6670: RDBMS databases: add Nullable Reject Code and Nullable User Message metadata properties to columns.

DI-7329: The error BeanShell script error is thrown when processing JSON attributes having space characters in their name with DESERIALIZER Hsql String as Json and DESERIALIZER Rdbms String as Json.

DI-4407: The Create Stage Indexes parameter has been added to the STAGING Rdbms As Table template.

DI-4576: The Create Load Indexes option has been added to the LOAD File to Rdbms template.

DI-4654: The Create indexes on Target Table parameter has been added to the INTEGRATION Rdbms to Rdbms (no load) template.

DI-4658: The Create indexes on Target Table parameter has been added to the INTEGRATION Rdbms template.

DI-4659: The Create indexes on Target Table parameter has been added to the INTEGRATION Hsql template.

DI-4740: In the Replicator XML to Rdbms template, the Record Separator parameter allows changing the record separator used in temporary files.

DI-4876: The Post Processing Operation parameter has been added to the Load Xml To Rdbms template. It allows choosing a post-process operation for the source files.

DI-5312: The metadata now supports hexadecimal properties.

DI-5329: The INTEGRATION Rdbms to XML template now supports writing JSON data with unescaped slashes.

DI-5439: The Certificate and Keys metadata is added, to define certificates, keystores and truststores.

DI-5741: The core RDBMS product has been improved. This improvement allows defining in technology definitions whether a datatype supports the NULL or NOT NULL constraints at creation.

DI-5759: Rename Process Actions from "Execute" to "Run" for clarity.

DI-5836: The CONSTRAINT_NULL_DDL generic XPath function has been added for databases. This function allows generating the NULL constraint DDL query.

DI-5854: In Rdbms technologies, a Database Default Value property has been added on columns. This property represents the default database value configured for the column on the database side.

DI-6047: A dedicated metadata is available that contains the built-in runtime entry within the internal resources. This new entry replaces the previously hardcoded entry in the preferences and related view.

DI-6223: The cdcMaxTimestamp option has been added to the Replicator Rdbms to Rdbms tool.

DI-6253: The tech:cdcHistorizeViewPath and tech:cdcHistorizeViewName CDC XPath functions are now available for databases.

DI-6307: In a mapping, when changing the data type of a column, the properties are refreshed dynamically.

DI-6539: New template criteria can be provided in technologies using XPath expressions.

DI-6570: The tech:fieldPath() and tech:isRepeated() functions have been added to the XML technology.

DI-1267: When using the Replicator tool, the create table statement does not contain delimiters around column names when the source is a file.

DI-4739: In the LOAD XML to Rdbms template, the names of load tables are unexpectedly truncated before the addition of the suffixes.

DI-5027: In the Semarchy xDM Integration template, the column delimiters on generation conditions are handled only for SQL Server, which leads to syntax errors on other databases using different delimiters.

DI-5219: In the STAGING Rdbms As Table template, a space is missing between the table name and the default table storage clauses. This causes the query to fail.

DI-5339: In the INTEGRATION File (FileExport) template, the File Format parameter is ignored and the template always considers the file as delimited.

DI-5416: In the Unload Rdbms to File, the clean step does not clean the temporary file when the split parameter is not specified.

DI-5466: In the Replicator RDBMS tool, the target column name delimiter is not applied when replicating a file to an Rdbms.

DI-5573: The SELECT keyword is missing in the query used for checking alternate keys in the REJECT Hsql template.

DI-5850: The WARNING_FIELDS_NOT_FOUND warning is unexpectedly displayed when mapping a field with a dash in its name from a source file.

DI-5998: The tech:mappingName() XPath expression returns an empty value.

DI-6222: The Transaction Type option does not apply to the Rdbms to Snowflake specific mode.

DI-6670: [RDBMS databases] Add Nullable Reject Code and Nullable User Message metadata properties to columns.

DI-5998: The tech:mappingName() XPath expression returns an empty value.

DI-6382: The Outgoing Server (SMPTP) and Incoming Account nodes in the email metadata now support Microsoft Outlook’s modern authentication system based on a token mechanism. See Getting started with the Email Server

DI-6249: The core RDBMS product has been improved. The information about the databases' support for indexing is now included within the technology definition.

DI-5896: A new template has been added to write JSON files using the JSON Lines Format: INTEGRATION Rdbms to JSON Lines.

DI-6218: The Replicator Rdbms tool fails to correctly generate the names of the tables that use delimiters in their names.

DI-6181: The Replicator Rdbms tries to create indexes even when target databases do not support indexing.

DI-6158: The Template INTEGRATION File (FileExport) template does not respect the case of the source column names when generating the file header.

DI-5850: The warning "WARNING_FIELDS_NOT_FOUND" is unexpectedly displayed when mapping from a source file a field with a dash in its name.

DI-6047: Provide a dedicated metadata that contains the built-in runtime entry within the internal resources, which replaces the hardcoded entry previously in the preferences and related view.

DI-5836: Add a new generic XPath function for databases that allows generating a NULL constraint DDL.

DI-5741: Improve the core RDBMS product to allow defining in technology definitions if a datatype supports the NULL or NOT NULL constraints at creation.

DI-5573: Template - REJECT Hsql - Missing SELECT keyword in the query checking alternate keys.

DI-5416: Template - Unload Rdbms to File - the clean step does not clean the temporary file when the split parameter is not specified.

DI-5276: Template - Replicator Rdbms - the SQL query generated from the action "createIndexOnWorkTable" was incorrect, the schema name was unexpectedly doubled

DI-128: JSON Metadata: Reverse-engineering JSON schemas

DI-496: SCP Get/Send File Actions: Store the list of processed files in a database.

DI-546: Execute Delivery Action: New parameter to configure the session log level

DI-2058: Execute Delivery Action: New parameter to limit the number of parallel sessions

DI-2467: New "Execute SFTP Command" action

DI-2641: Reverse-engineer foreign keys referencing tables in another schema when the other table and schema are available in the metadata

DI-3296: SOAP (WSDL): Configure retries, functional response codes and error messages capture to handle errors occurring on API calls.

DI-3358: Get/Send Files with SCP Actions: New "SCP Preserve File Date" property to preserve file timestamps after the transfer.

DI-3361: SQL File Export Action: New "SQL Exp SQL Query File Encoding" property to define the encoding of the query file.

DI-3379: SQL File Export Action: support exporting named objects instead of SQL queries resultsets

DI-3439: SCP Get File Action: Added support to bind to actions the downloaded files.

DI-3456: File Metadata: Only allow valid values for the "expression" property of computed fields

DI-3558: Actions supporting encoding properties now list sample encoding values.

DI-3701: Allow Components to contribute to Designer monitored statistics

DI-4053: Query Editor menu renamed to "Launch Query Editor"

DI-4153: JSON writing - ability to choose how to write empty objects and arrays

DI-4220: XML / JSON Metadata - sort alphabetically the property fields for clarity

DI-4727: Rebranding: Templates and sample projects

DI-4731: Rebranding: Template messages

DI-4800: HTTP Security and HTTP Proxy are now included as base components

DI-4813: Rebranding: Drivers classes and URLs

DI-4850: Target Filter - Implement target filters in HSQL Templates

DI-4962: Improved component dependencies and requirements management

DI-4023: When renaming a mapping, the underlying generated process is not renamed accordingly

DI-5034: Databases - A Foreign Key could not be created manually in a sub-Metadata File

DI-4942: Databases - review to display order of the various items such as datastores, columns, query folder, and more to make sure they are displayed in a comprehensive order in Designer

DI-4852: RDBMS databases technologies - new attribute "Technical Columns Prefix" available, which will be supported by Templates gradually

DI-3971: Metadata Products - "Explicit Join in Bracket" attribute which is defined in products should be displayed with "true" as default value in the interface when no value is set, as this is the value used behind the scenes by Designer when no value is defined

DI-4348: Temporary exchange files using XML format will now use XML 1.1 format to avoid issues with some special characters

DI-4415: Reject message for Not Null checks unexpectedly changed for database rejects, which could cause some misunderstanding

DI-4419: Template - Replicator Rdbms - Columns of work table in synchronization mode were not generated in the correct order in statements

DI-4421: Template - Replicator Rdbms - Incorrect delete statement generated when deleting from MySQL database

DI-3978: WSDL Metadata - support to alternatively define the Proxy settings through a proxy node from Proxy Metadata, and making it supported in Reverse REST Wizard

DI-1164: Concat Files Process Action - when using append mode, a new additional parameter has been added to define if a separator should be placed or not between the concatenated files

DI-2965: Concat Files Process Action - new parameter to choose the order in which the files are processed

DI-3757: Foreign Key - ability to select in the "referenced column" attribute, a column from a table which is on another schema

DI-2604: JMS Send Message From File Process Action - the display name has been modified to be coherent between the one displayed in the palette and the one defined when creating the action

DI-3262: Reject constraints - "User Message", "Reject Code" and "Security Level" attributes have been moved back to standard tab for clarity and to make them also appear again in Mapping when consulting constraint properties

DI-2642: Metadata - "CreationDdl" XPath expression does not produce the correct schema in foreign key reference clause

DI-3471: XML and JSON Metadatas - The new Property Field named "nodeAbsolutePosition" has been added

DI-3472: XML and JSON Metadatas - The new Property Field named "nodeUUID" has been added

DI-1970: File Metadata - ability to sort data when writing files through a new Property Field named "sortKey", which supports SORT_DESC and SORT_ASC tags to define the sort order

DI-2031: File Metadata - A list of possible values for “Property field” has been added in new attribute named Property to allow user to select value he want to use

DI-2033: File Metadata - definition of which property fields can be used as source or target

DI-2420: SQL File Export Process Action - A new parameter named "SQL_EXP_STRING_DELIMITER_REPLACEMENT" has been added to allow to replace the string delimiter that is found inside the data when exporting data to file

DI-2657: Template - Integration Rdbms to XML - The WRITE_KEY_VALUES parameter uses a reference: this is changed to an XPath retrieving the parameter value, to allow overriding this value from other Templates which are reusing it

DI-811: SCP Actions and SSH Metadata - Added a new "SSH Additional Connection Properties" parameter to define specific connection properties

DI-1426: SQL File Export Process Action - new value available for "generate header" parameter to allow indicating that the header should always be added (even in append mode)

DI-1716: NTLM Authentication - adding a dedicated wizard on NTML security node, and support NTLM security node in HTTP Security Metadata

DI-2479: XML Metadata - generation of XSD file is now in a dedicated wizard named "XSD Generation Tool"

DI-2618: LOAD Xml to Rdbms Template: correction of the error message "Unable to load class: com.stambia.avro.reader.visitor.AvroDatatTypeConverter" with Avro

DI-2697: JSON Property fields - the "nodeName" property field was not appearing in some other Metadata which are reusing JSON technology, such as HTTP REST Metadata

DI-2538: Mapping technology (map.tech) - support cdcMaxTimestamp option

DI-2731: Mapping technology (map.tech) - New parameter on XPath function tech:workName() which is available on steps to be able to enabled / disable object delimiters

DI-2709: Mapping technology (map.tech) - Staging Mask was not working when using [sessionTimestampName] on a database which has object name truncation size

DI-2714: Template - Unload Rdbms To File - fix split column prefix which is incorrect when mapping is multi-target

DI-2732: Template - UNLOAD Rdbms to File - generated temporary file names may unexpectedly contain object delimiters

DI-2777: Template - Load Rdbms to Rdbms - Index names should not contain a schema on some target databases (such as MySQL)

DI-3396: Databases - A new attribute named "Report Schema" have been added

DI-3507: Template - Integration Rdbms to Xml - Improved WRITE_KEY_VALUE parameter to be reusable when this Template is used in another Template

DI-3710: Template - Load Xml To Rdbms - support custom loading mechanism with streamHook

DI-3138: Email Metadata and Send Email Process Action - A new key/ value attribute is added on Outgoing Server to define extra connection properties

DI-3053: Read Email Process Action - MAIL_EXTRA_PROPERTIES was ignored during read email action.

DI-3403: INTEGRATION Rdbms to Wsdl - The expression to get the Metadata attribute "Send Client Credential" has been corrected to be passed properly to the Template

DI-778: Send Mail Action - Added a new "Mail content file encoding" parameter to define the encoding to use to read the content of the file specified by the "Mail content file" parameter

DI-133: WSDL Metadata - Remove the name attribute on Fault node which is not required anymore, to have a consistent behaviour with Input and Output nodes which does not have name attribute either

DI-1357: INTEGRATION Rdbms to Rdbms (no load) - Addition of a new parameter "Fetch size" to configure the number of lines of each fetch of data when querying the source database

DI-1550: INTEGRATION File (FileExport) Template - new parameter to alternatively define the string delimiter as character instead of hexadecimal representation

DI-1767: TOOL Ldap Extractor (search) - new parameter 'attributesFormattedAsSidString'

DI-1769: UNLOAD Rdbms to file supports defining if boolean values should be exported as numeric values or boolean values

DI-1771: INTEGRATION Rdbms to Rdbms (no load) - Addition of a new parameter "Split Parallel Degree" to enable parallel loading of source data

DI-1893: UNLOAD Rdbms to File: add support for split

DI-1894: Add a tooltip on "Enable CDC" attribute of an Rdbms column

DI-1903: Mapping generation (map.tech) - CDC improvement - tech:hasCdc() XPath function now supports defining a custom "sourceSetName", which will, most notably, allow the support of CDC on Mapping stages

DI-1904: Mapping generation (map.tech) - CDC improvement - new XPath function has been added on a step to improve the retrieval of CDC filter part (tech:cdcFilterPart2())

DI-1905: Mapping generation (map.tech) - CDC improvement - The test to generate the CDC Filter part on a source now also checks the fact that a CDC source exists

DI-2194: INTEGRATION File (FileExport) Template - the default numeric format specified on Metadata is now used automatically to define the numeric format used by the Template, and a new parameter on the Template allows to override it

DI-2849: Support in product definition to define a new group by syntax using aliases

DI-2862: RDBMS databases Metadata - new attributes "Default Table Storage Clause" and "Default Index Storage Clause" on temporary objects tab

DI-2864: Generic Templates - support the "Default Table Storage Clause" and "Default Index Storage Clause" attributes which are defined in Metadata

DI-1788: Templates updated - support the "Default Table Storage Clause" and "Default Index Storage Clause" attributes which are defined in Metadata

DI-1912: Templates updated - support having CDC sources on Templates which were not supporting it (such as staging templates)

DI-1910: Templates updated - New parameter 'Cdc Subscriber' on Templates on which it was not handled yet

DI-1909: Templates updated - New Parameters 'Unlock Cdc Table' and 'Lock Cdc Table' to configure the behaviour of CDC tables locking

DI-1139: Web Services invocation - addition of new attributes in Metadata and Templates to declare a Connection Timeout and a Read Timeout

DI-1828: Web Service authentication with OAuth2 - a new parameter named "Send Client Credentials" has been added to define how client id and client secret should be sent, with the support of a new mode to send them in Basic Auth header

DI-820: INTEGRATION File Template - when using "generate header" parameter alongside with "truncate=true" and "line separator=after", the generate header line was unexpectedly missing a line separator after it

DI-1605: INTEGRATION HSQL - lock and Unlock CDC steps should be executed only on sources which are not loaded through a load step

DI-1666: SSH Metadata - [techCode] and [productCode] XPath functions were unexpectedly returning "FTP" instead of "SSH"

DI-1764: INTEGRATION Hsql - 'Append Mode' parameter is ignored in some situations

DI-1948: Mapping generation (map.tech) - CDC improvement - Mapping stages were not taken into account by tech:hasCdcSource(), leading to issues when working with CDC

DI-1949: Mapping generation (map.tech) - CDC improvement - unable to distinguish directly journalized sources and sources involving journalization, which can lead to CDC actions being executed in some situations while they shouldn’t

DI-1908: Templates updated - The 'Cdc Subscriber' parameter was ignored in some Templates on Lock / Unlock CDC steps

DI-1907: Templates updated - The 'Cdc Subscriber' parameter was ignored in some Templates when querying the source data

DI-1761: Mapping Transparent Stage - support using transparent stage feature with "Integration Rdbms to Wsdl" Template

DI-200: SQL To Parameters Process Action: a sample script is now provided in the body when creating the action

DI-523: New Process Action "Raise Error" available to raise an error with a defined message

DI-1967: Email Metadata and Read Email Process Action: on Incoming account, new attribute to provide additional key / value connection properties