Google BigQuery component release notes

This page lists the main features added to the Google BigQuery Component.

Changelog

Version 2024.4.0

This version of the connector requires the Base component at version 2024.4.0 or later.

Feature improvements

  • DI-10011: Updated BigQuery actions and templates to define partitions during table creation steps.

  • DI-10155: Template INTEGRATION BigQuery: added support for Slowly-Changing Dimensions (SCD).

  • DI-10298: Improved speed and efficiency when loading positional files to Google BigQuery.

  • DI-10241: Upgraded component libraries to the latest versions.

Bug fixes

  • DI-10269: Template Integration BigQuery: fixed an issue that truncated fields in the group by clause.

Version 2024.3.0

This version of the connector requires Semarchy xDI version 2024.3.0 or later.

Feature improvements

  • DI-4619: Added new template: LOAD Xml to BigQuery

  • DI-9266: Added new template: LOAD JSON to BigQuery

  • DI-9587: Updated main component libraries to the latest versions.

  • DI-9611: Added new template: INTEGRATION BigQuery to File

  • DI-9867: Added new template: LOAD Salesforce to BigQuery

Bug fixes

  • DI-9286: Template INTEGRATION BigQuery: fixed an issue preventing the template from doing stricter error handling.

  • DI-9287: Template Load Rdbms to BigQuery: fixed an issue that prevented xDI from loading the CDC_CHANGE_TYPE column.

  • DI-9654: Fixed an issue that generated incorrect timestamps.

  • DI-9856: Fixed an issue that prevented Salesforce integrations from recognizing OAuth2 JSON properties.

Version 2024.2.0

Feature improvements

  • DI-8884: Updated Google components' libraries and dependencies.

Version 2024.1.10

This version of the connector requires the xDI base component at version 2024.1.10 or later.

Bug fixes

  • DI-10269: Template Integration BigQuery: fixed an issue that truncated fields in the group by clause.

Version 2024.1.7

Bug fixes

  • DI-9286: Template INTEGRATION BigQuery: fixed an issue preventing the template from doing stricter error handling.

  • DI-9287: Template Load Rdbms to BigQuery: fixed an issue that prevented xDI from loading the CDC_CHANGE_TYPE column.

Version 2024.1.6

Bug fixes

  • DI-9856: Fixed an issue that prevented Salesforce integrations from recognizing OAuth2 JSON properties.

Version 2024.1.5

Bug fixes

  • DI-9654: Fixed an issue that generated incorrect timestamps.

Version 2024.1.0

Feature improvements

  • DI-7621: Implement serializer functionality in Designer. Use the serializer in mappings to parse structured data from a source and flatten it to string data.

  • DI-7967: Add support for retrying a query after the following errors: 500 internalError, 400 tableUnavailable, and 403 quotaExceeded.

  • DI-8597: Update Google components to the latest version.

Bug fixes

  • DI-8799: Error when parsing datetime fields.

Version 2023.4.0

Bug fixes

  • DI-7653: When Google BigQuery is the target, some source datatypes are not converted to the correct target datatypes.

Version 2023.3.0

Feature improvements

  • DI-5987: New parameters connectionTimeout and readTimeout can be defined in the JDBC URL.

  • DI-6915: New template DESERIALIZER BigQuery String as Json.

  • DI-7270: Target filters have been implemented in the INTEGRATION BigQuery template.

Bug fixes

  • DI-6228: LOAD GCS File to BigQuery: The error NullPointerException is thrown on the "select on source data" step when loading a file to bigquery with direct mode.

  • DI-7893: The error Access Denied is displayed when accessing an external table in BigQuery that points to a Google Drive Spreadsheet.

Version 2023.1.16

Bug fixes

  • DI-9286: Template INTEGRATION BigQuery: fixed an issue preventing the template from doing stricter error handling.

  • DI-9287: Template Load Rdbms to BigQuery: fixed an issue that prevented xDI from loading the CDC_CHANGE_TYPE column.

Version 2023.1.14

Bug fixes

  • DI-9856: Fixed an issue that prevented Salesforce integrations from recognizing OAuth2 JSON properties.

Version 2023.1.9

Feature improvements

  • DI-7967: Add support for retrying a query after the following errors: 500 internalError, 400 tableUnavailable, and 403 quotaExceeded.

Version 2023.1.7

Bug fixes

  • DI-8799: Error when parsing datetime fields.

Version 2023.1.5

Feature improvements

  • DI-8182: Third-Party library upgrade.

Bug fixes

  • DI-8207: Google BigQuery mappings using Google Storage as temporary location are unexpectedly failing because of permission issues.

Version 2023.1.4

Bug fixes

  • DI-7981: Fix conversion issues with INT64, FLOAT64, BIGNUMERIC, DATETIME, and TIME data types, which could produce errors such as SQLFeatureNotSupportedException when BigQuery is used as the source.

Version 2023.1.0

Feature improvements

  • DI-4919: The location used to run queries can now be changed using the jobsLocation parameter in the JDBC URL.

  • DI-5839: A retry mode has bee added to re-run the query when an internalerror or jobBackendError error message is returned.

  • DI-6041: Detailed information about the error is displayed when a Google Big Query job fails.

  • DI-6140: Multiple third-party libraries upgrade.

  • DI-6244: The Support DEFAULT constraint option is set to false by default on all datatypes on BigQuery.

Bug fixes

  • DI-3067: The Create table script action fails on a Bigquery Table.

  • DI-4955: In the Reject BigQuery template, when checking foreign keys, the EXIST clause is used instead of the JOIN clause.

  • DI-4956: In the Reject BigQuery template, null values are not taken into account when checking the uniqueness of alternative keys.

  • DI-5367: In the BigQuery Integration Template, Reject Recycling is ignored when working with partitioned tables.

  • DI-5475: The date and timestamp formats that can be specified on the SQL File Export action are ignored when exporting data from Google BigQuery.

  • DI-5717: The INTEGRATION BigQuery template fails when the source contains the < or > characters in filters or joins and when the enableTargetPartitionPruning parameter is set to true.

  • DI-5827: When mapping an SQL Server date column having a NULL value to a Google BigQuery column, the NullPointerException error is thrown.

  • DI-6277: Multiple third-party libraries upgrade.

Version 5.3.12 (Component pack)

Bug fixes

  • DI-7893: An Access Denied error is displayed when accessing an external table in BigQuery that points to a Google Drive spreadsheet.

  • DI-7981: Conversion issues with INT64, FLOAT64, BIGNUMERIC, DATETIME, and TIME data types cause SQLFeatureNotSupportedException errors when using Google BigQuery as a source.

  • DI-8207: Google BigQuery mappings that use Google storage as a temporary location fail unexpectedly because of permission issues.

  • DI-8799: Error when parsing datetime fields.

Version 5.3.8 (component pack)

Feature improvements

  • DI-6262: The technology definition has been updated to mention that indexes are not supported. Thus, the generic templates do not try to create indexes on this database.

Version 3.0.2 (component pack)

Bug fixes

  • DI-5367: Template - INTEGRATION BigQuery - Reject Recycling is unexpectedly ignored when working with partitioned tables.

Version 3.0.1 (component pack)

Feature improvements

  • DI-4939: Template - Integration BigQuery - when deletion strategy is set to "key", create the preintegration table systematically for performances purposes

Bug fixes

  • DI-4830: Template - Integration BigQuery - null values in columns being part of update or delete keys were not taken into account

  • DI-4957: Template - Integration BigQuery - null values were not taken into account on the action "Recycle of previous rejected data"

Version 3.0.0 (component pack)

Feature improvements

  • DI-2530: Added automatic reverse of partitioned tables when reversing BigQuery tables

  • DI-2795: Google BigQuery: Failing queries return information beyond the failed Job ID if available.

  • DI-3848: Google BigQuery partitions: Add "Year" and "Month" to the list of "Timestamp truncation" values.

  • DI-4508: Update Components and Designer to take into account dedicated license permissions

  • DI-4731: Rebranding: Template messages

  • DI-4813: Rebranding: Drivers classes and URLs

  • DI-4962: Improved component dependencies and requirements management

Version 2.2.6 (Google BigQuery component)

Feature improvements

  • DI-4939: Template - Integration BigQuery - when deletion strategy is set to "key", create the preintegration table systematically for performances purposes

Bug fixes

  • DI-4830: Template - Integration BigQuery - null values in columns being part of update or delete keys were not taken into account

  • DI-4957: Template - Integration BigQuery - null values were not taken into account on the action "Recycle of previous rejected data"

Version 2.2.5 (Google BigQuery component)

Bug fixes

  • DI-4510: Template - LOAD GCS File to BigQuery - when loading data into BigQuery the templates is not using the correct work schema

Version 2.2.4 (Google BigQuery component)

Bug fixes

  • DI-4698: BigQuery - table and column names were unexpectedly truncated to 30 characters

Version 2.2.3 (Google BigQuery component)

Bug fixes

  • DI-4532: Template - Integration BigQuery - Flag of rows to update step should also be executed in insert only mode

Version 2.2.2 (Google BigQuery component)

Feature improvements

  • DI-4295: TOOL - BigQuery Table Operations - add "truncate target table" parameter

  • DI-4516: TOOL - BigQuery Table Operation - use SQL syntax for Create table statement instead of scripting

Version 2.2.1 (Google BigQuery component)

Feature improvements

  • DI-4453: Template - INTEGRATION BigQuery - Expressions executed on target where not supported with merge incremental method

  • DI-4455: Template - INTEGRATION BigQuery - New parameter "Deletion Strategy"

  • DI-4456: Template - INTEGRATION BigQuery - New parameter "Truncate Target Table"

  • DI-4457: Template - INTEGRATION BigQuery - Create as select is now used to create integration table

  • DI-4458: Template - INTEGRATION BigQuery - New parameters "Enable Target Partition Pruning" and "Target Partition Filter"

  • DI-4460: Template - INTEGRATION BigQuery - Change partition boundaries calculation method so it executes once in a Mapping

  • DI-4461: Template - INTEGRATION BigQuery - When using a merge, a comparison is made on 'Non target' fields to minimize and optimize the number of updated records

  • DI-4462: Template - INTEGRATION BigQuery - remove option 'Useless Update Detection Method' which is not used anymore

Bug fixes

  • DI-4454: Template - INTEGRATION BigQuery - Merge did not work with doUpdate only mode

  • DI-4459: Template - INTEGRATION BigQuery - Fix insertion of columns that are supposed to be evaluated on target when being in append mode

Version 2.2.0 (Google BigQuery component)

Feature improvements

  • DI-3597: Addition of two new data types - BIGNUMERIC and BIGDECIMAL

Bug fixes

  • DI-1209: Template - INTEGRATION BigQuery - handle nullity in update key when performing updates or comparison with target table

  • DI-3331: Google Cloud Storage and Google BigQuery wizards - When launching the wizard on an existing Metadata, the first step of the wizard did not automatically select the account defined in Metadata

Version 2.1.2 (Google BigQuery component)

Feature improvements

  • DI-4297: Template - Reject BigQuery - when using "Drop Reject Table" parameter, the drop and re-creation of the reject table is now performed in one query using REPLACE or IF NOT EXISTS clauses for performances and optimizations purposes

Bug fixes

  • DI-4285: Template - Integration BigQuery - sometimes the preintegration table is unexpectedly created and populated even if it is not used later in the Template

  • DI-4298: Template - Reject BigQuery - when both "Drop Reject Tables" and "Delete Previous Rejects" parameters are enabled at the same time, the delete of previous rejects should not be performed because it is not necessary, as the table is recreated empty

Version 2.1.1 (Google BigQuery component)

Bug fixes

  • DI-4256: Template - Integration BigQuery - incorrect query might be generated in some situations when using deleteInsert integration strategy

Version 2.1.0 (Google BigQuery component)

Feature improvements

  • DI-3510: EMF compare utility - Component has been updated to support EMF Compare comparison utility

Bug fixes

  • DI-1245: Google BigQuery Templates - when loading data into BigQuery, some Templates were not using the correct working schema

  • DI-2378: Template - Load Rdbms to Bigquery - an error such as "No such field: L1" was thrown when there was multiple time this Load Template for the same target datastore (parallelism was not handled properly)

  • DI-1165: Template - "INTEGRATION Bigquery" and "Load Rdbms to BigQuery" - SQL_STAT_INSERT and SQL_STAT_UPDATE statistics were not computed and not displayed

  • DI-1277: Google BigQuery - When loading DATE (or TIME) type from Bigquery into some databases (such as PostgreSQL) an error such as the following could unexpectedly be thrown "l1_c_time is of type time without time zone but expression is of type character varying"

  • DI-3833: Google BigQuery - Randomly, the "EngineExceptionI: A class was not found in the engine" error was displayed even when the Module was complete, because there was an unexpected jar in the Module which caused conflicts, and which has now been removed

Version 2.0.9 (Google BigQuery component)

Feature improvements

  • DI-2782: Template - LOAD GCS File to BigQuery - Add support for file_name property field

Bug fixes

  • DI-2153: Template - Load Rdbms to Bigquery - CDC_CHANGE_TYPE column is unexpectedly not loaded

Version 2.0.8 (Google BigQuery component)

Feature improvements

  • DI-1009: Template - INTEGRATION BigQuery (Hierarchical) BETA - New Template to write into repeated / nested fields

  • DI-1052: The isRepeated attribute was not reversed in the metadata, to use repeated fields.

Bug fixes

  • DI-3407: Templates - "LOAD GCS File to BigQuery" and "LOAD GCS JSON to BigQuery" Templates were throwing an error when using credentials string - WrappedException: Wrapped java.lang.NumberFormatException: For input string: "{"

  • DI-3409: Templates - "Load GCS File to BigQuery" and "Load GCS Json to BigQuery" Templates - an exception was thrown when using credentials "string" authentication method (Null Pointer in Method Invocation BSF info)

  • DI-3480: BigQuery Metadata - Java Type used for NUMERIC columns was incorrect

  • DI-3491: When reversing a BigQuery table with multiple records some of the records (not all of them) was reversed on the wrong level.

  • DI-3565: Template - INTEGRATION BigQuery - SQL_STAT_INSERT and SQL_STAT_UPDATE statistics were not displayed

Version 2.0.7 (Google BigQuery component)

Feature improvements

  • DI-1632: Ability to define partitions in Metadata, and to optimize read requests in Mappings based on them

Version 2.0.6 (Google BigQuery component)

Feature improvements

  • DI-1781: BigQuery - Support using constant values in GROUP BY clauses by automatically using aliases

  • DI-1944: Support a new authentication mode named "applicationDefault" which searches for authentication credentials in the current environment automatically through default Google’s mechanism

  • DI-1912: Templates updated - support having CDC sources on Templates which were not supporting it (such as staging templates)

  • DI-1910: Templates updated - New parameter 'Cdc Subscriber' on Templates on which it was not handled yet

  • DI-1909: Templates updated - New Parameters 'Unlock Cdc Table' and 'Lock Cdc Table' to configure the behaviour of CDC tables locking

Bug fixes

  • DI-1908: Templates updated - The 'Cdc Subscriber' parameter was ignored in some Templates on Lock / Unlock CDC steps

  • DI-1907: Templates updated - The 'Cdc Subscriber' parameter was ignored in some Templates when querying the source data

Version 2.0.5 (Google BigQuery component)

Feature improvements

  • DI-2105: Google BigQuery Component - update module definition to propose more accurate driver and urls