Google BigQuery component release notes

DI-10011: Updated BigQuery actions and templates to define partitions during table creation steps.

DI-10155: Template INTEGRATION BigQuery: added support for Slowly-Changing Dimensions (SCD).

DI-10298: Improved speed and efficiency when loading positional files to Google BigQuery.

DI-10241: Upgraded component libraries to the latest versions.

DI-10269: Template Integration BigQuery: fixed an issue that truncated fields in the group by clause.

DI-4619: Added new template: LOAD Xml to BigQuery

DI-9266: Added new template: LOAD JSON to BigQuery

DI-9587: Updated main component libraries to the latest versions.

DI-9611: Added new template: INTEGRATION BigQuery to File

DI-9867: Added new template: LOAD Salesforce to BigQuery

DI-9286: Template INTEGRATION BigQuery: fixed an issue preventing the template from doing stricter error handling.

DI-9287: Template Load Rdbms to BigQuery: fixed an issue that prevented xDI from loading the CDC_CHANGE_TYPE column.

DI-9654: Fixed an issue that generated incorrect timestamps.

DI-9856: Fixed an issue that prevented Salesforce integrations from recognizing OAuth2 JSON properties.

DI-8884: Updated Google components' libraries and dependencies.

DI-10878: Updated component libraries to the latest versions.

DI-10269: Template Integration BigQuery: fixed an issue that truncated fields in the group by clause.

DI-9286: Template INTEGRATION BigQuery: fixed an issue preventing the template from doing stricter error handling.

DI-9287: Template Load Rdbms to BigQuery: fixed an issue that prevented xDI from loading the CDC_CHANGE_TYPE column.

DI-9856: Fixed an issue that prevented Salesforce integrations from recognizing OAuth2 JSON properties.

DI-9654: Fixed an issue that generated incorrect timestamps.

DI-7621: Implement serializer functionality in Designer. Use the serializer in mappings to parse structured data from a source and flatten it to string data.

DI-7967: Add support for retrying a query after the following errors: 500 internalError, 400 tableUnavailable, and 403 quotaExceeded.

DI-8597: Update Google components to the latest version.

DI-8799: Error when parsing datetime fields.

DI-7653: When Google BigQuery is the target, some source datatypes are not converted to the correct target datatypes.

DI-5987: New parameters connectionTimeout and readTimeout can be defined in the JDBC URL.

DI-6915: New template DESERIALIZER BigQuery String as Json.

DI-7270: Target filters have been implemented in the INTEGRATION BigQuery template.

DI-6228: LOAD GCS File to BigQuery: The error NullPointerException is thrown on the "select on source data" step when loading a file to bigquery with direct mode.

DI-7893: The error Access Denied is displayed when accessing an external table in BigQuery that points to a Google Drive Spreadsheet.

DI-9286: Template INTEGRATION BigQuery: fixed an issue preventing the template from doing stricter error handling.

DI-9287: Template Load Rdbms to BigQuery: fixed an issue that prevented xDI from loading the CDC_CHANGE_TYPE column.

DI-9856: Fixed an issue that prevented Salesforce integrations from recognizing OAuth2 JSON properties.

DI-7967: Add support for retrying a query after the following errors: 500 internalError, 400 tableUnavailable, and 403 quotaExceeded.

DI-8799: Error when parsing datetime fields.

DI-8182: Third-Party library upgrade.

DI-8207: Google BigQuery mappings using Google Storage as temporary location are unexpectedly failing because of permission issues.

DI-7981: Fix conversion issues with INT64, FLOAT64, BIGNUMERIC, DATETIME, and TIME data types, which could produce errors such as SQLFeatureNotSupportedException when BigQuery is used as the source.

DI-4919: The location used to run queries can now be changed using the jobsLocation parameter in the JDBC URL.

DI-5839: A retry mode has bee added to re-run the query when an internalerror or jobBackendError error message is returned.

DI-6041: Detailed information about the error is displayed when a Google Big Query job fails.

DI-6140: Multiple third-party libraries upgrade.

DI-6244: The Support DEFAULT constraint option is set to false by default on all datatypes on BigQuery.

DI-3067: The Create table script action fails on a Bigquery Table.

DI-4955: In the Reject BigQuery template, when checking foreign keys, the EXIST clause is used instead of the JOIN clause.

DI-4956: In the Reject BigQuery template, null values are not taken into account when checking the uniqueness of alternative keys.

DI-5367: In the BigQuery Integration Template, Reject Recycling is ignored when working with partitioned tables.

DI-5475: The date and timestamp formats that can be specified on the SQL File Export action are ignored when exporting data from Google BigQuery.

DI-5717: The INTEGRATION BigQuery template fails when the source contains the < or > characters in filters or joins and when the enableTargetPartitionPruning parameter is set to true.

DI-5827: When mapping an SQL Server date column having a NULL value to a Google BigQuery column, the NullPointerException error is thrown.

DI-6277: Multiple third-party libraries upgrade.

DI-7893: An Access Denied error is displayed when accessing an external table in BigQuery that points to a Google Drive spreadsheet.

DI-7981: Conversion issues with INT64, FLOAT64, BIGNUMERIC, DATETIME, and TIME data types cause SQLFeatureNotSupportedException errors when using Google BigQuery as a source.

DI-8207: Google BigQuery mappings that use Google storage as a temporary location fail unexpectedly because of permission issues.

DI-8799: Error when parsing datetime fields.

DI-6262: The technology definition has been updated to mention that indexes are not supported. Thus, the generic templates do not try to create indexes on this database.

DI-5367: Template - INTEGRATION BigQuery - Reject Recycling is unexpectedly ignored when working with partitioned tables.

DI-4939: Template - Integration BigQuery - when deletion strategy is set to "key", create the preintegration table systematically for performances purposes

DI-4830: Template - Integration BigQuery - null values in columns being part of update or delete keys were not taken into account

DI-4957: Template - Integration BigQuery - null values were not taken into account on the action "Recycle of previous rejected data"

DI-2530: Added automatic reverse of partitioned tables when reversing BigQuery tables

DI-2795: Google BigQuery: Failing queries return information beyond the failed Job ID if available.

DI-3848: Google BigQuery partitions: Add "Year" and "Month" to the list of "Timestamp truncation" values.

DI-4508: Update Components and Designer to take into account dedicated license permissions

DI-4731: Rebranding: Template messages

DI-4813: Rebranding: Drivers classes and URLs

DI-4962: Improved component dependencies and requirements management

DI-4939: Template - Integration BigQuery - when deletion strategy is set to "key", create the preintegration table systematically for performances purposes

DI-4830: Template - Integration BigQuery - null values in columns being part of update or delete keys were not taken into account

DI-4957: Template - Integration BigQuery - null values were not taken into account on the action "Recycle of previous rejected data"

DI-4510: Template - LOAD GCS File to BigQuery - when loading data into BigQuery the templates is not using the correct work schema

DI-4698: BigQuery - table and column names were unexpectedly truncated to 30 characters

DI-4532: Template - Integration BigQuery - Flag of rows to update step should also be executed in insert only mode

DI-4295: TOOL - BigQuery Table Operations - add "truncate target table" parameter

DI-4516: TOOL - BigQuery Table Operation - use SQL syntax for Create table statement instead of scripting

DI-4453: Template - INTEGRATION BigQuery - Expressions executed on target where not supported with merge incremental method

DI-4455: Template - INTEGRATION BigQuery - New parameter "Deletion Strategy"

DI-4456: Template - INTEGRATION BigQuery - New parameter "Truncate Target Table"

DI-4457: Template - INTEGRATION BigQuery - Create as select is now used to create integration table

DI-4458: Template - INTEGRATION BigQuery - New parameters "Enable Target Partition Pruning" and "Target Partition Filter"

DI-4460: Template - INTEGRATION BigQuery - Change partition boundaries calculation method so it executes once in a Mapping

DI-4461: Template - INTEGRATION BigQuery - When using a merge, a comparison is made on 'Non target' fields to minimize and optimize the number of updated records

DI-4462: Template - INTEGRATION BigQuery - remove option 'Useless Update Detection Method' which is not used anymore

DI-4454: Template - INTEGRATION BigQuery - Merge did not work with doUpdate only mode

DI-4459: Template - INTEGRATION BigQuery - Fix insertion of columns that are supposed to be evaluated on target when being in append mode

DI-3597: Addition of two new data types - BIGNUMERIC and BIGDECIMAL

DI-1209: Template - INTEGRATION BigQuery - handle nullity in update key when performing updates or comparison with target table

DI-3331: Google Cloud Storage and Google BigQuery wizards - When launching the wizard on an existing Metadata, the first step of the wizard did not automatically select the account defined in Metadata

DI-4297: Template - Reject BigQuery - when using "Drop Reject Table" parameter, the drop and re-creation of the reject table is now performed in one query using REPLACE or IF NOT EXISTS clauses for performances and optimizations purposes

DI-4285: Template - Integration BigQuery - sometimes the preintegration table is unexpectedly created and populated even if it is not used later in the Template

DI-4298: Template - Reject BigQuery - when both "Drop Reject Tables" and "Delete Previous Rejects" parameters are enabled at the same time, the delete of previous rejects should not be performed because it is not necessary, as the table is recreated empty

DI-4256: Template - Integration BigQuery - incorrect query might be generated in some situations when using deleteInsert integration strategy

DI-3510: EMF compare utility - Component has been updated to support EMF Compare comparison utility

DI-1245: Google BigQuery Templates - when loading data into BigQuery, some Templates were not using the correct working schema

DI-2378: Template - Load Rdbms to Bigquery - an error such as "No such field: L1" was thrown when there was multiple time this Load Template for the same target datastore (parallelism was not handled properly)

DI-1165: Template - "INTEGRATION Bigquery" and "Load Rdbms to BigQuery" - SQL_STAT_INSERT and SQL_STAT_UPDATE statistics were not computed and not displayed

DI-1277: Google BigQuery - When loading DATE (or TIME) type from Bigquery into some databases (such as PostgreSQL) an error such as the following could unexpectedly be thrown "l1_c_time is of type time without time zone but expression is of type character varying"

DI-3833: Google BigQuery - Randomly, the "EngineExceptionI: A class was not found in the engine" error was displayed even when the Module was complete, because there was an unexpected jar in the Module which caused conflicts, and which has now been removed

DI-2782: Template - LOAD GCS File to BigQuery - Add support for file_name property field

DI-2153: Template - Load Rdbms to Bigquery - CDC_CHANGE_TYPE column is unexpectedly not loaded

DI-1009: Template - INTEGRATION BigQuery (Hierarchical) BETA - New Template to write into repeated / nested fields

DI-1052: The isRepeated attribute was not reversed in the metadata, to use repeated fields.

DI-3407: Templates - "LOAD GCS File to BigQuery" and "LOAD GCS JSON to BigQuery" Templates were throwing an error when using credentials string - WrappedException: Wrapped java.lang.NumberFormatException: For input string: "{"

DI-3409: Templates - "Load GCS File to BigQuery" and "Load GCS Json to BigQuery" Templates - an exception was thrown when using credentials "string" authentication method (Null Pointer in Method Invocation BSF info)

DI-3480: BigQuery Metadata - Java Type used for NUMERIC columns was incorrect

DI-3491: When reversing a BigQuery table with multiple records some of the records (not all of them) was reversed on the wrong level.

DI-3565: Template - INTEGRATION BigQuery - SQL_STAT_INSERT and SQL_STAT_UPDATE statistics were not displayed

DI-1632: Ability to define partitions in Metadata, and to optimize read requests in Mappings based on them

DI-1781: BigQuery - Support using constant values in GROUP BY clauses by automatically using aliases

DI-1944: Support a new authentication mode named "applicationDefault" which searches for authentication credentials in the current environment automatically through default Google’s mechanism

DI-1912: Templates updated - support having CDC sources on Templates which were not supporting it (such as staging templates)

DI-1910: Templates updated - New parameter 'Cdc Subscriber' on Templates on which it was not handled yet

DI-1909: Templates updated - New Parameters 'Unlock Cdc Table' and 'Lock Cdc Table' to configure the behaviour of CDC tables locking

DI-1908: Templates updated - The 'Cdc Subscriber' parameter was ignored in some Templates on Lock / Unlock CDC steps

DI-1907: Templates updated - The 'Cdc Subscriber' parameter was ignored in some Templates when querying the source data

DI-2105: Google BigQuery Component - update module definition to propose more accurate driver and urls