What's new in R2020-08 - 7.3

Talend Big Data products Release Notes

author
Talend Documentation Team
EnrichVersion
7.3
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Open Studio for Big Data
Talend Real-Time Big Data Platform
task
Installation and Upgrade
Release Notes

The R2020-08 Studio monthly release contains the following new features.

Big Data: new features

Feature

Description

Product

Lightweight dependencies for CDH 6.x
When you run a Job on CDH 6.x distributions, you can reduce the time spent by the Job to be launched when you select the Use lightweight dependencies check box from the Spark configuration tab in the Run view. It reduces the number of libraries to only the Talend libraries. This could prevent issues about dependencies, missing signature, wrong JAR version or missing JAR for example. With this option, you can use another classpath, different from the Cloudera default one, by selecting the Use custom classpath check box and entering the JARs you want to use in a regex syntax separated by a comma.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

Customizing precision in the schema for output components You can now select a precision different from the standard for the BigDecimal type when you update the output schema for the following components:
  • tDeltaLakeOutput
  • tFileOutputDelimited
  • tFileOutputParquet
  • tHiveOutput

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

tS3Configuration: setting the name of the DynamoDB table in EMRFS When you use the EMRFS consistent view option, you can enter the name of the metadata DynamoDB table to be used.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

tDeltaLakeInput and tDeltaLakeOutput: new path available to store the data You can specify an external path to another filesystem different from the DBFS (ADLS Gen2 or S3) to store the data.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

tDeltaLakeOutput: new operations provided in the Action property You can drop a table which results in the deletion and recreation of the table. You can also truncate a table which results in the deletion of data while the schema remains.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

tDeltaLakeOutput: new optimize property provided in the Basic settings view You can optimize the layout of Delta Lake data on Databricks.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

Using a Hadoop configuration file with Spark Batch and Spark Streaming Jobs You can connect Spark Batch and Spark Streaming Jobs to a Hadoop cluster in the Repository using a configuration JAR file. You specify the path to this file either in the Spark Configuration of the Job or in the Hadoop cluster configuration. This option is available for both Yarn cluster and Yarn client on non-Cloud distributions. Optionally, you can contextualize this connection parameter to automatically connect to the right cluster based on the environment in which you run the Job.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

Support for High Availability for EMR 5.23 or later

High availability is available when you run Talend Jobs with the Amazon EMR distribution in version 5.23 or later. You can now have multiple master nodes in your cluster.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

Data Integration: new features

Feature

Description

Product

tDataprepRun enhancement The tDataprepRun component now shows an error message when creating a new preparation with dynamic schema.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

tELTMap enhancement In the ELT Map Editor of the tELTMap component, you can now enter a multi-line expression for an output column and access proposals including input columns, output columns, and context variables by pressing Ctrl+Space in a new pop-up dialog box.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

POM generation enhancement A new option Exclude deleted items is available for generating the POM file for Maven build. With this option selected, the modules for the deleted items will be excluded from the POM file of current project, and the source for the deleted test cases will not be generated.
Note: You need to resynchronize the POM file to apply the new settings of this option.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

Talend type mapping enhancement You can now set the default pattern for each date type in Talend type mapping file. This allows the date pattern for the date type columns to be set automatically when retrieving or guessing schema from a table.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

tSSH library upgraded

Ganymed is now deprecated and the component now supports a new library: Apache mina-sshd.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

tSalesforceInput: new query mode provided

The tSalesforceInput component provides the BulkV2 query mode, which allows you to query even larger amount of data. The component also provides the Split query results into small sets option for the BulkV2 mode, allowing you to split query results into sets that are of specific size.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

Formatting date using date pattern defined in schema

The tSnowflakeOutput and the tSnowflakeOutputBulkExec components provide the Use schema date pattern option, allowing you to format dates using the date pattern defined in the schema.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

tSnowflakeInput: option renamed and improved

The Allow snowflake to convert columns and tables to uppercase option changes to Use unquoted object identifiers, with the function improved.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

tFTPRename: table column name fixed

The tFTPRename component supports only filenames in the Files field and the column name Filemask changes to Filename.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

Data Quality: new features

Feature

Description

Product

Phone number standardization Phone numbers can now be validated for a given region:
  • New column in the tStandardizePhoneNumber component: isValidPhoneNumberForRegion
  • New Phone number statistics indicator: Valid Phone Number for Region Count

The Google libphonenumber library has also been updated to the most recent version.

Talend Big Data Platform

Talend Real-Time Big Data Platform

ESB: new features

Feature

Description

Product

Microservices

Camel metrics are now exposed to Prometheus in Microservices to monitor the execution of Routes, JVM memory, CPU consumption, and so on.

Talend Real-Time Big Data Platform

Continuous Integration: new features

Feature

Description

Product

POM file generation - new parameter The mvn org.talend.ci:builder-maven-plugin:7.3.3:generateAllPoms command allows you to re-generate all .pom files of a project before building it. It is also useful in case you want to test a new version of the product before migrating it.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

Custom script - new parameter The mvn org.talend.ci:builder-maven-plugin:7.3.3:executeScript command allows you to write your own script with CommandLine commands and execute it at build time.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

Camel metrics exposure to Prometheus - new parameter You are now able to use the -Dstudio.prometheus.metrics=true parameter while publishing ESB artifacts to Docker in order to expose Camel metrics to Prometheus and get more details about the deployed Routes.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

Debug mode - new parameter You are now able to use the -Dstudio.talendDebug=true parameter to get additional logs. This parameter could be useful when trying to debug build issues with the support team.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform

Build improvement Depending on your Talend Studio project settings, all project items that are located in the recycle bin can now be excluded from the Continuous Integration build.

Talend Big Data

Talend Big Data Platform

Talend Real-Time Big Data Platform