TPS-4924-RT (cumulative patch) - 7.3

Version
7.3
Language
English (United States)
Product
Talend ESB
Module
Talend ESB

TPS-4924-RT (cumulative patch)

Info Value
Patch Name Patch_20210907_TPS-4924_v1-RT-7.3.1
Release Date 2021-09-07
Target Version 20200219_1130-7.3.1
Product affected Talend ESB Runtime

Introduction

This patch is cumulative. It includes the previous generally available patches from Talend ESB Runtime 7.3.1.

NOTE: To download this patch, liaise with your Support contact at Talend.

Fixed issues

This patch contains the following fixes:

APPINT

  • TPRUN-1846: ESB Runtime deploys unauthenticated Jolokia by default
  • TPRUN-1683: Unable to resolve org.talend.esb.event-logging.elasticsearch-client while installing Runtime patch
  • TPRUN-1043: Karaf patch for cve-2020-11980 in Talend ESB runtime 7.3.1
  • TPRUN-1235: Update of CXF to 3.3.11
  • TPRUN-1234: Update of Jetty to 9.4.39 or later
  • TPRUN-1232: Update json-smart(-action) to 2.4.7 in tesb-eventlogging
  • TPRUN-1046: Manage default passwords in Runtime: Remove, make changeable and encrypt
  • TPRUN-1091: [CVE HIGH] Correct vulnerable transitive dependency of Avro 1.8.2 on commons-compress
  • TPRUN-1090: Upgrade XStream to 1.4.17
  • TPRUN-1012: When a route is deployed to runtime with talend-data-mapper, it restarts/refreshes all the routes that have been deployed already to runtime
  • TPRUN-1099: Error on runtime start: Could not start the servlet context for context path []
  • TPRUN-733: [Runtime] Update CXF to 3.3.10 due to CVE-2020-1954
  • TPRUN-919: Update XML Graphics dependency in Syncope
  • TPRUN-915: Update Apache ActiveMQ to 5.15.15
  • APPINT-32936: CVE:Upgrade commons-codec-1.11 to 1.15
  • APPINT-32767: cREST overwrite Content-Language header on runtime
  • APPINT-32586: Upgrade XStream to 1.4.16
  • APPINT-32722: Update Json-smart to 2.4.2
  • APPINT-32247: Already deployed Routes get refreshed when deploying/undeploying Route with Groovy
  • APPINT-31889: (Runtime) Update Jackson version to 2.11.4
  • APPINT-32214: Unexpected logging to Talend ESB Karaf console
  • APPINT-32161: Update authorization test keys as they have expired
  • APPINT-31681: Don't require keystore configuration if signing events is disabled
  • APPINT-31916: CVE: Upgrade Jetty version
  • APPINT-31812: Performance issues with Runtime, SAM after upgrade
  • APPINT-31736: bean-validator: Unable to initialize 'javax.el.ExpressionFactory'
  • APPINT-31470: Error when trying to connect to WebSocket from Runtime
  • APPINT-31663: Update BouncyCastle to 1.68 to fix CVE-2020-28052
  • APPINT-31578: Issue applying TPS-4527/Patch_20201218_R2020-12_v1-RT-7.3.1: Error downloading mvn:org.bouncycastle/bcprov-jdk15on/1.65
  • APPINT-30779: High CPU consumption by ESB routes in 7.3.1 version
  • APPINT-30782: Already deployed Routes got refreshed when deploy the Route with camel-bean-validator, camel-aws, etc as dependencies
  • APPINT-30580: Update Jackson to 1.9.15-TALEND
  • APPINT-30326: Remove camel-quartz dependency from Event Logging
  • APPINT-30241: Update Jolokia to 1.6.2
  • APPINT-30238: Update Camel features to pick up CVE fixes
  • APPINT-30166: Update Snakeyaml to 1.26
  • APPINT-30125: Update Cryptacular version
  • APPINT-28497: Update Apache Tika to 1.24.1 + Jackrabbit to 2.18.6
  • APPINT-31051: No more authentication methods available
  • APPINT-30992: Conflicting spifly bundle versions leading to jetty random behaviour
  • APPINT-30676: ESB patch doesn't remove all previous talend-data-mapper features
  • APPINT-30396: Issue with 'sleep 10' during ESB patch installation
  • APPINT-30308: Error Updating talend-data-mapper in Unix
  • APPINT-29858: unresolved dependencies [(&(language=js)(objectClass=org.apache.camel.spi.LanguageResolver))] with Java 11
  • APPINT-29867: Avoid org.talend.libraries.jmx export META-INF.services
  • APPINT-29786: Problems using groovy.json
  • APPINT-29133: Update dom4j to 2.1.3
  • APPINT-28966: MQTT: consume messages published before client starts up
  • APPINT-29278: Swagger UI not getting updated
  • APPINT-29223: Unsolicited restart of Talend resources while deploying/undeploying routes
  • APPINT-28029: Update to use Spring 5.1.14.RELEASE

TPSVC

  • TPRUN-392 Update vulnerable ANT version in 7.3 JobServer
  • TPRUN-326 Change JobServer encryption to use aesGCM
  • TPSVC-16933 Update Jackson to 2.11.4 or exclude if not needed
  • TPSVC-16967 Update HttpClient version to 4.5.13
  • TPSVC-16969 Update Commons IO to 2.8.0
  • TPSVC-16934 Update BeanUtils to 1.9.4 and Bouncy Castle Provider to 1.68
  • TPSVC-13908 JobServer lifecycle broken in OSGi environments
  • TPSVC-16463: Upgrade jobserver dependency libraries version
  • TPSVC-14107: Parameter Delimiter tab (\t) treated as string in tFileOuputDelimited if artifact published fromTalend Studio 7.2 and task published in 7.1 updated
  • TPS-4318: JobServer memory leak related to ZeroMQ mailbox (TPSVC-12728)

TDM

  • TDM-8683 Update XStream version used by TDM
  • TDM-8856 Remove conflicting bundle mvn:org.talend.transform/org.apache.xml.resolver
  • TDM-8843 EDI ISA16 should be used for component repetition, but Talend Studio is using the default of instead and not picking up the mapped ':'
  • TDM-8810 cMAP - Output is lost if cMap is terminal
  • TDM-8761 Eclipse runtime:route of main project use map refer reference project's customer bean throw warning
  • TDM-8694 Message with single quote messes the XQuery
  • TDM-8681 Security: Upgrade Commons Collections
  • TDM-8682 Security: Hibernate dependency
  • TDM-8660 EDI Reader not reporting wrong element on certain errors
  • TDM-8659 tHMapRecord job run fail use spark 2.3 on 741 which created and works on 721
  • TDM-8648 [tHMap]HL7V2 Warnings are not shown in the Run Log when an HL7v2 transformation is used
  • TDM-8635 Remove dependency on DQ lib 6.0.1
  • TDM-8603 Issue with upgrade to Studio 7.3.1
  • TDM-8599 Replace avro-based configuration with regular JSON
  • TDM-8580 Job with multiple tRunJob fail with NoClassDefFoundError
  • TDM-8574 The specified value cannot be converted to the specified type
  • TDM-8571 Can't connect to mysql db with JDK11
  • TDM-8524 [internal] Prepare runtime for native compilation and GraalVM
  • TDM-8516 Hikari DataSource and associated pool are not closed when route is stopped
  • TDM-8484 Json with Map Group,structure can't show as csv
  • TDM-8482 JSON Writer produces wrong XML Attributes
  • TDM-8446 Facing memory issues with a job using TDM after migrating to 7.1
  • TDM-8415 Support Map Group as root when writing Avro datum
  • TDM-8409 tHMap with payload output of HL7V2 representation has an NPE execution error
  • TDM-8391 JSON: problem to write array of map
  • TDM-8364 TDM IO WriteURL broken
  • TDM-8363 Map isn't working after "R2020-09" patch installation (Error: "Input to cast cannot be atomized")
  • TDM-8359 Warning about overflow is incorrect for negative Cobol numbers
  • TDM-8327 NumberFormatException when running an imported project with a Map rep on the output map element
  • TDM-8326 Cobol Reader stops on 0xFF values with Variable Blocked format
  • TDM-8323 show document for json/xml structure with UTF-8 BOM encoding will return error
  • TDM-8318 Cobol Reader should silently truncate records with VB option
  • TDM-8308 Implicit Decimal Not In Output
  • TDM-8307 High memory usage by TDMEndpoint class in Runtime
  • TDM-8293 highlight is not right when show document for json with null element or invisible group
  • TDM-8225 cMap throws classcastException and not able to map a property from java bean
  • TDM-8217 Warning should not be issued for BTS and FTS segments
  • TDM-8210 Unable to MAP HL7 with CSV
  • TDM-8198 Export more packages in org.talend.transform.saxonpe.osgi
  • TDM-8163 Add new Function FormatDateTime
  • TDM-8125 DatabaseLookup creating new DataSources for each message on the ESB
  • TDM-8106 Remove dependency on org.codehaus.jackson in JSON io module
  • TDM-8094 Databaselookup fails on new runtime unless it is a top-level expression
  • TDM-8092 XML Reader should honor encoding set in the XML Representation
  • TDM-8089 Problem with camel headers when cJMS and cMap are used
  • TDM-8084 [7.3.1] Using thmap is getting an error when using a map with X12_5050_HIPPA structure
  • TDM-8074 Field alignment in positional flat file structures
  • TDM-7969 TDM adds unencrypted passwords to error message
  • TDM-7908 ReadNested within CSV or HashMap Representation fails
  • TDM-7789 CSV reader should use the optimization done for the CSV writer
  • TDM-7781 Result is incorrect when map attributes from xml to flat
  • TDM-7780 Result is incorrect when map attributes from xml to json
  • TDM-6896 Upgrade Saxon library to 9.9
  • TDM-6619 Mapper bundles in state 'Failure' after deployment
  • TPS-4793 [7.3.1] cMAP - Output is lost if cMap is terminal (TDM-8810)

Prerequisites

Consider the following requirements for your system:

  • Talend ESB Runtime 7.3.1 must be installed.

  • Depending on the product, {container} is Talend-ESB-V7.3.1/container/ or Talend-Runtime-V7.3.1/

  • Before applying the patch, and if old TDM patches have been installed (ie: org.talend.transform.runtime.distrib-X.Y.Z_yyyyMMdd_HHmm.zip), please check the repository files are actually available on system, using this command:

    karaf@trun()> feature:version-list talend-data-mapper | grep file
    Version             | Repository | Repository URL
    --------------------+------------+---------------------------------------------------------------------------------------------------------
    7.3.1.20200413_0622 |            | file:/opt/TALEND/org.talend.transform.runtime.distrib-7.3.1_20200413_0651/features.talend-esb.xml
    7.3.1.20200528_1359 |            | file:/opt/TALEND/org.talend.transform.runtime.distrib-7.3.1_20200528_1415/features.talend-esb.xml
    

Here for instance, check these files are available:

/opt/TALEND/org.talend.transform.runtime.distrib-7.3.1_20200413_0651/features.talend-esb.xml
/opt/TALEND/org.talend.transform.runtime.distrib-7.3.1_20200528_1415/features.talend-esb.xml

If not, make sure to re-extract the old TDM patches to make these files available at the above locations After successful execution of the current patch, these files can be removed

  • Before applying the patch, and if TAC is used, latest TAC patch should be installed

  • Before applying the patch, please change the following properties in file {container}/etc/org.apache.karaf.jaas.cfg

    encryption.enabled = true
    encryption.name = basic (or jasypt)
    
  • Only in case that you have custom changes made in {container}/etc/org.talend.remote.jobserver.server.cfg, you need to backup this file and set the following properties explicit to your file and then copy it back after the patch is applied

    # Set to false to avoid the creation of a temporary context file - useful for big data jobs where the default context is inside the job jar
    # Note that if the default context is not found JS automatically uses the command line to transfer context parameters even if
    # SECURE_CONTEXTS=true
    org.talend.remote.jobserver.commons.config.JobServerConfiguration.SECURE_CONTEXTS=true
    
    # Set the timeout(milliseconds) for job TRACE and STATS LOGS, value shouldn't be less than 0;
    # 0 millisecond means an infinite timeout
    org.talend.remote.jobserver.commons.config.JobServerConfiguration.TRACE_STATS_TIMEOUT=0
    
  • TPS-4318: JobServer memory leak related to ZeroMQ mailbox (TPSVC-12728) requires configuration in {container}/etc/org.talend.remote.jobserver.server.cfg:

    org.talend.remote.jobserver.server.TalendJobServer.ENABLED_PROCESS_MESSAGE=false
    
  • TPRUN-1846: feature tesb-jmx-http-agent based on jolokia has been removed due to security reasons. If jolokia is still needed, please manually use secured jolokia feature:

    feature:install jolokia
    
    Authorized users are declared in {container}/etc/users.properties
  • The patch replaces the files {container}/bin/trun, {container}/bin/trun.bat, {container}/bin/setmem, {container}/bin/setmem.bat, and {container}/bin/inc. If you have made previous changes to one of these files, you should move them to the file {container}/bin/setenv respectively {container}/bin/setenv.bat. These files are meant for customizations and will not be replaced during patch application.

Installation

Container

New Container (never started)

  • Extract & replace the content of ZIP directory container into {container} directory

Structure after extract & replace should be :

{container}
├───bin     : existing dir
├───deploy  : existing dir
├───etc     : existing dir superseeded by patch
├───...
├───patches : dir from current or previous patch
│   └───Patch_20210907_TPS-4924_v1-RT-7.3.1
│           patch.bat
│           patch.commands
│           patch.sh
│           patch_examples.bat
│           patch_examples.sh
├───system  : existing dir superseeded by patch
│   ├───... : existing dir
│   ├───... : directories from patch
├───...
{container}/patches/Patch_20210907_TPS-4924_v1-RT-7.3.1/patch_examples.bat{container}/patches/Patch_20210907_TPS-4924_v1-RT-7.3.1/patch_examples.sh

Existing Container

  • WARNING: As the patch contains upgrades of complete frameworks, environment variable JAVA_MAX_MEM should be set to a sufficiently high value, at least 2048M
  • Start Runtime Container
  • Extract & replace the content of ZIP directory container into {container} directory

Structure after extract & replace should be :

{container}
├───bin     : existing dir
├───deploy  : existing dir
├───etc     : existing dir superseeded by patch
├───...
├───patches : dir from current or previous patch
│   └───Patch_20210907_TPS-4924_v1-RT-7.3.1
│           patch.bat
│           patch.commands
│           patch.sh
│           patch_examples.bat
│           patch_examples.sh
├───system  : existing dir superseeded by patch
│   ├───... : existing dir
│   ├───... : directories from patch
├───...
  • Ensure username/password are right in {container}/patches/Patch_20210907_TPS-4924_v1-RT-7.3.1/patch.bat or {container}/patches/Patch_20210907_TPS-4924_v1-RT-7.3.1/patch.sh

    ... -u {username} -p {password} -f patch.commands ... 
    
  • Execute {container}/patches/Patch_20210907_TPS-4924_v1-RT-7.3.1/patch.bat or {container}/patches/Patch_20210907_TPS-4924_v1-RT-7.3.1/patch.sh

  • Ensure directory {container}/patches/Patch_20210907_TPS-4924_v1-RT-7.3.1/ contains new log files :
  • xxx-installation.log: patch installation log
  • xxx-init.log: state before patch installation
  • xxx-installed.log: state after patch installation

Please note that Routes using cMap (TDM feature) are not automatically restarted by the patch procedure.
You will need to restart the Runtime Container for changes to take effect.
etc/keystoresetc/keystores-backup-TIMESTAMP-backup-TIMESTAMP