R2023-01-RT (monthly release cumulative patch)

Introduction

This patch is cumulative. It includes the previous generally available patches from Talend ESB Runtime 7.3.1.R2022-09-RT.

NOTE: To download this patch, liaise with your Support contact at Talend.

Fixed issues

This patch contains the following fixes:

TESB

• TPRUN-5023: CVE-2022-46364 - update CXF to 3.4.10
• TPRUN-4777: CVE-2022-45589 - SQL Injection attacks vulnerability (since 7.3.1-2022-09-RT)
• TPRUN-5025: [7.3] Update ehcache to version 3 in tesb-authorization
• TPRUN-5020: CVE-2022-40145 - backport security fix to TESB customized Karaf
• TPRUN-4027: [7.3.1] Exception when executing route with groovy
• TPRUN-4706: Integrate jobserver 7.3.1.20221206_1150_patch
• TPRUN-4971: [7.3.1] CVE-2022-30126,org.apache.tika:tika-core:1.27 - update to tika 1.28.4
• TPRUN-4561: CVE-2022-42889, org.apache.commons:commons-text:[1.4-1.9]
• TPRUN-4972: [7.3.1] Prevent runtime patches > R2022-07 from installing on default install
• TPRUN-4976: [7.3.1] Update release notes with gen1/runtime common update reco
• TPRUN-4290: CVE-2022-34169: Xalan 2.7.2 is removed
• TPRUN-4514: CVE-2022-42003,CVE-2022-42004, jackson-databind-2.13.2.2.jar
• TPRUN-4414: CVE-2022-40149: jettison upgrade to 1.5.1
• TPRUN-4695: Make access port configurable in tesb-derby-starter
• TPRUN-4871: [CVE-2022-31692] Spring-security update to 2.6.9.
• TPRUN-4497: Fail to execute "feature:install camel-spring-redis" on Runtime

Job Server

• TPRUN-3405: The FileListener does not jail the path to the jobserver deploy directory.
• TPRUN-1296: Backport 'Prevent path manipulation attack in the FileServer' to 7.3.
• TPRUN-3450: JobServer should not weaken TLS in the TACClient (backport to 7.3)
• TPRUN-3451: CommandServer Denial of Service vulnerability (backport to 7.3)
• TPRUN-3508: AuthorizationKey is logged
• TPRUN-3697: JobServer should close stream of temporary context.
• TPRUN-3604: Unzipper Incorrect size limit check and created files not deleted in case of error
• TPRUN-3777: Non thread safe ClasspathJar writing
• TPRUN-3679: Modularize function required for user impersonation.
• TPS-5285: [7.3.1] Code cleanup & deprecation of 'launchFromShellScript' (TPRUN-3775)
• TPRUN-3605: Unzipper add limits for nesting and path length.
• TPRUN-3784: Update JobServer configuration/docs related to TLS version
• TPRUN-3948: Align versions of JAVA source/target, dependencies and plugins on pom(s).xml
• TPS-5359: [7.3.1] JobServer File server has no authentication. (TPRUN-3518)
• TPRUN-4022: Update patch creation process
• TPRUN-3916: Use RockyLinux as base image for JobServer docker in tests
• TPRUN-4131: Check Zip Slip and Zip Symlink vulnerabilities
• TPRUN-4126: Upgrade to OSHI 6.2.2
• TPRUN-3836: Improve error message in case Job archive checks fail
• TPRUN-3523: Add ability to disable the monitoring service
• TPRUN-1740: Simplify approach to let users install patches and (windows) services
• TPRUN-4023: Reduce merging pain between active branches due to different logging framework
• TPRUN-4267: Folder name length check not working for ZIP without folder entries
• TPRUN-4238: Attempt to publish a large job (while FileServer authentication is available?) causes a command server timeout
• TPRUN-4400: JobServer client checkServer returns wrong compatibility info
• TPRUN-4255: Do not log warnings when properties are not set but default value exists
• TPRUN-4355: Ensure Copyright is up-to-date for JAVA classes with UnitTesting
• TPRUN-4269: After Unzipper Exception partially unzipped file remain
• TPRUN-3519: Add constraints on jobs to prevent DoS attacks
• TPS-5372: [7.3.1] Adding File path traversal guard (TPRUN-4050)
• TPRUN-4515: Delete deployedJobPath directory before re-deploying
• TPRUN-4486: JobServer - Cleanings
• TPRUN-4447: JobServer start_jconsole.bat script has wrong classpath
• TPRUN-4761: Issue with FileEventsPacket
• TPRUN-4048: Review Merge compulsory requirements
• TPRUN-4005: Reading issue due to improper locking of job resuming log
• TPRUN-3520: Check job archive signature
• TPRUN-4753: Job archives that do not have a signature can be executed
• TPS-5388: [7.3.1] Reading issue due to improper locking of job resuming log ( TPRUN-4005 )
• TPRUN-4523: Update osgi.cmpn to 5.0.0+ and org.osgi.core to 6.0.0+
• TPRUN-4892: parallel send protection error with tac and virtual servers
• TPRUN-4898: JobServer checks cause problems for TAC deployments

TDM

• TDM-9289 Remove ExecutionProperties from the ExecutionStatus
• TDM-9278 [OldRuntime]Execution status is accumulated when there are multiple executions for a tHMap
• TDM-9226 Null item in JSON array is omitted on output
• TDM-9178 CVE: org.hibernate:hibernate-core:[5.0.9-5.3.20.Final]
• TDM-9033 Add representation options to reduce size of JSON output
• TDM-9029 NullPointerException on Show Document for JSON not matching data
• TDM-9018 tuj can't stopped (job tdmDIColumnsSingleColumn_ParallelizedJob can't be stopped)
• TDM-8946 Add capability to put and get values in a hashmap saved in the Runtime ExecutionProperties
• TDM-8927 One xml structure show as csv get error
• TDM-8903 Expression with combination of 0-scale Decimal and Trim input option fails
• TDM-8951 Restarting ESB Runtime produces 'Resource is not open' error in log
• TDM-8851 Option to wrap the output to the array even if there is a single object
• TDM-8683 Update XStream version used by TDM
• TDM-8856 Remove conflicting bundle mvn:org.talend.transform/org.apache.xml.resolver
• TDM-8843 EDI ISA16 should be used for component repetition, but Talend Studio is using the default of \ instead and not picking up the mapped ':'
• TDM-8810 cMAP - Output is lost if cMap is terminal
• TDM-8761 Eclipse runtime:route of main project use map refer reference project's customer bean throw warning
• TDM-8694 Message with single quote messes the XQuery
• TDM-8681 Security: Upgrade Commons Collections
• TDM-8682 Security: Hibernate dependency
• TDM-8660 EDI Reader not reporting wrong element on certain errors
• TDM-8659 tHMapRecord job run fail use spark 2.3 on 741 which created and works on 721
• TDM-8648 [tHMap]HL7V2 Warnings are not shown in the Run Log when an HL7v2 transformation is used
• TDM-8635 Remove dependency on DQ lib 6.0.1
• TDM-8603 Issue with upgrade to Studio 7.3.1
• TDM-8599 Replace avro-based configuration with regular JSON
• TDM-8580 Job with multiple tRunJob fail with NoClassDefFoundError
• TDM-8574 The specified value cannot be converted to the specified type
• TDM-8571 Can't connect to mysql db with JDK11
• TDM-8524 [internal] Prepare runtime for native compilation and GraalVM
• TDM-8516 Hikari DataSource and associated pool are not closed when route is stopped
• TDM-8484 Json with Map Group,structure can't show as csv
• TDM-8482 JSON Writer produces wrong XML Attributes
• TDM-8446 Facing memory issues with a job using TDM after migrating to 7.1
• TDM-8415 Support Map Group as root when writing Avro datum
• TDM-8409 tHMap with payload output of HL7V2 representation has an NPE execution error
• TDM-8391 JSON: problem to write array of map
• TDM-8364 TDM IO WriteURL broken
• TDM-8363 Map isn't working after "R2020-09" patch installation (Error: "Input to cast cannot be atomized")
• TDM-8359 Warning about overflow is incorrect for negative Cobol numbers
• TDM-8327 NumberFormatException when running an imported project with a Map rep on the output map element
• TDM-8326 Cobol Reader stops on 0xFF values with Variable Blocked format
• TDM-8323 show document for json/xml structure with UTF-8 BOM encoding will return error
• TDM-8318 Cobol Reader should silently truncate records with VB option
• TDM-8308 Implicit Decimal Not In Output
• TDM-8307 High memory usage by TDMEndpoint class in Runtime
• TDM-8293 highlight is not right when show document for json with null element or invisible group
• TDM-8225 cMap throws classcastException and not able to map a property from java bean
• TDM-8217 Warning should not be issued for BTS and FTS segments
• TDM-8210 Unable to MAP HL7 with CSV
• TDM-8198 Export more packages in org.talend.transform.saxonpe.osgi
• TDM-8163 Add new Function FormatDateTime
• TDM-8125 DatabaseLookup creating new DataSources for each message on the ESB
• TDM-8106 Remove dependency on org.codehaus.jackson in JSON io module
• TDM-8094 Databaselookup fails on new runtime unless it is a top-level expression
• TDM-8092 XML Reader should honor encoding set in the XML Representation
• TDM-8089 Problem with camel headers when cJMS and cMap are used
• TDM-8084 [7.3.1] Using thmap is getting an error when using a map with X125050HIPPA structure
• TDM-8074 Field alignment in positional flat file structures
• TDM-7969 TDM adds unencrypted passwords to error message
• TDM-7908 ReadNested within CSV or HashMap Representation fails
• TDM-7789 CSV reader should use the optimization done for the CSV writer
• TDM-7781 Result is incorrect when map attributes from xml to flat
• TDM-7780 Result is incorrect when map attributes from xml to json
• TDM-7427 data type optional segment is in test run result
• TDM-6896 Upgrade Saxon library to 9.9
• TDM-6619 Mapper bundles in state 'Failure' after deployment
• TPS-4793 [7.3.1] cMAP - Output is lost if cMap is terminal (TDM-8810)

Prerequisites

Consider the following requirements for your system:

• Talend ESB Runtime 7.3.1.R2022-09-RT must be installed.

• Depending on the product, {container} is Talend-ESB-V7.3.1.R2022-09-RT/container/ or Talend-Runtime-V7.3.1.R2022-09-RT/

• Before applying the patch, and if old TDM patches have been installed (ie: org.talend.transform.runtime.distrib-X.Y.Z_yyyyMMdd_HHmm.zip), please check the repository files are actually available on system, using this command:

  karaf@trun()> feature:version-list talend-data-mapper | grep file
  Version             | Repository | Repository URL
  --------------------+------------+---------------------------------------------------------------------------------------------------------
  7.3.1.R2022-09-RT.20200413_0622 |            | file:/opt/TALEND/org.talend.transform.runtime.distrib-7.3.1.R2022-09-RT_20200413_0651/features.talend-esb.xml
  7.3.1.R2022-09-RT.20200528_1359 |            | file:/opt/TALEND/org.talend.transform.runtime.distrib-7.3.1.R2022-09-RT_20200528_1415/features.talend-esb.xml
  

Here for instance, check these files are available:

/opt/TALEND/org.talend.transform.runtime.distrib-7.3.1.R2022-09-RT_20200413_0651/features.talend-esb.xml
/opt/TALEND/org.talend.transform.runtime.distrib-7.3.1.R2022-09-RT_20200528_1415/features.talend-esb.xml

If not, make sure to re-extract the old TDM patches to make these files available at the above locations After successful execution of the current patch, these files can be removed

• Before applying the patch, and if TAC is used, latest TAC patch should be installed

• Before applying the patch, please change the following properties in file {container}/etc/org.apache.karaf.jaas.cfg

  encryption.enabled = true
  encryption.name = basic (or jasypt)
  

For all inserted properties:

• if property already present (commented or uncommented), won't insert
• if property not already present, will backup related file in dir {container}/patches/Patch_20230105_R2023-01_v1-RT-7.3.1.R2022-09-RT/backup/ and insert property

For all updated properties:

• if property commented or not already present, won't update
• if property already present, will backup related file in dir {container}/patches/Patch_20230105_R2023-01_v1-RT-7.3.1.R2022-09-RT/backup/ and update property

If any change required, update value after patch execution.

• Patch will insert these properties in {container}/etc/org.talend.remote.jobserver.server.cfg:

  # Set password of server side ssl key (command and file server) - optional
  org.talend.remote.server.ssl.keyPassword=<jobserver_key_password>
  # Set password of server side ssl key (monitoring server) - optional
  org.talend.jmxmp.ssl.keyPassword=<monitoring_server_key_password>
  # Set this to true to disable hostname verification for the TACClient  - optional
  #org.talend.remote.jobserver.commons.config.JobServerConfiguration.TLS_DISABLE_CN_CHECK=true
  # Max size in bytes that an unzipped archive is allowed to be. The default is 1G.
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_UNZIPPED_SIZE=1073741824
  # Max number of entries allowed in a zipped archive
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ZIPPED_ENTRIES=2048
  # Maximum length of zip file names:
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ZIP_NAME_LENGTH=240
  # Restrict the length of any folder name in paths inside the zip file:
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_UNZIPPED_FOLDER_NAME_LENGTH=240
  # Restrict the length of any file name inside the zip file:
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_UNZIPPED_FILE_NAME_LENGTH=240
  # Restrict the nesting levels of folders inside the zip file:
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ZIP_DEPTH=64
  # Enable the Monitoring port or not. true by default
  org.talend.remote.jobserver.server.TalendJobServer.ENABLE_MONITORING_PORT=true
  # Set to true to enable authorization for all job file deployments ( Requires additional configuration for TAC and Studio. )
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.FILESERVER_AUTHORIZATION=false
  # Maximum number of file listeners, 0 = No limit
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_FILE_LISTENERS=6000
  # Maximum number of library dependencies embedded in a job
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_LIBRARY_DESC_NB=1000
  # Maximum size of all library dependency names for a job
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_LIBRARY_DESC_SIZE=100KB
  # Maximum number of deployed jobs, 0 = No Limit
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_JOB_NB=6000
  
  # Max size that a job archive is allowed to be. The default is 1G, 0 = No limit
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_JOB_FILE_SIZE=1G
  
  # Maximum size of TalendJobServersFiles/archiveJobs folder, 0 = No limit, 0 = No Limit
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ARCHIVES_DIR_SIZE=100G
  
  # Activate job archive signature, 1 or more values separated by comma (',').
  # Possible values are:
  # - 'ON_DEPLOY' (legacy & default if no correct value provided)
  # - 'ON_UPLOAD' (advised)
  org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_ARCHIVE_SIGNATURE_CHECK=ON_DEPLOY
  

• TPS-4318: JobServer memory leak related to ZeroMQ mailbox (TPSVC-12728) requires configuration in {container}/etc/org.talend.remote.jobserver.server.cfg:

  org.talend.remote.jobserver.server.TalendJobServer.ENABLED_PROCESS_MESSAGE=false
  

• TPRUN-1846: feature tesb-jmx-http-agent based on jolokia has been removed due to security reasons. If jolokia is still needed, please manually use secured jolokia feature:

  feature:install jolokia
  

  Authorized users are declared in {container}/etc/users.properties

• TPRUN-3009: default configuration in {container}/etc/org.talend.esb.auxiliary.storage.service.cfg for key security.signature.properties is:

  security.signature.properties = file:${tesb.home}/etc/keystores/serviceKeystore.properties
  

  if custom changes have been made, ensure the value references an absolute path.
  For instance, if expected keystore is {container}/etc/customKeystore.properties, this previous declaration:

  security.signature.properties = customKeystore.properties
  

  should be updated to:

  security.signature.properties = file:${tesb.home}/etc/keystores/customKeystore.properties
  

• The patch replaces the files {container}/bin/trun, {container}/bin/trun.bat, {container}/bin/setmem, {container}/bin/setmem.bat, and {container}/bin/inc. If you have made previous changes to one of these files, you should move them to the file {container}/bin/setenv respectively {container}/bin/setenv.bat. These files are meant for customizations and will not be replaced during patch application.

Installation

Container

• Start Runtime Container
• Extract & replace the content of ZIP directory container into {container} directory

Structure after extract & replace should be :

{container}
├───bin     : existing dir
├───deploy  : existing dir
├───etc     : existing dir
├───...
├───patches : dir from current or previous patch
│   └───Patch_20230105_R2023-01_v1-RT-7.3.1.R2022-09-RT
│           patch.bat
│           patch.commands
│           patch.sh
│           logs : directory for logs installation
├───system  : existing dir
│   ├───... : existing dir
├───...

• Ensure username/password are right in {container}/patches/Patch_20230105_R2023-01_v1-RT-7.3.1.R2022-09-RT/patch.bat or {container}/patches/Patch_20230105_R2023-01_v1-RT-7.3.1.R2022-09-RT/patch.sh

  ... -u {username} -p {password} -f patch.commands ... 
  

• Execute {container}/patches/Patch_20230105_R2023-01_v1-RT-7.3.1.R2022-09-RT/patch.bat or {container}/patches/Patch_20230105_R2023-01_v1-RT-7.3.1.R2022-09-RT/patch.sh

• Ensure directory {container}/patches/Patch_20230105_R2023-01_v1-RT-7.3.1.R2022-09-RT/logs contains new log files :
  • xxx-installation.log: patch installation log
  • xxx-init.log: state before patch installation
  • xxx-installed.log: state after patch installation

    Please note that Routes using cMap (TDM feature) are not automatically restarted by the patch procedure.
    You will need to restart the Runtime Container for changes to take effect.
    

    Runtime patches may contain Java keystore updates. In this case, the previous keystores are preserved in the following places:
• Keystores from etc/keystores are backed up in directory {container}/patches/Patch_20230105_R2023-01_v1-RT-7.3.1.R2022-09-RT/backup/etc/keystores/.
• Example keystores are backed up in the directory where they are found with the suffix -backup-TIMESTAMP.

Notes

Enhancement of the SAP connector add-on

The configuration of the "talend-sapjco3-connector" in version 5.5.1 allows to define additional SAP endpoints adding prefixed properties. Here is a sample for an endpoint named "PEERCONNECTIONPOOL":

jco.client.ashost = myfirsthost.example.org
jco.client.sysnr = 00
jco.client.client = 800
jco.client.user = DEVUSRA
jco.client.passwd = ***
jco.client.lang = EN
jco.destination.peak_limit = 10
jco.destination.pool_capacity = 3

endpoint.SAP_PEER_CONNECTION_POOL.jco.client.ashost = mysecondhost.example.org
endpoint.SAP_PEER_CONNECTION_POOL.jco.client.sysnr = 00
endpoint.SAP_PEER_CONNECTION_POOL.jco.client.client = 100
endpoint.SAP_PEER_CONNECTION_POOL.jco.client.user = DEVUSRB
endpoint.SAP_PEER_CONNECTION_POOL.jco.client.passwd = ***
endpoint.SAP_PEER_CONNECTION_POOL.jco.client.lang = EN
endpoint.SAP_PEER_CONNECTION_POOL.jco.destination.peak_limit = 10
endpoint.SAP_PEER_CONNECTION_POOL.jco.destination.pool_capacity = 3

Default AlgorithmSuite from Basic128Sha256 to Basic256Sha256 (TPRUN-2631)

All AlgorithmSuites of policies with SAML, are updated from Basic128Sha256 to Basic256Sha256 for these features:

• talend-job-controller
• tesb-locator-soap-service
• tesb-sam-service-soap

Configuration can be checked on these files, having value set to SAML:

If services are configured to use SAML:

• you need to ensure external clients (executing out of container) use an updated policy when reaching these endpoints
• you need to manually redeploy artifacts generated from Studio for models exposing/consuming endpoints using Service Locator or Service Activity Monitoring

Default Algorithm for password encryption/decryption (TPRUN-2601)

Algorithm encryption for all ENC(xxx) passwords is upgraded by default to PBEWITHSHA256AND256BITAES-CBC-BC. All passwords declared as ENC(xxx) in configuration files or Talend Administration Center must be regenerated through these commands in Runtime console (please ensure environment variable TESB_ENV_PASSWORD is set):

karaf@trun()> feature:install tesb-encryptor-command
karaf@trun()> tesb:encrypt-text {textToEncrypt}

Algorithm can be configured by setting environment variable TESB_ENV_ALGORITHM.
If old ENC(xxx) values are still needed, update the algorithm to previous one by setting environment variable TESB_ENV_ALGORITHM to PBEWITHSHA256AND128BITAES-CBC-BC and restart Runtime.