Masking sensitive connection information in the task run logs in Talend Management Console - Cloud

Talend Remote Engine User Guide for Linux
Version
Cloud
Language
English
Operating system
Linux
Product
Talend Cloud
Module
Talend Remote Engine
Content
Design and Development
Installation and Upgrade
Last publication date
2024-02-23

With Remote Engine v2.12.3 onwards, if displayed in the logs in Talend Management Console, the connection information is automatically masked. This information could be any of the connection parameters you have locked in Talend Management Console, such as the host, the port number or the username.

About this task

If your engine version is between v2.10.8 and v2.12.2, this information is not automatically masked. You need to perform the following operations to configure Talend Remote Engine to mask it in those logs. Otherwise, ignore this section.

Procedure

  1. In Talend Management Console, verify that the connection parameters to be masked in logs have been locked.
    For further information about a connection in Talend Management Console and locking a connection parameter, see Creating a custom connection.
  2. In the machine where your Remote Engine is installed, browse to the <RemoteEngineInstallationDirectory>/etc folder and open the org.talend.ipaas.rt.jobserver.client.cfg file.
  3. Define the job.log.msg.sensitive.hidden property:
    job.log.msg.sensitive.hidden=true
  4. Save the file.

Results

This change is taken into account directly without the engine having to get restarted. The locked connection parameters are masked in the task run logs in Talend Management Console, for example:
Screenshot showing sensitive connection information is masked.
Troubleshooting:

Masking sensitive connection information may increase CPU and memory consumption, when a lot of information is being masked while a large number of logs are held in memory.

To mitigate this spike, fine-tune the following parameters:

  1. Browse to the <RemoteEngineInstallationDirectory>/etc folder and open the org.talend.ipaas.rt.jobserver.client.cfg file.
  2. Adjust the values as follows:
    The following table presents the parameters used to adjust the size of log rows and the number of stored log events
    Parameter Description
    job.log.msg.size.max Its default value is 10000 characters, meaning the maximum size allowed for each log row.

    By decreasing this value to limit the size of each log row to be handled, you truncate each log row to reduce the resource consumption.

    If truncating logs is an issue for you, consider adding additional resources to the engine.
    job.log.msg.buffer.size.max This parameter sets the maximum number of log events that can be temporarily stored in memory, with a default value of 10000.

    During high peaks of concurrent log events, this maximum size may be reached, resulting in the loss of log data. To prevent this loss, increase this buffer size and accordingly, allocate additional memory resources to your engine.

    Note: To alleviate the load of your memory and CPU, you can also add a thread.timeout parameter to specify the maximum time that log processing can run before timing out. The default value is 5 minutes, at the end of which log processing is stopped and context information cleaned up. By decreasing this value, you release resources more frequently, but be aware that the logs that have not been processed are lost. If this is a concern, consider allocating more resources to your engine.
    When this timeout event occurs, it is logged as a WARN event. A log entry of this event could read like:
    Force cleanup context for empty streamSet, executionId {}
  3. Each character requires 2 bytes of memory. This way, when calculating additional resources required for log processing, the calculation can be done as follows:
    • For 2000 characters on each log row and 2000 log events, 80MB of memory is required.
    • For 20000 characters on each log row and 5000 log events, 200MB of memory is required.
    • For 20000 characters on each log row and 20000 log events, 800MB of memory is required.

    After increasing this allocation in the host machine, if your engine has been deployed as a system service, customize the JVM configuration in the <RemoteEngineInstallationDirectory>/etc/talend-remote-engine-wrapper.conf file accordingly.