Exporting logs to an external folder - Cloud

Talend Remote Engine User Guide for Linux

Version
Cloud
Language
English
Operating system
Linux
Product
Talend Cloud
Module
Talend Remote Engine
Content
Design and Development
Installation and Upgrade
Last publication date
2024-02-23

You can export the engine logs to a dedicated folder for a third-party tool to handle the log files.

You need to maintain and clean up this folder by yourself.

Procedure

  1. Open the <RemoteEngineInstallationDirectory>/etc/org.ops4j.pax.logging.cfg file and add the following lines:

    Example

    # Activate the External folder appender.
    # Be sure to configure/change the MY_EXTERNAL_FOLDER to have logs at the place you want.
    log4j2.rootLogger.appenderRef.ExternalFolder.ref = ExternalFolder
    # External folder appender
    log4j2.appender.external.type = Routing
    log4j2.appender.external.name = ExternalFolder
    log4j2.appender.external.rewritePolicy.type = TalendKarafRewritePolicy
    log4j2.appender.external.routes.type = Routes
    log4j2.appender.external.routes.pattern = $\\{ctx:flowExecutionId\\}
    log4j2.appender.external.routes.route1.type = Route
    log4j2.appender.external.routes.route1.rolling.type = RollingFile
    log4j2.appender.external.routes.route1.rolling.name = flowexecution-$\\{ctx:flowExecutionId\\}
    log4j2.appender.external.routes.route1.rolling.fileName = MY_EXTERNAL_FOLDER/$\\{ctx:flowID\\}/$\\{ctx:flowExecutionId\\}.log
    log4j2.appender.external.routes.route1.rolling.filePattern = MY_EXTERNAL_FOLDER/$\\{ctx:flowID\\}/$\\{ctx:flowExecutionId\\}.log.%i
    # Use the 3 lines below for json pattern
    log4j2.appender.external.routes.route1.rolling.layout.type = JsonTemplateLayout
    log4j2.appender.external.routes.route1.rolling.layout.eventTemplateUri=${karaf.base.uri}/etc/jsonLogMinTemplate.json
    log4j2.appender.external.routes.route1.rolling.layout.stackTraceElementTemplateUri=${karaf.base.uri}/etc/StackTraceElementLayout.json
    # Uncomment two lines below for text pattern 
    #log4j2.appender.external.routes.route1.rolling.layout.type = PatternLayout
    #log4j2.appender.external.routes.route1.rolling.layout.pattern = ${log4j2.pattern}
    log4j2.appender.external.routes.route1.rolling.policy.type = SizeBasedTriggeringPolicy
    log4j2.appender.external.routes.route1.rolling.policy.size = 100MB
    log4j2.appender.external.routes.route1.rolling.strategy.type = DefaultRolloverStrategy
    log4j2.appender.external.routes.route1.rolling.strategy.max = 1
    # By having this set to ${ctx:flowExecutionId} it will match when flowExecutionId is not set in the context
    log4j2.appender.external.routes.route2.type = Route
    log4j2.appender.external.routes.route2.key = $\\{ctx:flowExecutionId\\}
    log4j2.appender.external.routes.route2.null.type = CountingNoOp
    log4j2.appender.external.routes.route2.null.name = CountingNoOp
    This code is only an example. When you copy it to use it in your system, do not forget to change MY_EXTERNAL_FOLDER to the actual directory in which you want to store the logs, for example, /home/usr/my-re-log on Linux or D:/remote-engine/my-log-folder on Windows.

    The $\\{ctx:flowID\\}/$\\{ctx:flowExecutionId\\} part represents the variables to be used to name the log files. If you do not need a dynamically generated log file name, replace them with plain text.

    All these changes make the log4j2.appender.external.routes.route1.rolling.fileName and the log4j2.appender.external.routes.route1.rolling.filePattern lines look like:
    • On Windows:
      log4j2.appender.external.routes.route1.rolling.fileName = C:/my-personal-folder/my-log-foler/my-log.log
      log4j2.appender.external.routes.route1.rolling.filePattern = C:/my-personal-folder/my-log-foler/my-log.log.%i
    • On Linux:
      log4j2.appender.external.routes.route1.rolling.fileName = /home/usr/my-re-log/my-log.log
      log4j2.appender.external.routes.route1.rolling.filePattern = /home/usr/my-re-log/my-log.log.%i
  2. Save the file.