Setting up the Logging parameters - 8.0

Talend Administration Center User Guide
Version
8.0
Language
English
Product
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
Module
Talend Administration Center
Content
Administration and Monitoring
Last publication date
2024-04-15

About this task

In Talend Administration Center, there are three types of collected logs:

  • the technical logs, that gather technical events like the status of connections and servers, and so on.
  • the business logs, that gather all users' actions, like the creation, connection, deletion, and edition of a user, a Job and so on.
  • the key rotation logs, that gather the execution of the master key rotation.

Those logs are collected by FileBeat and displayed on the Logging page thanks to the Kibana Web application whose URL needs to be configured in the Monitoring node of the Configuration page as explained in Setting up the link to the Monitoring modules. For more information about the extended logging module, see Displaying log events.

Note: Kibana is not supported on IE11.

You can modify the log file paths and threshold. To do so, complete the following:

Procedure

  1. On the Menu tree view, click Configuration.
  2. Click the Logs group to display its parameters.
    Note: If the path to the three types of logs is not configured, all logs are stored by default in the Catalina logs at the following path: <tomcat_path>/webapps/org.talend.administrator/WEB-INF/class/log4j2.xml.
    The parameters are editable in Talend Administration Center user interface according to your use:

    Parameter

    Value

    Technical file appender

    Type in the path to the technical log file of Talend Administration Center.

    The specified folder must exist. For example, if you enter c:/logs/technical.log, the technical.log file will be created in the existing c:/logs folder. The log file contains technical information logged by classes on the Talend Administration Center runtime, for example task execution start information and so on. You can safely remove the log file.

    Note: If no path has been set or if the path is invalid, the technical logs will be appended in Tomcat logs.

    Technical log threshold

    Select the level of logs you want to append from TRACE, DEBUG, INFO, WARN, ERROR and OFF from the drop-down list.
    Set technical log limit by Select from Time and File size and count for setting technical log rotation criteria.
    • Time: Set log rotation by days. When this option is selected, each file size will be 1MB and unlimited number of files will be created.
    • File size and count: Set log rotation by file size and count.
    Max. size per log file(MB) This option is available when File size and count is selected from the Set technical log limit by list. Set the maximum size of each technical log file. Only integer values between 1~N (within 6 characters) are allowed.
    Rotate logs after file number This option is available when File size and count is selected from the Set technical log limit by list. Set the number of technical log files for log rotation. Only integer values between 1~N (within 6 characters) are allowed.
    Rotate logs older than(in days) This option is available when Time is selected from the Set technical log limit by list. Set the maximum life time of technical log files in days. Only integer values between 1~N (within 6 characters) are allowed.
    Business log file path

    Type in the path to the business log file of Talend Administration Center.

    The specified folder must exist. For example, if you enter c:/logs/business.log, the business.log file will be created in the existing c:/logs folder. The log file contains business information logged by classes on the Talend Administration Center runtime, for example task or user create information and so on. You can safely remove the log file.

    Note: If no path has been set or if the path is invalid, the business logs will not be recorded.
    Set business log limit by Select from Time and File size and count for setting business log rotation criteria.
    • Time: Set log rotation by days. When this option is selected, each file size will be 1MB and unlimited number of files will be created.
    • File size and count: Set log rotation by file size and count.
    Max size per log file(MB) This option is available when File size and count is selected from the Set business log limit by list. Set the maximum size of each business log file. Only integer values between 1~N (within 6 characters) are allowed.
    Rotate logs after file number This option is available when File size and count is selected from the Set business log limit by list. Set the number of business log files for log rotation. Only integer values between 1~N (within 6 characters) are allowed.
    Rotate logs older than(in days) This option is available when Time is selected from the Set technical log limit by list. Set the maximum life time of technical log files in days. Only integer values between 1~N (within 6 characters) are allowed.
    Key Rotation log file path

    Type in the path to the master key rotation log file of Talend Administration Center.

    The specified folder must exist. For example, if you enter c:/logs/keyRotation.log, the keyRotation.log file will be created in the existing c:/logs folder. You can safely remove the log file.

    Note: If no path has been set or if the path is invalid, the master key rotation logs will not be recorded.
    Set Key Rotation log limit by Select from Time and File size and count for setting master key rotation log rotation criteria.
    • Time: Set log rotation by days. When this option is selected, each file size will be 1MB and unlimited number of files will be created.
    • File size and count: Set log rotation by file size and count.
    Max size per log file (MB) This option is available when File size and count is selected from the Set Key Rotation log limit by list. Set the maximum size of each master key rotation log file. Only integer values between 1~N (within 6 characters) are allowed.
    Rotate logs after file number This option is available when File size and count is selected from the Set Key Rotation log limit by list. Set the number of master key rotation log files for log rotation. Only integer values between 1~N (within 6 characters) are allowed.
    Rotate logs older than (in days) This option is available when Time is selected from the Set Key Rotation log limit by list. Set the maximum life time of master key rotation log files in days. Only integer values between 1~N (within 6 characters) are allowed.
    Other parameters can also be edited in the configuration.properties file located in <Tomcat_home>\webapps\org.talend.administrator\WEB-INF\:
    log.buffered and log.buffered.size By default, logs are buffered (log.buffered value is set to true), which means some log lines may be incomplete.

    If you want to get the full list of business logs, technical logs and rotation logs immediately after the events have occurred, you need to:

    1. set log.buffered to false,
    2. define log.buffered.size value (default value is 16).
    Note: For Catalina logs, go to path <tomcat_path>/webapps/org.talend.administrator/WEB-INF/class/log4j2.xml and edit the file as in the following sample:
    <Console name="STDOUT" target="SYSTEM_OUT">
    <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1} - %m%n"/>
    <BufferedIo bufferSize="1024"/>
</Console>
    The format of the technical logs and the business logs follow the log4j Class PatternLayout. For more information see https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html. The conversion pattern %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1} %x - %m%n is used:

    Conversion Character

    Effect

    %d{yyyy-MM-dd HH:mm:ss}

    Used to output the date of the logging event. The data format is yyyy-MM-dd HH:mm:ss, for example, 2018-02-07 18:58:15.

    %-5p

    Used to output the priority of the logging event. %-5p means the priority of the logging event should be left justified to a width of five characters, for example, INFO, WARN, DEBUG.
    %c{1}

    Used to output the Class name.

    %x

    Used to output the NDC (nested diagnostic context) associated with the thread that generated the logging event.

    %m

    Used to output the application supplied message associated with the logging event.

    %n

    Outputs the platform dependent line separator character or characters.
    A sample log 2018-03-14 09:36:17 DEBUG TokenRequiredPolicy - Checking token shows the following information:
    • datatime,
    • log level,
    • java class name,
    • the log message, which should include the activity detail.
    In order to display the JDBC query execution times (and thus, the time spent to connect to the database), it is possible to enable hibernate.generate_statistics logs:
    1. Go to <Tomcat_home>\webapps\org.talend.administrator\WEB-INF\classes and open configuration.properties file. Set the following parameter: hibernate.generate_statistics=true.
    2. Go to <Tomcat_home>\webapps\org.talend.administrator\WEB-INF\classes and open log4j.xml file. Set the debug level for the following folders: "org.hibernate.SQL", "org.hibernate.HQL", "org.hibernate.stat" and "org.hibernate.engine.internal.StatisticalLoggingSessionEventListener".
    3. Restart Talend Administration Center.