Define a rotation strategy for the logs in Talend Identity and Access Management - 8.0

Version
8.0
Language
English (United States)
Product
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
Module
Talend Identity and Access Management
Content
Administration and Monitoring > Monitoring logs

How to define a rotation strategy for logs in Talend Identity and Access Management

Logs produced by Talend Identity and Access Management could grow huge in size and amount. You may define a log rotation strategy to automatically archive or delete useless logs.
Different types of logs are stored in the $IAM_HOME/apache-tomcat/logs folder.
  • catalina.out: this is the global log, recording global events on the Tomcat server.
  • Tomcat server logs:
    • catalina.<date>.log
    • localhost.<date>.log
    • manager.<date>.log
    • host-manager.<date>.log
    According to the configuration file of these logs, $IAM_HOME/apache-tomcat/conf/logging.properties, only the logs of the last 90 days are kept.
  • Access logs: localhost_access_log.txt
  • Spring Boot applications:
    • idp.log
    • oidc.log
    • scim.log
    • sts.log
    • audit.log
    • sts-tac.log
    • audit-tac.log
  • Syncope:
    • core-connid.log
    • core.log
    • core-persistence.log
    • core-rest.log
    • console.log
    • enduser.log

The catalina.out log, the Tomcat server logs and the access logs are all standard Tomcat logs, independent of the Talend products. If you need to define a rotation strategy for these logs, see the third-party documentation about these logs, for example, logrotate for the catalina.out log or the access logs.

The following sections explain how to define a log rotation strategy for the Spring Boot logs and the Syncope logs.

Spring Boot log rotation

Create logback.xml files to configure how the Spring Boot logs are generated.

For information about the Spring Boot logging system, see https://www.baeldung.com/spring-boot-logging

Procedure

  1. In $IAM_HOME/apache-tomcat/conf/iam.properties, add a new line to indicate the path to the configuration files of the Spring Boot logs.

    Example

    logging.config=classpath:logback.xml
  2. Create a logback.xml file for each of the related Talend applications:

    Example

    vim $IAM_HOME/apache-tomcat/webapps/(idp|oidc|scim|sts|sts-tac)/WEB-INF/classes/logback.xml
    Note: Note that the logback.xml files for scim.log and sts-tac.log already exist.
    The content of each logback.xml file should look like:
    <configuration>
             <property name="LOG_PATTERN"
                       value="%d{yyyy-MM-dd HH:mm:ss.SSS} %highlight(-%5p) [%t] %-40.40logger{39} : %m%n" 
             />
             <appender name="FILE"
                       class="ch.qos.logback.core.rolling.RollingFileAppender">
                   <file>${CATALINA_BASE:-.}/logs/<app>.log</file>
                   <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                      <fileNamePattern>${CATALINA_BASE:-.}/logs/<app>.%d{yyyy-MM-dd}.log</fileNamePattern>
                      <maxHistory>10</maxHistory>
                      <totalSizeCap>100 mb</totalSizeCap>
                   </rollingPolicy>
                   <encoder>
                      <pattern>${LOG_PATTERN}</pattern>
                   </encoder>
             </appender>
             <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
                   <encoder>
                      <pattern>${LOG_PATTERN}</pattern>
                   </encoder>
             </appender>
             <root level="INFO">
                   <appender-ref ref="FILE" />
                   <appender-ref ref="STDOUT" />
             </root>
    </configuration>
    For sts.log and sts-tac.log, you can add the audit related appender to their logback.xml files. Then the content should look like:
    <configuration>
             <property name="LOG_PATTERN"
                       value="%d{yyyy-MM-dd HH:mm:ss.SSS} -%highlight(%5p) [%t] %-40.40logger{39} : %m%n" />
    <!-- %d [%t] %-5p %c %x - %m%n -->
             <appender name="FILE"
                       class="ch.qos.logback.core.rolling.RollingFileAppender">
                   <file>${CATALINA_BASE:-.}/logs/<app>.log</file>
                   <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                      <fileNamePattern>${CATALINA_BASE:-.}/logs/<app>.%d{yyyy-MM-dd}.log</fileNamePattern>
                      <maxHistory>10</maxHistory>
                      <totalSizeCap>100 mb</totalSizeCap>
                   </rollingPolicy>
                   <encoder>
                      <pattern>${LOG_PATTERN}</pattern>
                   </encoder>
             </appender>
             <appender name="AUDIT"
                       class="ch.qos.logback.core.rolling.RollingFileAppender">
                   <file>${CATALINA_BASE:-.}/logs/<app>/audit-tac.log</file>
                   <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                       <fileNamePattern>${CATALINA_BASE:-.}/logs/<app>/audit-tac.%d{yyyy-MMdd}.log</fileNamePattern>
                       <maxHistory>10</maxHistory>
                       <totalSizeCap>20 mb</totalSizeCap>
                   </rollingPolicy>
                   <encoder>
                       <pattern>${LOG_PATTERN}</pattern>
                   </encoder>
             </appender>
             <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
                   <encoder>
                       <pattern>${LOG_PATTERN}</pattern>
                   </encoder>
             </appender>
             <logger name="org.talend.identity.sts.event.map.MapEventLogger" level="DEBUG" additivity="false">
                   <appender-ref ref="AUDIT" />
             </logger>
             <root level="INFO">
                   <appender-ref ref="FILE" />
                   <appender-ref ref="STDOUT" />
             </root>
    </configuration>
    In the example, the log level is defined in each logback.xml file, but you can change the log level globally by adding the following line to the iam.properties file:
    logging.level.root=DEBUG
    The value of the parameter could be
    • TRACE
    • DEBUG
    • INFO
    • WARN
    • ERROR

Syncope log rotation

Define the rotation policy in $IAM_HOME/apache-tomcat/webapps/<app>/WEB-INF/classes/log4j2.xml.

For information about this Syncope log rotation appender, see the Apache documentation: https://logging.apache.org/log4j/2.x/manual/appenders.html

Procedure

  1. Open $IAM_HOME/apache-tomcat/webapps/<app>/WEB-INF/classes/log4j2.xml and find the appender used to define the log rotation strategy. This appender could look like:

    Example

    <appenders>
        <RollingRandomAccessFile name="main" fileName="${sys:catalina.base}/logs/enduser.log"
                                 filePattern="${sys:catalina.base}/logs/enduser-%d{yyyy-MM-dd}.log.gz"
                                 immediateFlush="false" append="true">
             <PatternLayout>
                <pattern>%d{HH:mm:ss.SSS} %-5level %logger - %msg%n</pattern>
             </PatternLayout>
             <Policies>
                <TimeBasedTriggeringPolicy/>
                <SizeBasedTriggeringPolicy size="250 MB"/>
             </Policies>
        </RollingRandomAccessFile>
    </appenders>
    This default configuration means rotation takes place every day and the total file size is limited to 250 MB.
  2. If you need to add an automatic cleanup policy, add a roller strategy to the RollingRandomAccessFile appender. For example:

    Example

    <appenders>
        <RollingRandomAccessFile name="main" fileName="${sys:catalina.base}/logs/enduser.log"
                                 filePattern="${sys:catalina.base}/logs/enduser-%d{yyyy-MM-dd}.log.gz"
                                 immediateFlush="false" append="true">
             <PatternLayout>
                <pattern>%d{HH:mm:ss.SSS} %-5level %logger - %msg%n</pattern>
             </PatternLayout>
             <Policies>
                <TimeBasedTriggeringPolicy/>
                <SizeBasedTriggeringPolicy size="250 MB"/>
             </Policies>
             <DefaultRolloverStrategy max="100">
               <Delete basePath="${baseDir}" maxDepth="2">
                  <IfFileName glob="*/app-*.log.gz">
                     <IfLastModified age="30d">
                       <IfAny>
                        <IfAccumulatedFileSize exceeds="100 GB" />
                        <IfAccumulatedFileCount exceeds="10" />
                       </IfAny>
                     </IfLastModified>
                  </IfFileName>
                </Delete>
             </DefaultRolloverStrategy>
        </RollingRandomAccessFile>
    </appenders>
    
    Note: The Syncope application uses Log4J 2 to manage its log levels. Those levels could be
    • OFF
    • FATAL
    • ERROR
    • WARN
    • INFO
    • DEBUG
    • TRACE
    • ALL
    For further information about these levels, see the documentation from Apache Log4J: Apache Log4J 2 custom logging levels.