Define a rotation strategy for the logs in Talend Identity and Access Management

How to define a rotation strategy for logs in Talend Identity and Access Management

Logs produced by Talend Identity and Access Management could grow huge in size and amount. You may define a log rotation strategy to automatically archive or delete useless logs.

Different types of logs are stored in the $IAM_HOME/apache-tomcat/logs folder.

catalina.out: this is the global log, recording global events on the Tomcat server.
Tomcat server logs:
- catalina.<date>.log
- localhost.<date>.log
- manager.<date>.log
- host-manager.<date>.log
According to the configuration file of these logs, $IAM_HOME/apache-tomcat/conf/logging.properties, only the logs of the last 90 days are kept.
Access logs: localhost_access_log.txt
Spring Boot applications:
- idp.log
- oidc.log
- scim.log
- sts.log
- audit.log
- sts-tac.log
- audit-tac.log
Syncope:
- core-connid.log
- core.log
- core-persistence.log
- core-rest.log
- console.log
- enduser.log

The catalina.out log, the Tomcat server logs and the access logs are all standard Tomcat logs, independent of the Talend products. If you need to define a rotation strategy for these logs, see the third-party documentation about these logs, for example, logrotate for the catalina.out log or the access logs.

The following sections explain how to define a log rotation strategy for the Spring Boot logs and the Syncope logs.

Spring Boot log rotation

Create logback.xml files to configure how the Spring Boot logs are generated.

For information about the Spring Boot logging system, see https://www.baeldung.com/spring-boot-logging

Procedure

In $IAM_HOME/apache-tomcat/conf/iam.properties, add a new line to indicate the path to the configuration files of the Spring Boot logs.
Example
```
logging.config=classpath:logback.xml
```

Create a logback.xml file for each of the related Talend applications:

Example

vim $IAM_HOME/apache-tomcat/webapps/(idp|oidc|scim|sts|sts-tac)/WEB-INF/classes/logback.xml

Note: Note that the logback.xml files for scim.log and sts-tac.log already exist.

The content of each logback.xml file should look like:

<configuration>
         <property name="LOG_PATTERN"
                   value="%d{yyyy-MM-dd HH:mm:ss.SSS} %highlight(-%5p) [%t] %-40.40logger{39} : %m%n" 
         />
         <appender name="FILE"
                   class="ch.qos.logback.core.rolling.RollingFileAppender">
               <file>${CATALINA_BASE:-.}/logs/<app>.log</file>
               <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                  <fileNamePattern>${CATALINA_BASE:-.}/logs/<app>.%d{yyyy-MM-dd}.log</fileNamePattern>
                  <maxHistory>10</maxHistory>
                  <totalSizeCap>100 mb</totalSizeCap>
               </rollingPolicy>
               <encoder>
                  <pattern>${LOG_PATTERN}</pattern>
               </encoder>
         </appender>
         <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
               <encoder>
                  <pattern>${LOG_PATTERN}</pattern>
               </encoder>
         </appender>
         <root level="INFO">
               <appender-ref ref="FILE" />
               <appender-ref ref="STDOUT" />
         </root>
</configuration>

For sts.log and sts-tac.log, you can add the audit related appender to their logback.xml files. Then the content should look like:

<configuration>
         <property name="LOG_PATTERN"
                   value="%d{yyyy-MM-dd HH:mm:ss.SSS} -%highlight(%5p) [%t] %-40.40logger{39} : %m%n" />
<!-- %d [%t] %-5p %c %x - %m%n -->
         <appender name="FILE"
                   class="ch.qos.logback.core.rolling.RollingFileAppender">
               <file>${CATALINA_BASE:-.}/logs/<app>.log</file>
               <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                  <fileNamePattern>${CATALINA_BASE:-.}/logs/<app>.%d{yyyy-MM-dd}.log</fileNamePattern>
                  <maxHistory>10</maxHistory>
                  <totalSizeCap>100 mb</totalSizeCap>
               </rollingPolicy>
               <encoder>
                  <pattern>${LOG_PATTERN}</pattern>
               </encoder>
         </appender>
         <appender name="AUDIT"
                   class="ch.qos.logback.core.rolling.RollingFileAppender">
               <file>${CATALINA_BASE:-.}/logs/<app>/audit-tac.log</file>
               <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
                   <fileNamePattern>${CATALINA_BASE:-.}/logs/<app>/audit-tac.%d{yyyy-MMdd}.log</fileNamePattern>
                   <maxHistory>10</maxHistory>
                   <totalSizeCap>20 mb</totalSizeCap>
               </rollingPolicy>
               <encoder>
                   <pattern>${LOG_PATTERN}</pattern>
               </encoder>
         </appender>
         <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
               <encoder>
                   <pattern>${LOG_PATTERN}</pattern>
               </encoder>
         </appender>
         <logger name="org.talend.identity.sts.event.map.MapEventLogger" level="DEBUG" additivity="false">
               <appender-ref ref="AUDIT" />
         </logger>
         <root level="INFO">
               <appender-ref ref="FILE" />
               <appender-ref ref="STDOUT" />
         </root>
</configuration>

In the example, the log level is defined in each logback.xml file, but you can change the log level globally by adding the following line to the iam.properties file:

logging.level.root=DEBUG

The value of the parameter could be

TRACE
DEBUG
INFO
WARN
ERROR

Syncope log rotation

Define the rotation policy in $IAM_HOME/apache-tomcat/webapps/<app>/WEB-INF/classes/log4j2.xml.

For information about this Syncope log rotation appender, see the Apache documentation: https://logging.apache.org/log4j/2.x/manual/appenders.html

Procedure

Open $IAM_HOME/apache-tomcat/webapps/<app>/WEB-INF/classes/log4j2.xml and find the appender used to define the log rotation strategy. This appender could look like:

Example

<appenders>
    <RollingRandomAccessFile name="main" fileName="${sys:catalina.base}/logs/enduser.log"
                             filePattern="${sys:catalina.base}/logs/enduser-%d{yyyy-MM-dd}.log.gz"
                             immediateFlush="false" append="true">
         <PatternLayout>
            <pattern>%d{HH:mm:ss.SSS} %-5level %logger - %msg%n</pattern>
         </PatternLayout>
         <Policies>
            <TimeBasedTriggeringPolicy/>
            <SizeBasedTriggeringPolicy size="250 MB"/>
         </Policies>
    </RollingRandomAccessFile>
</appenders>

This default configuration means rotation takes place every day and the total file size is limited to 250 MB.

If you need to add an automatic cleanup policy, add a roller strategy to the RollingRandomAccessFile appender. For example:

Example

<appenders>
    <RollingRandomAccessFile name="main" fileName="${sys:catalina.base}/logs/enduser.log"
                             filePattern="${sys:catalina.base}/logs/enduser-%d{yyyy-MM-dd}.log.gz"
                             immediateFlush="false" append="true">
         <PatternLayout>
            <pattern>%d{HH:mm:ss.SSS} %-5level %logger - %msg%n</pattern>
         </PatternLayout>
         <Policies>
            <TimeBasedTriggeringPolicy/>
            <SizeBasedTriggeringPolicy size="250 MB"/>
         </Policies>
         <DefaultRolloverStrategy max="100">
           <Delete basePath="${baseDir}" maxDepth="2">
              <IfFileName glob="*/app-*.log.gz">
                 <IfLastModified age="30d">
                   <IfAny>
                    <IfAccumulatedFileSize exceeds="100 GB" />
                    <IfAccumulatedFileCount exceeds="10" />
                   </IfAny>
                 </IfLastModified>
              </IfFileName>
            </Delete>
         </DefaultRolloverStrategy>
    </RollingRandomAccessFile>
</appenders>

Note: The Syncope application uses Log4J 2 to manage its log levels. Those levels could be

OFF
FATAL
ERROR
WARN
INFO
DEBUG
TRACE
ALL

For further information about these levels, see the documentation from Apache Log4J: Apache Log4J 2 custom logging levels.