Enabling and configuring log rotation on the engine - Cloud

Talend Remote Engine Gen2 Quick Start Guide
Version
Cloud
Language
English
Product
Talend Cloud
Module
Talend Management Console
Talend Pipeline Designer
Content
Deployment > Deploying > Executing Pipelines
Installation and Upgrade

Logs related to the pipelines that are executed on the Remote Engine Gen2 are written inside the Livy container /tmp/livy/jobs where the log rotation is disabled by default.

Parameters Values Description
LIVY_JOB_LOG_PURGE_ENABLE true or false false (default value): log rotation is disabled.

true : enables log rotation
LIVY_JOB_LOG_RETAIN_DAYS Integer, the default value is 30 When log rotation is enabled, logs older than XX (in days) are removed.
LIVY_JOB_LOG_RETAIN_DISK Integer, the default value is 1048576 When log rotation is enabled and the total disk space used by the logs folder is greater than XX (in Bytes) all the existing logs are removed.

Procedure

  1. Go to the Remote Engine Gen2 installation directory.
  2. Edit the following file used to start the engine:
    default/docker-compose.yml if you are using the engine in the AWS USA, AWS Europe, AWS Asia-Pacific or Azure regions.

    eap/docker-compose.yml if you are using the engine as part of the Early Adopter Program.

  3. Find the service named livy and add the appropriate parameters and values.

    Example

    To enable log rotation and remove logs older than 3 months with a clean trigger of 10 GB:
    livy:
	...
	environment:
		...
		LIVY_JOB_LOG_PURGE_ENABLE: "true"
		LIVY_JOB_LOG_RETAIN_DAYS: 90
		LIVY_JOB_LOG_RETAIN_DISK: 10485760
  4. Restart Livy by executing this command: 
    docker-compose up -d livy