Enabling and configuring log rotation on the engine

Logs related to the pipelines that are executed on the Remote Engine Gen2 are written inside the Livy container /tmp/livy/jobs where the log rotation is disabled by default.

The following table presents the log rotation parameters
Parameters	Values	Description
LIVY_JOB_LOG_PURGE_ENABLE	true or false	false (default value): log rotation is disabled. true : enables log rotation
LIVY_JOB_LOG_RETAIN_DAYS	Integer, the default value is `30`	When log rotation is enabled, logs older than XX (in days) are removed.
LIVY_JOB_LOG_RETAIN_DISK	Integer, the default value is `1048576`	When log rotation is enabled and the total disk space used by the logs folder is greater than XX (in Bytes) all the existing logs are removed.

Procedure

Go to the Remote Engine Gen2 installation directory.
Edit the following file used to start the engine:
default/docker-compose.yml if you are using the engine in the AWS USA, AWS Europe, AWS Asia-Pacific or Azure regions.
eap/docker-compose.yml if you are using the engine as part of the Early Adopter Program.
Find the service named livy and add the appropriate parameters and values.
Example
To enable log rotation and remove logs older than 3 months with a clean trigger of 10 GB:
```
livy:
	...
	environment:
		...
		LIVY_JOB_LOG_PURGE_ENABLE: "true"
		LIVY_JOB_LOG_RETAIN_DAYS: 90
		LIVY_JOB_LOG_RETAIN_DISK: 10485760
```
Restart Livy by executing this command:
```
docker-compose up -d livy
```

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!

Leave your feedback here