Logs related to the pipelines that are executed on the Remote Engine Gen2 are written inside the Livy container /tmp/livy/jobs where the log rotation is disabled by default.
|LIVY_JOB_LOG_RETAIN_DAYS||Integer, the default value is 30||When log rotation is enabled, logs older than XX (in days) are removed.|
|LIVY_JOB_LOG_RETAIN_DISK||Integer, the default value is 1048576||When log rotation is enabled and the total disk
space used by the logs folder is greater than
- Go to the Remote Engine Gen2 installation directory.
Edit the following file used to start the engine:
default/docker-compose.yml if you are using the engine in the AWS USA, AWS Europe, AWS Asia-Pacific or Azure regions.
eap/docker-compose.yml if you are using the engine as part of the Early Adopter Program.
Find the service named livy and add the appropriate
parameters and values.
ExampleTo enable log rotation and remove logs older than 3 months with a clean trigger of 10 GB:
livy: ... environment: ... LIVY_JOB_LOG_PURGE_ENABLE: "true" LIVY_JOB_LOG_RETAIN_DAYS: 90 LIVY_JOB_LOG_RETAIN_DISK: 10485760
Restart Livy by executing this command:
docker-compose up -d livy