By default, the logs of all your
executions are persisted in Apache Livy but you have the possibility to disable or
re-enable them manually in order to avoid the overhead of logging.
This configuration only applies for users who installed Remote Engine Gen2 on premises or on their
private VPC (AWS or Azure).
Procedure
-
Go to the Remote Engine Gen2 installation directory.
-
Open the following file to edit it:
<engine_directory>/default/.env if you are using the
engine in the AWS USA, AWS Europe, AWS Asia-Pacific or Azure regions.
<engine_directory>/eap/.env if you are using the
engine as part of the Early Adopter Program.
-
Edit the following line to disable or enable logs:
-
Save the file.
-
In the docker-compose.yml file, configure the Livy volume to
be used. This file is found in the default or
eap folder depending on the data center you are using.
Example
Add the following section to specify the volume:
livy:
image: ${LIVY_IMG}
networks:
default:
aliases:
- livy
environment:
...
volumes:
- <HOST_FOLDER_PATH>:/tmp/livy-jobs-logs:rw
Then change the value
of
<HOST_FOLDER_PATH> to
/tmp/livy-jobs-logs,
for example.
This configuration creates the log host folder with the read and write
permissions and mounts it in the Livy Docker container.
-
Restart your Remote Engine Gen2 to take your changes into account.