You are having issues executing pipelines on the Remote Engine Gen2 and would like to display the
execution logs that are stored in Livy.
Troubleshooting
Procedure
-
Run the following command to monitor the execution and display the
corresponding Livy logs:
docker-compose logs -f --tail 50 livy
This command displays the last 50 lines of Livy logs.
- Optional:
If you want to display and print all the logs of your Remote Engine Gen2, run the following command:
docker-compose logs --tail 100000 > my-engine-all-logs.txt
This command would transfer the last 10000 log rows in a
.txt file.