You are having issues executing pipelines on the Remote Engine Gen2 and would like to display the execution logs that are stored in Livy.
Run the following command to monitor the execution and display the
corresponding Livy logs:
docker-compose logs -f --tail 50 livy
This command displays the last 50 lines of Livy logs.
If you want to display and print all the logs of your Remote Engine Gen2, run the following command:
docker-compose logs --tail 100000 > my-engine-all-logs.txt
This command would transfer the last 10000 log rows in a .txt file.