What is currently being executed on my engine?
You would like to see which pipelines/preparations run on your Remote Engine Gen2.
Troubleshooting 1: Map the Livy port 80 to an external port to access the Livy web UI
Procedure
Cause
Each pipeline/preparation being executed is assigned a dynamic network port starting from 4040. Depending on the number of concurrent executions, a wide-enough range of ports starting from 4040 must be exposed in the livy section of the docker-compose.yml file to enable the Spark Web UI to be accessible from the running host.
Troubleshooting 2: Expose the Spark web UI ports to see the execution plans of the Spark pipelines executed on the engine
Procedure
Did this page help you?
If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!