What is currently being executed on my engine?

What is currently being executed on my engine? - Cloud

Talend Remote Engine Gen2 Quick Start Guide

Version

Cloud

Language

English

Product

Talend Cloud

Module

Talend Management Console

Talend Pipeline Designer

Content

Deployment > Deploying > Executing Pipelines

Installation and Upgrade

Last publication date

2024-01-24

You would like to see which pipelines/preparations run on your Remote Engine Gen2.

Troubleshooting 1: Map the Livy port 80 to an external port to access the Livy web UI

Procedure

Run the following command to stop the Remote Engine Gen2:
remote-engine.bat stop if you are on a Windows system

./remote-engine.sh stop if you are on a Unix system
Open the following file located in the engine directory:
default/docker-compose.yml if you are using the engine in the AWS USA, AWS Europe, AWS Asia-Pacific or Azure regions.
eap/docker-compose.yml if you are using the engine as part of the Early Adopter Program.
Add the following in the livy section (8001 is used as a port example, you could use any available port):
```
ports:
	- 8001:80/tcp
```
Save your changes and restart the engine to take these changes into account.
Access the Livy User Interface with your browser: http://localhost:8001/ui

You should see the pipelines/preparations (called batch sessions in Livy) that are currently being executed as well as previous executions with their end statuses (success, fail, killed) and their logs.
Run the following command to see the Livy logs related to the pipelines/preparations executed in real time:
```
docker logs -f remote-engine_livy_1
```

Cause

Each pipeline/preparation being executed is assigned a dynamic network port starting from 4040. Depending on the number of concurrent executions, a wide-enough range of ports starting from 4040 must be exposed in the livy section of the docker-compose.yml file to enable the Spark Web UI to be accessible from the running host.

Troubleshooting 2: Expose the Spark web UI ports to see the execution plans of the Spark pipelines executed on the engine

Procedure

Run the following command to stop the Remote Engine Gen2:
remote-engine.bat stop if you are on a Windows system

./remote-engine.sh stop if you are on a Unix system
Open the docker-compose.yml and add the following in the livy section (example to see the Spark UI of three concurrent pipelines/preparations that are being executed):
```
ports:
	- 4040:4040/tcp
	- 4041:4041/tcp
	- 4042:4042/tcp
```