Upgrade your Remote Engine without the need to undeploy or redeploy any data service Job or Route tasks currently running on the engine.
Microservice tasks are automatically halted with the first Remote Engine shutdown and resumed upon the startup of the new Remote Engine version.
Before you begin
- The Remote Engine to upgrade must be v2.11.8 and onwards. If you are using v2.11.7 or an earlier version, upgrade them to the latest version before starting this procedure as explained in Upgrading Talend Remote Engine.
- Generate a service account token (SAT) to use a service account to perform
this operation. For more information on how to generate this token, see Generating a service account
Using a service account is a widely accepted security practice for automation in a production environment. While using a personal access token (PAT) is technically possible, it is recommended to utilize a service account for enhanced security.
For more information on how to generate a personal access token, see Generating a Personal Access Token.
- Gather the IDs of the engines to be upgraded, using either the related API
endpoint or the Talend Cloud Management Console web page:
- API endpoint: Get available Remote Engines.
In the response, the engine IDs are found in the
idfield. Copy the IDs of the engines to be upgraded and keep them in a place of your choice for later use.
- Talend Cloud Management Console web page: a Remote Engine ID is found in the
Overview tab of the Engine
details blade. If the engine in question is in a cluster, this URL reads like
In this case, the ID is
- API endpoint: Get available Remote Engines. In the response, the engine IDs are found in the
Use the specific API endpoint to unpair the Remote Engine to be upgraded from
the designated engine item on Talend Cloud Management Console by running the following command:
Do not use the Talend Cloud Management Console web UI to unpair the engine. Use the following endpoint exclusively.
curl -X DELETE "$RE_APFIRST_ENDPOINT/processing/runtimes/remote-engines/$RE_ENGINE_ID/pairing?keepOsgiDeployments=true&keepMicroserviceDeployments=true" -H "Authorization: Bearer $RE_TOKEN"In this command:
$RE_APFIRST_ENDPOINTrepresents the URL of Talend Cloud Management Console, for example,
For the API endpoint URL of your region, see Endpoint regions.
$RE_ENGINE_IDis for the ID of the engine to be unpaired.
$RE_TOKENis the service account token, which is recommended, or the personal access token of the account to be used.
When the command is executed, a new preauthorized key is automatically generated. Keep this key in a place of your choice, because you need to use it to pair the new Remote Engine in the following steps.
Wait for at least 2 minutes to ensure that the engine is correctly
ExampleThe following message from the engine logs is an example that indicates the unpairing has been correctly completed.
On the host machine of the unpaired engine, stop the engine using the following
If this engine has been installed as a system service, follow your OS procedure to shut down the service.
Install and configure the new Remote Engine version.
- If you use the installer to install this new engine, do not use the actual preauthorized key yet when you are prompted to provide a key; provide a fake authorized key instead.
- If you have installed the new engine as a system service, stop the service before proceeding to the following steps.
Copy data from the old engine to the new engine.
This copy helps ensure the continuous running of your data service Job tasks or Route tasks. For this reason, copy each type of data to their designated folders as expected by Remote Engine.
Copy-paste the dsrunner folder from
This folder contains the task information.
Copy-paste the cls folder from
This folder contains the task logs.
On the new Remote Engine, update the path to the new engine to be used
in every .metadata file that is stored in each of
the task folders in
Then replace it with the path to the new Remote Engine:
If you are using Windows, these paths look like
Still on the new Remote Engine, update the path to the log files for
This information is found in every <TASKID>-<EXECUTION_ID>-state.dat file that is stored in New_RemoteEngine_installation/data/cls.
ExampleFor example, open the New_RemoteEngine_installation/data/cls/640ef0c6a45e743a0b2580ca-5f69d7c4-e3d8-4d41-b248-b94a20d7cbf0-state.dat file and find the following line, which is the path you need to update:
Then replace it with the new path on the new Remote Engine:
On Windows, this path look like
- Copy-paste the dsrunner folder from Old_RemoteEngine_installation/data/ to New_RemoteEngine_installation/data/.
Pair the new Remote Engine to the same Talend Cloud Management Console engine item that you previously unpaired from the old Remote Engine.
file, specify the endpoint for pairing of Talend Cloud Management Console in the following line:
For the exact endpoint URL to be used, see the Remote Engine pairing entry of your region in Regions and URLs.
If you have installed the new engine using the installer, you can skip this step because the endpoint should have been automatically specified.
replace the fake preauthorized key with the actual one generated by the
unpairing call at the beginning of the procedure.
If you have lost this preauthorized key, find it back in the Remote Engine key field of the engine item to be paired on Talend Cloud Management Console. For further information about how to find this key, see Creating a Remote Engine.
- In the New_RemoteEngine_installation/etc/org.talend.ipaas.rt.pairing.client.cfg file, specify the endpoint for pairing of Talend Cloud Management Console in the following line:
Start the new Remote Engine.
See further information at Starting a Remote Engine.
Verify the task status on Talend Cloud Management Console by checking that data service Job tasks and Route tasks are up and running. Logs and metrics should also be available for all these tasks.