Upgrading Remote Engine v2.11.8 onwards for data service Job or Route tasks - Cloud

Talend Cloud Installation and Upgrade Guide
Version
Cloud
Language
English
Operating system
Windows
Product
Talend Cloud
Module
Talend Artifact Repository
Talend Data Stewardship
Talend Management Console
Talend Remote Engine
Talend SAP RFC Server
Talend Studio
Content
Installation and Upgrade
Last publication date
2023-12-07

Upgrade your Remote Engine without the need to undeploy or redeploy any data service Job or Route tasks currently running on the engine.

Microservice tasks are automatically halted with the first Remote Engine shutdown and resumed upon the startup of the new Remote Engine version.

Before you begin

  • The Remote Engine to upgrade must be v2.11.8 and onwards. If you are using v2.11.7 or an earlier version, upgrade them to the latest version before starting this procedure as explained in Upgrading Talend Remote Engine.
  • Generate a service account token (SAT) to use a service account to perform this operation. For more information on how to generate this token, see Generating a service account token.

    Using a service account is a widely accepted security practice for automation in a production environment. While using a personal access token (PAT) is technically possible, it is recommended to utilize a service account for enhanced security.

    For more information on how to generate a personal access token, see Generating a Personal Access Token.

  • Gather the IDs of the engines to be upgraded, using either the related API endpoint or the Talend Cloud Management Console web page:
    • API endpoint: Get available Remote Engines. In the response, the engine IDs are found in the id field. Copy the IDs of the engines to be upgraded and keep them in a place of your choice for later use.
    • Talend Cloud Management Console web page: a Remote Engine ID is found in the Overview tab of the Engine details blade.
      Engine details dialog box
      If the engine in question is in a cluster, this URL reads like
      https://tmc.eu.cloud.talend.com/engines/5bf0452b853ce935a13ed9e6/cluster/646c7931d5380f70c1d217fc/assigned-engine/646c790a2373b74412c26047/details/info
      In this case, the ID is 646c790a2373b74412c26047.

Procedure

  1. Use the specific API endpoint to unpair the Remote Engine to be upgraded from the designated engine item on Talend Cloud Management Console by running the following command:
    Do not use the Talend Cloud Management Console web UI to unpair the engine. Use the following endpoint exclusively.

    Example

    curl -X DELETE "$RE_APFIRST_ENDPOINT/processing/runtimes/remote-engines/$RE_ENGINE_ID/pairing?keepOsgiDeployments=true&keepMicroserviceDeployments=true" -H "Authorization: Bearer $RE_TOKEN"
    In this command:
    • $RE_APFIRST_ENDPOINT represents the URL of Talend Cloud Management Console, for example, https://api.us.cloud.talend.com/.

      For the API endpoint URL of your region, see Endpoint regions.

    • $RE_ENGINE_ID is for the ID of the engine to be unpaired.
    • $RE_TOKEN is the service account token, which is recommended, or the personal access token of the account to be used.
    You can either replace these variables with their actual values in your command, or define them in the environment of your API requests. If you use Talend API Tester to issue your API endpoint, see Creating an environment to learn about how to define variables.

    When the command is executed, a new preauthorized key is automatically generated. Keep this key in a place of your choice, because you need to use it to pair the new Remote Engine in the following steps.

  2. Wait for at least 2 minutes to ensure that the engine is correctly unpaired.

    Example

    The following message from the engine logs is an example that indicates the unpairing has been correctly completed.
    Logs that indicate the engine state is retired with the line "re_state": "RETIRED".
  3. On the host machine of the unpaired engine, stop the engine using the following command: 
    <RemoteEngine_installation>/bin/stop
    If this engine has been installed as a system service, follow your OS procedure to shut down the service.
  4. Install and configure the new Remote Engine version.
    • If you use the installer to install this new engine, do not use the actual preauthorized key yet when you are prompted to provide a key; provide a fake authorized key instead.
    • If you have installed the new engine as a system service, stop the service before proceeding to the following steps.
  5. Copy data from the old engine to the new engine.
    This copy helps ensure the continuous running of your data service Job tasks or Route tasks. For this reason, copy each type of data to their designated folders as expected by Remote Engine.
    1. Copy-paste the dsrunner folder from Old_RemoteEngine_installation/data/ to New_RemoteEngine_installation/data/.
      This folder contains the task information.
    2. Copy-paste the cls folder from Old_RemoteEngine_installation/data/ to New_RemoteEngine_installation/data/.
      This folder contains the task logs.
    3. On the new Remote Engine, update the path to the new engine to be used in every .metadata file that is stored in each of the task folders in New_RemoteEngine_installation/data/dsrunner/exec/osgi.

      Example

      Path to the osgi folder content.

      For example, in this folder New_RemoteEngine_installation/data/dsrunner/osgi/02141bff-bffe-4975-8835-7dd9ff77b62a, open the .metadata file and find the following line, which is the path to the engine to be used:
      "artifactFile":"Old_RemoteEngine_installation/data/dsrunner/exec/osgi/5f69d7c4-e3d8-4d41-b248-b94a20d7cbf0/SimpleRecipientList_OSGI_0.1.0.20231303094541.kar"
      Then replace it with the path to the new Remote Engine:
      "artifactFile":"New_RemoteEngine_installation/data/dsrunner/exec/osgi/5f69d7c4-e3d8-4d41-b248-b94a20d7cbf0/SimpleRecipientList_OSGI_0.1.0.20231303094541.kar"
      If you are using Windows, these paths look like
      RemoteEngine_installation\\data\\dsrunner\\exec\\osgi\\02141bff-bffe-4975-8835-7dd9ff77b62a\\ro_20230209_SimpleTimerRoute_801_OSGi_0.1.2.20232505013343.kar
    4. Still on the new Remote Engine, update the path to the log files for each execution.
      This information is found in every <TASKID>-<EXECUTION_ID>-state.dat file that is stored in New_RemoteEngine_installation/data/cls.

      Example

      For example, open the New_RemoteEngine_installation/data/cls/640ef0c6a45e743a0b2580ca-5f69d7c4-e3d8-4d41-b248-b94a20d7cbf0-state.dat file and find the following line, which is the path you need to update:
      "currentFilePath":"Old_RemoteEngine_installation/data/cls/640ef0c6a45e743a0b2580ca-5f69d7c4-e3d8-4d41-b248-b94a20d7cbf0-2023-05-10-14-001.json.log"
      Then replace it with the new path on the new Remote Engine:
      "currentFilePath":"New_RemoteEngine_installation/data/cls/640ef0c6a45e743a0b2580ca-5f69d7c4-e3d8-4d41-b248-b94a20d7cbf0-2023-05-10-14-001.json.log"
      On Windows, this path look like
      "currentFilePath":"RemoteEngine_installation\\data\\cls\\647867abadf89464d65ab1f2-02141bff-bffe-4975-8835-7dd9ff77b62a-2023-06-01-11-001.json.log
  6. Pair the new Remote Engine to the same Talend Cloud Management Console engine item that you previously unpaired from the old Remote Engine.
    1. In the New_RemoteEngine_installation/etc/org.talend.ipaas.rt.pairing.client.cfg file, specify the endpoint for pairing of Talend Cloud Management Console in the following line: 
      pairing.service.url=https://pair.<your_datacenter>.cloud.talend.com

      For the exact endpoint URL to be used, see the Remote Engine pairing entry of your region in Regions and URLs.

      If you have installed the new engine using the installer, you can skip this step because the endpoint should have been automatically specified.

    2. In New_RemoteEngine_installation/etc/preauthorized.key.cfg, replace the fake preauthorized key with the actual one generated by the unpairing call at the beginning of the procedure.
      If you have lost this preauthorized key, find it back in the Remote Engine key field of the engine item to be paired on Talend Cloud Management Console. For further information about how to find this key, see Creating a Remote Engine.
  7. Start the new Remote Engine.
    See further information at Starting a Remote Engine.

Results

Verify the task status on Talend Cloud Management Console by checking that data service Job tasks and Route tasks are up and running. Logs and metrics should also be available for all these tasks.