Skip to main content Skip to complementary content
Announcements

Running tasks in parallel on Remote Engine

A Talend Remote Engine has always been able to run different tasks in parallel. From v2.12.0 onwards, it also supports concurrent run of identical tasks.

In other words, you can run multiple instances of a same task in parallel on a Remote Engine v2.12.0 and onwards.

This section explains the settings to be used to manage this parallel execution capability of a single Remote Engine, with or without this engine being part of a Remote Engine cluster.

Before you begin

  • Ensure that you have the latest engine version installed.
  • Ensure to apply concurrent programming techniques on your Job design and implementation so that your task is thread-safe, that is to say, adapted to parallel executions, especially if you decide to enable parallel runs of a task on a single Remote Engine. For example, the parallel runs must have consistent access to shared resources.

Procedure

  1. Stop your engine.
  2. Browse to the <RemoteEngineInstallationDirectory>/etc folder.
  3. Open the org.talend.ipaas.rt.deployment.agent.cfg file and edit parameters as follows:
    The following table presents the editable parameters in org.talend.ipaas.rt.deployment.agent.cfg file
    Parallel runs of identical tasks
    • If you need to enable this support, ensure that the following line is present: 
      job.deployment.direct=true

      For Remote Engine v2.12.4 and onwards, direct deployment becomes the default and only deployment used by the engine. This job.deployment.direct parameter is thus removed, since you do not need to set it anymore.

      At the same time, you must allow parallel executions of this task to eventually make use of this engine configuration, that is to say, perform either of the following operations:
      • In Talend Management Console, select the Allow parallel execution of this task option when doing the task configuration.
      • Via API, set the parallelExecutionAllowed parameter as true when issuing requests to create or update tasks.
      Otherwise, parallel run does not apply despite this engine configuration.
    Maximum number of concurrent runs

    By default, only 3 parallel runs are allowed, whether the tasks to be run are different or identical.

    • You can extend this limit to allow for as many parallel executions of different tasks as you want, if you have enough resources.
      max.deployed.flows=3
      If you set the limit of concurrent tasks to 0, there will be no limitation on the number of tasks you can run on the remote engine in parallel, provided you have enough resources.
    In a Remote Engine cluster When a Remote Engine has been added to a cluster, parallel runs of identical or different tasks are by default spread out across the cluster.
    However, when the tasks to be run are identical, you can boost the parallel executions on a single Remote Engine, by setting the following parameter to be true:
    cluster.engine.same.task.parallel.run=true
  4. Restart your engine.

Results

You can start executing tasks on your Talend Remote Engine after it becomes Available again in Talend Management Console.
Troubleshooting: When multiple threads of your task are being deployed, one thread may accidentally overwrite another thread, resulting in the failure of the overall deployment.
To repair this issue, do the following:
  1. Open the org.talend.ipaas.rt.deployment.agent.cfg file.
  2. Add a new parameter to the file:
    lock.same.task.jobs.parallel.deployment=true
  3. Save the file and restart your engine.

This parameter locks a thread during the deployment to prevent it from being overwritten by the other parallel threads of the same Job.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!

Leave your feedback here