A Talend Remote Engine has always been able to run different tasks in parallel. From v2.12.0 onwards, it also supports concurrent run of identical tasks.
In other words, you can run multiple instances of a same task in parallel on a Remote Engine v2.12.0 and onwards.
This section explains the settings to be used to manage this parallel execution capability of a single Remote Engine, with or without this engine being part of a Remote Engine cluster.
Before you begin
- Ensure that you have the latest engine version installed.
- Ensure to apply concurrent programming techniques on your Job design and implementation so that your task is thread-safe, that is to say, adapted to parallel executions, especially if you decide to enable parallel runs of a task on a single Remote Engine. For example, the parallel runs must have consistent access to shared resources.
- Stop your engine.
- Browse to the <RemoteEngineInstallationDirectory>/etc folder.
Open the org.talend.ipaas.rt.deployment.agent.cfg file:
Option Description Parallel runs of identical tasks
- If you need to enable this support, ensure that the following line is
For Remote Engine v2.12.4 and onwards, direct deployment becomes the default and only deployment used by the engine. This
job.deployment.directparameter is thus removed, since you do not need to set it anymore.At the same time, you must allow parallel executions of this task to eventually make use of this engine configuration, that is to say, perform either of the following operations:Otherwise, parallel run does not apply despite this engine configuration.
- In Talend Cloud Management Console, select the Allow parallel execution of this task option when doing the task configuration.
- Via API, set the
parallelExecutionAllowedparameter as true when issuing requests to create or update tasks.
Maximum number of concurrent runs
By default, only 3 parallel runs are allowed, whether the tasks to be run are different or identical.
- You can extend this limit to allow for as many parallel executions of
different tasks as you want, if you have enough
If you set the limit of concurrent tasks to 0, there will be no limitation on the number of tasks you can run on the remote engine in parallel, provided you have enough resources.
In a Remote Engine cluster When a Remote Engine has been added to a cluster, parallel runs of identical or different tasks are by default spread out across the cluster.However, when the tasks to be run are identical, you can boost the parallel executions on a single Remote Engine, by setting the following parameter to be
- If you need to enable this support, ensure that the following line is present:
- Restart your engine.
- Open the org.talend.ipaas.rt.deployment.agent.cfg file.
- Add a new parameter to the
- Save the file and restart your engine.
This parameter locks a thread during the deployment to prevent it from being overwritten by the other parallel threads of the same Job.