Running tasks in parallel on a single Remote Engine - Cloud

Talend Remote Engine User Guide for Linux

Version
Cloud
Language
English (United States)
EnrichDitaval
linux
Product
Talend Cloud
Module
Talend Remote Engine
Content
Design and Development
Installation and Upgrade

A Talend Remote Engine has always been able to run different tasks in parallel. From v2.12.0 onwards, it also supports concurrent run of identical tasks.

In other words, you can run multiple instances of a same task in parallel on a Remote Engine v2.12.0 and onwards.

This section explains the settings to be used to manage this parallel execution capability of a single Remote Engine, with or without this engine being part of a Remote Engine cluster.

Before you begin

  • Ensure that you have the latest engine version installed.
  • Ensure to apply concurrent programming techniques on your Job design and implementation so that your task is thread-safe, that is to say, adapted to parallel executions, especially if you decide to enable parallel runs of a task on a single Remote Engine. For example, the parallel runs must have consistent access to shared resources.

Procedure

  1. Stop your engine.
  2. Browse to the <RemoteEngineInstallationDirectory>/etc folder.
  3. Open the org.talend.ipaas.rt.deployment.agent.cfg file:
    Option Description
    Parallel runs of identical tasks
    • If you need to enable this support, ensure that the following line is present:
      job.deployment.direct=true
      This is also the default setting.
      At the same time, you must allow parallel executions of this task to eventually make use of this engine configuration, that is to say, perform either of the following operations:
      • In Talend Cloud Management Console, select the Allow parallel execution of this task option when doing the task configuration.
      • Via API, set the parallelExecutionAllowed parameter as true when issuing requests to create or update tasks.
      Otherwise, parallel run does not apply despite this engine configuration.
    Maximum number of concurrent runs

    By default, only 3 parallel runs are allowed, whether the tasks to be run are different or identical.

    • You can extend this limit to allow for as many parallel executions of different tasks as you want, if you have enough resources.
      max.deployed.flows=3
      If you set the limit of concurrent tasks to 0, there will be no limitation on the number of tasks you can run on the remote engine in parallel, provided you have enough resources.
    In a Remote Engine cluster When a Remote Engine has been added to a cluster, parallel runs of identical or different tasks are by default spread out across the cluster.
    However, when the tasks to be run are identical, you can boost the parallel executions on a single Remote Engine, by setting the following parameter to be true:
    cluster.engine.same.task.parallel.run=true
  4. Restart your engine.

Results

You can start executing tasks on your Talend Remote Engine after it becomes Available again in Talend Cloud Management Console.