Execution engines in Talend Cloud - Cloud

Talend Management Console User Guide

Version
Cloud
Language
English
Product
Talend Cloud
Module
Talend Management Console
Content
Administration and Monitoring > Managing projects
Administration and Monitoring > Managing users
Deployment > Deploying > Executing Pipelines
Deployment > Deploying > Executing Tasks
Deployment > Scheduling > Scheduling Tasks
Last publication date
2024-02-29
You can use the following types of engines to execute your tasks and plans in Talend Management Console.

Cloud Engine

The Cloud Engine is a compute resource managed by Talend in the cloud that executes Job tasks. You can allocate Cloud Engines to environments proportionally to the number of concurrent task executions you plan to run.
  • A default Cloud Engine is automatically available. It cannot be assigned to any specific environment, that is to say, this default Cloud Engine is always an unassigned engine.

    This engine is therefore not displayed in the Environment details blade.

  • Unassigned Cloud Engines can be used by all environments. If Cloud Engines are not allocated to environments, you may not be able to run certain tasks, because other tasks keep all the engines occupied.

    The default environment can use unassigned Cloud Engines only.

  • Each Cloud Engine, including this default Cloud Engine, requires 45000 engine tokens to be able to run and allows up to 3 tasks to run in parallel on it. These 45000 tokens are released once this engine is shut down.

    If you choose to run your task on Cloud Exclusive, no other task will be executed on that Cloud Engine. Cloud Exclusive engines can only be used in environments that do not have Cloud Engines assigned to them.

    If three different tasks or the same task is already running on a Cloud Engine, another engine will be selected to the task execution.

    The following system resource limitations are valid for Jobs running on Cloud Engines:

    • vCPU: 2
    • Disk usage: 200GB
    • Memory usage: 8GB
    • Throughput: 225GB per hour

    For Jobs that require more resources, consider using Talend Remote Engine as the target runtime.

  • A running Cloud Engine is verified every one hour with regard to its usage. If no task run has been found during this hour, this Cloud Engine is automatically shut down; otherwise, it keeps running and another shutdown verification is scheduled for the end of another one hour. This is the general lifecycle of a Cloud Engine.

    However, if a warm-up event occurs, for example, a task run schedule is set up on this Cloud Engine, the ongoing shutdown schedule is canceled and the shutdown verification is re-scheduled for the end of another hour.

    If your license is the community level, user login to Talend Cloud is also considered to be a warm-up event. In this case, the default Cloud Engine is started upon user login.

    This shutdown verification mechanism helps avoid shutting down a Cloud Engine too frequently, as starting a Cloud Engine always requires quite some ramp-up time.

Talend Remote Engine

Remote Engines allow you to run Job, Route, and data service tasks that use on-premises applications and databases. By default, maximum 3 tasks can run in parallel on the same engine. If you are using a Remote Engine to run identical tasks simultaneously, your Remote Engine must be v2.12.0 onwards.

Outbound communication between Talend Management Console and Talend Remote Engine is fully secured as data is not staged.

Data service and Route tasks can only be deployed on Remote Engines. OSGi type deployments require that Talend Runtime 7.1.1 version or later is installed and running on the same machine as the Talend Remote Engine. Microservices do not require any other application for Route and REST Data service deployments, except the Data Service Runner module inside the Remote Engine. Data service tasks can be executed only on engines that have a Microservice or Talend Runtime (for OSGi) run profile configured.

Remote Engines can be grouped together to form clusters. Any number of clusters can be created in an environment. However, Remote Engines added to a cluster cannot be used to execute tasks directly from Talend Studio.

A Remote Engine requires 9000 engine tokens to be able to run.

For more information about Talend Remote Engine and run profiles linked to Talend Studio artifacts, see Configuring Job run profiles.

Cloud Engine for Design

The Cloud Engine for Design is a built-in runner that allow you to easily design pipelines without having to set up any processing engines. You can execute pipelines with different resource allocations on a Cloud Engine for Design.

The following resource limitations are valid for pipelines running on Cloud Engine for Design:

  • Memory usage: 8GB
  • Two pipelines can be run in parallel

For advanced processing of data it is recommended to install the secure Remote Engine Gen2.

Remote Engine Gen2

A Remote Engine Gen2 is a secure execution engine on which you can safely run pipelines. By using this engine, you can make sure that data processing happens in a safe and secure environment as Talend never has access to the data and resources of your pipelines. You have control over your execution environment and resources as you are able to create and configure the engine in your own environment (Virtual Private Cloud or on premises). The engine also ensures optimal performance and security by increasing the data locality instead of moving large data to computation.

You can execute pipelines with different resource allocations on a Remote Engine Gen2 by using run profiles.

For more information about Remote Engine Gen2, see the Talend Remote Engine Gen2 Quick Start Guide.