Recommendations about environment and configuration for the Talend Administration Center - 7.2

EnrichVersion
7.2
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
task
Administration and Monitoring
EnrichPlatform
Talend Administration Center

Recommendations about environment and configuration for the Talend Administration Center

This article applies to users of Talend Administration Center (TAC) who want to optimize their environment to support a given amount of concurrent tasks.

The following recommendations are based on Talend Administration Center version 6.0.

Note that these recommendations are currently incomplete, the following ones still need to be investigated:

  • recommended resources according to the number of logged users from Studio
  • recommended resources according to the number of logged Talend Administration Center users
  • recommended resources according to the number of concurrent executions of plans

Recommended resources according to the number of concurrent task executions

(*) using CPU Intel(R) Xeon(R) L5640 @ 2.27GHz

(**) using MySQL

500 concurrent task and plan executions

1000 concurrent task and plan executions

2000 concurrent task and plan executions

Recommended minimal CPU (*) number for each Talend Administration Center host

2

4

8

Recommended minimal memory for each Talend Administration Center host

>= 3000 MB

>= 4000 MB

>= 8000 MB

Recommended minimal memory for each Talend Administration Center JVM (-Xmx)

>= 1500 MB

>= 3000 MB

>= 6000 MB

Recommended minimal CPU (*) number for database(**) host

2

4

6

Recommended minimal memory for database(**) host

>= 1500 MB

>= 3000 MB

>= 6000 MB

Recommended minimal number of remote JobServers

1

2

2

Recommended minimal CPU (*) number for each JobServer host

(apart from CPU needed for JVM of executed jobs)

1

2

2

Recommended minimal memory for each JobServer host

(apart from memory needed for JVM of executed jobs)

>= 1000 MB

>= 2500 MB

>= 5000 MB

Recommended minimal memory for each JobServer JVM (-Xmx)

>= 250 MB

>= 500 MB

>= 1000 MB

Recommended configuration

Description

Location

Configuration property

Default/Minimal value

Recommended value

Maximum number of database connections in the Quartz connection pool

Talend Administration Center configuration file

"WEB-INF/classes/quartz.properties" :

org.quartz.dataSource.QRTZ_DS.maxConnections

30

MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS

Maximum number of concurrent Jobs handled by the Scheduler

Talend Administration Center configuration file

"WEB-INF/classes/quartz.properties" :

org.quartz.threadPool.threadCount

30

MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS

Maximum database connections for Talend Administration Center (apart from Quartz)

Talend Administration Center configuration file

"WEB-INF/classes/configuration.properties" :

hibernate.c3p0.max_size

32

MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS + MAX_CONCURRENT_LOGGED_USERS

It defines the period between each remote Job check

Talend Administration Center database table configuration

scheduler.conf.taskStatusRefreshTime

1

MAX_CONCURRENT_TASK_EXECUTIONS / 500

Defines the size of thread pool which checks the latest executions at startup

Talend Administration Center database table configuration

dashboard.conf.taskExecutionsHistory.threadPoolSize

10

( MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS ) / 25

Defines the size of thread pool which checks all the tasks at startup

Talend Administration Center database table configuration

scheduler.conf.simultaneousThreadsForStatusRefresh

5

MAX_CONCURRENT_TASK_EXECUTIONS / 50

Defines the number of maximum opened files for database process

Host of database server

Maximum opened files:

For example, under Linux set the Mysql configuration property "open_files_limit" and ensure that the system file limit is >= to the formula on the right

(depends on operating system)

( MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS + MAX_CONCURRENT_LOGGED_USERS ) x 3

Defines the number of maximum connections allowed to the database

Database server

Max connections:

For example, set the Mysql configuration property "max_connections = 10000"

(depends on database vendor)

( MAX_CONCURRENT_TASK_EXECUTIONS + MAX_CONCURRENT_PLAN_EXECUTIONS + MAX_CONCURRENT_LOGGED_USERS ) x 3

Defines the maximum number of concurrent connections accepted by the JobServer

JobServer configuration file

"conf/TalendJobServer.properties" :

org.talend.remote.server.MultiSocketServer.

MAX_CONCURRENT_CONNECTIONS

1000

MAX_CONCURRENT_JOBS_EXECUTIONS x 2

Definition of variables used in the table above

Variable Description

MAX_CONCURRENT_JOBS_EXECUTIONS

Maximum expected number of concurrent executed Jobs on JobServer side

MAX_CONCURRENT_LOGGED_USERS

Maximum expected number of concurrent logged users (Talend Administration Center + Studio) on Talend Administration Center side

MAX_CONCURRENT_PLAN_EXECUTIONS

Maximum expected number of concurrent plan executions on Talend Administration Center side

MAX_CONCURRENT_TASK_EXECUTIONS

Maximum expected number of concurrent task executions on Talend Administration Center side