Improving Job execution time - 8.0

Talend Administration Center User Guide

Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
Talend Administration Center
Administration and Monitoring
Last publication date

remoteDataRetriever.threadPool.size parameter can be used to improve job execution time when many Jobs are running simultaneously.

This parameter produces longer running jobs, which is meant to address a side effect of the current log retrieving mechanism: each execution implies a new thread, which in turn increases the number of tasks and induces job execution delays.

Go to the configuration table of the database and edit remoteDataRetriever.threadPool.size value (number of threads in the pool). By default, the value is set to 30 threads.

Note: If you need to go back to the old log retrieving mechanism, go to the file and set jobserver.log.retriever.deprecated to true. In this case, remoteDataRetriever.threadPool.size is ignored. With the old log retrieving mechanism, the number of tasks is reduced but the memory consumption is increased.