Improving performance when loading data from the Staging Area - 7.0

Talend MDM Platform Installation Guide for Windows

EnrichVersion
7.0
EnrichProdName
Talend MDM Platform
task
Installation and Upgrade
EnrichPlatform
Talend Activity Monitoring Console
Talend Administration Center
Talend Artifact Repository
Talend CommandLine
Talend Data Preparation
Talend Data Stewardship
Talend DQ Portal
Talend ESB
Talend Identity and Access Management
Talend Installer
Talend JobServer
Talend Log Server
Talend MDM Server
Talend MDM Web UI
Talend Repository Manager
Talend Runtime
Talend SAP RFC Server
Talend Studio
The mdm.conf file, which is stored in the <$INSTALLDIR>\conf directory, includes several properties which can be edited to improve performance.

Property

Type

Description

staging.validation.updatereport

boolean

When you run the validation process that loads data from the Staging Area to Talend MDM, by default this process generates update reports that record inserts in the Journal and fire any relevant Triggers.

To instruct MDM Server not to generate update reports when loading data from the Staging Area, set this property to false.

staging.validation.pool

int

By default, MDM Server assigns two threads to the validation process that loads data from the Staging Area to Talend MDM.

If the machine running MDM Server has some spare CPU, edit the value of this property to increase the number of threads that can be used.

staging.validation.commit

int

By default, when loading data from the Staging Area to Talend MDM, Talend MDM commits records to the master database every 1000 validated objects.

To improve performance, increase this value to reduce the number of commits made to the database.

staging.validation.buffer.threshold

int

A buffer is used to transfer records from the Staging Area to the master database. By default, the buffer holds a maximum of 1000 records; once this threshold is reached, the action of reading from the Staging Area will be paused and a check will be made every second to see if the number of records in the buffer has decreased.

Since reading from the Staging Area is always faster than writing to the master database, reduce the value of the buffer size if you encounter memory issues.