Improving performance when loading data from the Staging Area - 8.0

Talend Installation Guide for Linux

Version
8.0
Language
English
EnrichDitaval
Linux
EnrichSubscriptionType
Subscription
Product
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
Module
Talend Activity Monitoring Console
Talend Administration Center
Talend Artifact Repository
Talend CommandLine
Talend Data Preparation
Talend Data Stewardship
Talend ESB
Talend Identity and Access Management
Talend Installer
Talend JobServer
Talend LogServer
Talend MDM Server
Talend MDM Web UI
Talend Runtime
Talend SAP RFC Server
Talend Studio
Content
Installation and Upgrade
Available in...

Data Fabric

MDM Platform

The mdm.conf file, which is stored in the <$INSTALLDIR>/conf directory, includes several properties which can be edited to improve performance.

Property

Type

Description

staging.validation.updatereport

boolean

When you run the validation process that loads data from the Staging Area to Talend MDM, by default this process generates update reports that record inserts in the Journal and fire any relevant Triggers.

To instruct MDM Server not to generate update reports when loading data from the Staging Area, set this property to false.

staging.validation.pool

int

By default, MDM Server assigns two threads to the validation process that loads data from the Staging Area to Talend MDM.

If the machine running MDM Server has some spare CPU, edit the value of this property to increase the number of threads that can be used.

staging.validation.commit

int

By default, when loading data from the Staging Area to Talend MDM, Talend MDM commits records to the master database every 1000 validated objects.

To improve performance, increase this value to reduce the number of commits made to the database.

staging.validation.buffer.threshold

int

A buffer is used to transfer records from the Staging Area to the master database. By default, the buffer holds a maximum of 1000 records; once this threshold is reached, the action of reading from the Staging Area will be paused and a check will be made every second to see if the number of records in the buffer has decreased.

Since reading from the Staging Area is always faster than writing to the master database, reduce the value of the buffer size if you encounter memory issues.