Improving performance when loading data from the Staging Area

staging.validation.updatereport

boolean

When you run the validation process that loads data from the Staging Area to Talend MDM, by default this process generates update reports that record inserts in the Journal and fire any relevant Triggers.

To instruct MDM Server not to generate update reports when loading data from the Staging Area, set this property to false.

staging.validation.pool

int

By default, MDM Server assigns two threads to the validation process that loads data from the Staging Area to Talend MDM.

If the machine running MDM Server has some spare CPU, edit the value of this property to increase the number of threads that can be used.

staging.validation.commit

int

By default, when loading data from the Staging Area to Talend MDM, Talend MDM commits records to the master database every 1000 validated objects.

To improve performance, increase this value to reduce the number of commits made to the database.

staging.validation.buffer.threshold

int

A buffer is used to transfer records from the Staging Area to the master database. By default, the buffer holds a maximum of 1000 records; once this threshold is reached, the action of reading from the Staging Area will be paused and a check will be made every second to see if the number of records in the buffer has decreased.

Since reading from the Staging Area is always faster than writing to the master database, reduce the value of the buffer size if you encounter memory issues.