Customizing Data APIs usage - Cloud

Talend Remote Engine Gen2 Quick Start Guide

Talend Cloud
Talend Management Console
Talend Pipeline Designer
Deployment > Deploying > Executing Pipelines
Installation and Upgrade

You can edit some configuration parameters to refine the volume of data that can be exchanged between the Remote Engine Gen2 and the database you created a Data API on, depending on your usage.

Open the default/.env file in the Remote Engine Gen2 installation directory, and set your preferred values for the following parameters:

Parameter Description
DATASET_API_NUMBER_CONNECTIONS_PER_DATABASE This parameter allows you to control the number of permanent connections opened by the Remote Engine Gen2 with a database.

More connections means that you can handle more requests in parallel, but in return, the database can also protect itself by setting a maximum number of parallel connections, regardless of the origin.

The default value is 5.

DATASET_API_JDBC_FETCH_SIZE This parameter allows you to control the number of records coming from the database, that the Remote Engine Gen2 will have to process at a given time.

Setting this value to 0 will disable this feature. With each request to the database, all records are retrieved in one operation.

Setting a low value will reduce the memory footprint of the JVM, but will increase the number of exchanges with the database.

Setting a high value will reduce the number of exchanges with the database.

The default value is 1000.

DATASET_API_ODATA_MAX_SERVER_PAGE_SIZE This parameter allows you to control how much records are paginated on the server side and written to the OData client.

OData clients are usually able to automatically consume several consecutive pages.

Setting a low value mean that the server only handles a few records at a time, and consumes less memory.

Setting a high value means that the server will handle a high number of records at a time.

Setting this parameter to 0 means there will be no server side pagination

The default value is 10000 for the Remote Engine Gen2, and 1000 when using the Cloud Engine for Design.