Installing Talend Data Preparation in cluster mode

Installing Talend Data Preparation in cluster mode - Cloud

Talend Cloud Hybrid Installation Guide

Version

Cloud

Language

English

Operating system

Linux

Product

Talend Cloud

Module

Talend Data Preparation

Talend Data Stewardship

Talend Management Console

Content

Installation and Upgrade

Last publication date

2023-12-08

To install Talend Data Preparation in cluster mode, you need to make some modifications in the <Data_Preparation_Path>/config/application.properties configuration file.

To perform this installation, you need to install and configure as many instances of Talend Data Preparation and its dependencies as necessary.

Before you begin

You have configured a Load Balancer for Talend Data Preparation.
You have configured MongoDB in cluster mode. For more information, see MongoDB documentation.
You have configured Kafka and ZooKeeper in cluster mode. For more information, see Zookeeper documentation and Kafka documentation

Procedure

Install a first Talend Data Preparation instance.
For more information on the installation procedure, see Installing Talend Data Preparation manually.

In the <Data_Preparation_Path>/config/application.properties file, edit the spring.data.mongodb.host property to specify the hosts and ports of the several MongoDB instances.

Use the following syntax:

spring.data.mongodb.host=<host1>:<port1>,<host2>:<port2>,...,<hostN>

The hosts and ports for the different URLs must be concatenated, except for the last host, that will inherit the value of the mongodb.port property. For example:

mongodb.host=mongorep-mongodb-replica-1.mongorep-mongodb-replica.default.svc.cluster.local:27017,
mongorep-mongodb-replica-0.mongorep-mongodb-replica.default.svc.cluster.local:27017,
mongorep-mongodb-replica-2.mongorep-mongodb-replica.default.svc.cluster.local:27017,
mongorep-mongodb-replica-3.mongorep-mongodb-replica.default.svc.cluster.local
mongodb.port=27017

Edit the service.cache.file.location and dataset.content.store.file.location properties to specify the location of your Network File System, or shared folder that must be available to all the Talend Data Preparation instances. For example:
```
service.cache.file.location=sharedContent/
dataset.content.store.file.location=sharedContent/store/datasets/content/
```
Edit the properties specifying the hosts and ports for the Kafka and ZooKeeper instances.
In the same way as the MongoDB URLs, the Kafka and ZooKeeper hosts and ports must be concatenated, except for the last port, that is inherited from the dedicated properties.
```
spring.cloud.stream.kafka.binder.brokers=host1:9092,host2:9092,host3
spring.cloud.stream.kafka.binder.zkNodes=host1:2181,host2:2181,host3
spring.cloud.stream.kafka.binder.defaultBrokerPort=9092
spring.cloud.stream.kafka.binder.defaultZkPort=2181 
```
To increase the session duration and reduce the risk of unexpected logouts, add the following lines:
```
security.token.renew-after=600
security.token.invalid-after=3600
```

To enable session sharing, add the following lines:

server.session.timeout=1800
spring.session.store-type=MONGODB
spring.session.timeout=1800
spring.session.mongodb.collection-name=tdp_sessions

Repeat the above steps to install and configure other instances of Talend Data Preparation.

Results

You have installed several Talend Data Preparation instances and configured them to work in cluster mode.