Installing Talend Data Preparation in cluster mode - 6.5

Talend Big Data Installation Guide for Windows

EnrichVersion
6.5
EnrichProdName
Talend Big Data
task
Installation and Upgrade
EnrichPlatform
Talend Activity Monitoring Console
Talend Administration Center
Talend Artifact Repository
Talend CommandLine
Talend Data Preparation
Talend Data Stewardship
Talend Identity and Access Management
Talend Installer
Talend JobServer
Talend Log Server
Talend Runtime
Talend Studio

To install Talend Data Preparation in cluster mode, you need to make some additions to the <Data_Preparation_Path>/config/application.properties configuration file.

To perform this installation, you need to install and configure as many instances of Talend Data Preparation, and its dependencies, as necessary.

Before you begin

  • Each product, namely Talend Data Preparation and optionally Talend Dictionary Service, requires the use of a Load Balancer.
  • You have configured MongoDB in cluster mode. For more information, see the MongoDB documentation.
  • You have configured Kafka and Zookeeper in cluster mode. For more information, see the Zookeeper documentation and the Kafka documentation.

Procedure

  1. Install a first Talend Data Preparation instance.

    For more information on the Talend Data Preparation installation procedure, see Installing Talend Data Preparation manually.

  2. In the <Data_Preparation_Path>/config/application.properties file, edit the mongodb.host property to specify the hosts and ports of the several MongoDB instances.

    Use the following syntax: mongodb.host=<host1>:<port1>,<host2>:<port2>,…,<hostN>. The hosts and ports for the different URLs must be concatenated, except for the last host, that will inherit the value of the mongodb.port property. For example:

    mongodb.host=mongorep-mongodb-replica-1.mongorep-mongodb-replica.default.svc.cluster.local:27017,
    mongorep-mongodb-replica-0.mongorep-mongodb-replica.default.svc.cluster.local:27017,
    mongorep-mongodb-replica-2.mongorep-mongodb-replica.default.svc.cluster.local:27017,
    mongorep-mongodb-replica-3.mongorep-mongodb-replica.default.svc.cluster.local
    mongodb.port=27017
  3. Edit the service.cache.file.location and dataset.content.store.file.location properties to specify the location of your Network File System, or shared folder that must be available to all the Talend Data Preparation instances.
    service.cache.file.location=sharedContent/
    dataset.content.store.file.location=sharedContent/store/datasets/content/
  4. Edit the properties specifying the hosts and ports for the Kafka and Zookeeper instances.

    In the same way as the MongoDB URLs, the Kafka and Zookeeper hosts and ports must be concatenated, except for the last port, that is inherited from the dedicated properties.

    spring.cloud.stream.kafka.binder.brokers=host1:9092,host2:9092,host3
    spring.cloud.stream.kafka.binder.zkNodes=host1:2181,host2:2181,host3
    spring.cloud.stream.kafka.binder.defaultBrokerPort=9092
    spring.cloud.stream.kafka.binder.defaultZkPort=2181
  5. To increase the session duration and reduce the risk of unexpected logouts, add the following lines:
    security.token.renew-after=600
    security.token.invalid-after=3600
  6. Repeat this installation and configuration procedure for each instance of Talend Data Preparation that you want to install.

Results

The several Talend Data Preparation instances have been installed and configured to work in cluster mode.