Installing Talend Data Stewardship in cluster mode - 7.3

Talend Installation Guide

Version
7.3
Language
English
Operating system
Linux
Subscription type
Subscription
Product
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
Module
Talend Activity Monitoring Console
Talend Administration Center
Talend Artifact Repository
Talend CommandLine
Talend Data Preparation
Talend Data Stewardship
Talend DQ Portal
Talend ESB
Talend Identity and Access Management
Talend Installer
Talend JobServer
Talend LogServer
Talend MDM Server
Talend MDM Web UI
Talend Repository Manager
Talend Runtime
Talend SAP RFC Server
Talend Studio
Content
Installation and Upgrade
Last publication date
2022-10-30

To install Talend Data Stewardship in cluster mode, you need to make some modifications in the <Data_Stewardship_Path>/tds/apache-tomcat/conf/data-stewardship.properties configuration file.

To perform this installation, you need to install and configure as many instances of Talend Data Stewardship and its dependencies as necessary.

Before you begin

Procedure

  1. Install a first Talend Data Stewardship instance.
    For more information on the installation procedure, see Installing Talend Data Stewardship manually.
  2. In the <Data_Stewardship_Path>/tds/apache-tomcat/conf/data-stewardship.properties file, edit the mongodb.host property to specify the hosts and ports of the several MongoDB instances.
    Use the following syntax:
    spring.data.mongodb.host=<host1>:<port1>,<host2>:<port2>,...,<hostN>
    The hosts and ports for the different URLs must be concatenated, except for the last host, that will inherit the value of the mongodb.port property. For example:
    spring.data.mongodb.host=mongorep-mongodb-replica-1.mongorep-mongodbreplica.
    default.svc.cluster.local:27017,
    mongorep-mongodb-replica-0.mongorep-mongodbreplica.
    default.svc.cluster.local:27017,
    mongorep-mongodb-replica-2.mongorep-mongodbreplica.
    default.svc.cluster.local:27017,
    mongorep-mongodb-replica-3.mongorep-mongodbreplica.
    default.svc.cluster.local
    spring.data.mongodb.host=27017
  3. Edit the properties specifying the hosts and ports for the Kafka and ZooKeeper instances.
    In the same way as the MongoDB URLs, the Kafka and ZooKeeper hosts and ports must be concatenated, except for the last port, that is inherited from the dedicated properties.
    talend.kafka.brokers=host1:9092,host2:9092,host3
    talend.kafka.port=9092
    talend.zookeeper.nodes=host1:2181,host2:2181,host3
    talend.zookeeper.port=2181
    Specify also the below peer port parameters which identify the host name with the port number.
    kafka.broker=host1:9092,host2:9092,host3:9092
    schema.kafka.broker=host1:9092,host2:9092,host3:9092
    
  4. To increase the session duration and reduce the risk of unexpected logouts, add the following lines:
    security.token.renew-after=600
    security.token.invalid-after=3600
  5. Repeat the above steps to install and configure other instances of Talend Data Stewardship.
    Make sure to increment the values for the below parameters at <Data_Stewardship_Path>/tds/apache-tomcat/conf/data-stewardship.properties for each Talend Data Stewardship instance to have a unique property per instance:
    tds.dqDictionary.group=TDSCoreDqDictionaryGroup1
    schema.dqDictionary.group=SchemaServiceDqDictionaryGroup1
  6. Edit the <Data_Stewardship_Path>/iam/apache-tomcat/clients/tds-client.json files to add the redirection URLs in the post_logout_redirect_uris and redirect_uris fields specifying the load balancer ports.
    Optionally, to access directly one of the Talend Data Stewardship instances add the redirection URLs of the other instances in the fields.
  7. Create partitions for Kafka topics in each Talend Data Stewardship instance:
    1. Launch a Talend Data Stewardship instance. This automatically creates several Kafka topics.
    2. Stop the instance and define the partitions per topics manually. You need to define as many partitions as Kafka nodes.
      For more information, see Kafka documentation.
    3. Restart the instance.

Results

You have installed several Talend Data Stewardship instances and configured them to work in cluster mode.

Note: If you have a Platform license which includes Talend Dictionary Service, you may want to install it in cluster mode as well. For more information, see Installing Talend Dictionary Service in cluster mode.