To install Talend Data Stewardship in cluster mode, you need to make some modifications in the <Data_Stewardship_Path>/tds/apache-tomcat/conf/data-stewardship.properties configuration file.
To perform this installation, you need to install and configure as many instances of Talend Data Stewardship and its dependencies as necessary.
Before you begin
- You have configured a Load Balancer for Talend Data Stewardship.
- You have configured MongoDB in cluster mode. For more information, seeMongoDB documentation.
- You have configured Kafka and Zookeeper in cluster mode. For more information, see Zookeeper documentation and Kafka documentation
- You have configured Talend Identity and Access Management in cluster mode. For more information, see Installing Talend Identity and Access Management in cluster mode.
Install a first Talend Data Stewardship
For more information on the installation procedure, see Installing and configuring Talend Data Stewardship.
file, edit the
mongodb.hostproperty to specify the hosts and ports of the several MongoDB instances.Use the following syntax:
spring.data.mongodb.host=<host1>:<port1>,<host2>:<port2>,...,<hostN>The hosts and ports for the different URLs must be concatenated, except for the last host, that will inherit the value of the
mongodb.portproperty. For example:
spring.data.mongodb.host=mongorep-mongodb-replica-1.mongorep-mongodbreplica. default.svc.cluster.local:27017, mongorep-mongodb-replica-0.mongorep-mongodbreplica. default.svc.cluster.local:27017, mongorep-mongodb-replica-2.mongorep-mongodbreplica. default.svc.cluster.local:27017, mongorep-mongodb-replica-3.mongorep-mongodbreplica. default.svc.cluster.local spring.data.mongodb.host=27017
Edit the properties specifying the hosts and ports for the Kafka and Zookeeper
In the same way as the MongoDB URLs, the Kafka and Zookeeper hosts and ports must be concatenated, except for the last port, that is inherited from the dedicated properties.
talend.kafka.brokers=host1:9092,host2:9092,host3 talend.kafka.port=9092 talend.zookeeper.nodes=host1:2181,host2:2181,host3 talend.zookeeper.port=2181Specify also the below peer port parameters which identify the host name with the port number.
To increase the session duration and reduce the risk of unexpected logouts, add
the following lines:
Repeat the above steps to install and configure other instances of Talend Data Stewardship.
Make sure to increment the values for the below parameters at <Data_Stewardship_Path>/tds/apache-tomcat/conf/data-stewardship.properties for each Talend Data Stewardship instance to have a unique property per instance:
files to add the redirection URLs in the
redirect_urisfields specifying the load balancer ports.Optionally, to access directly one of the Talend Data Stewardship instances add the redirection URLs of the other instances in the fields.
Create partitions for Kafka topics in each Talend Data Stewardship instance:
- Launch a Talend Data Stewardship instance. This automatically creates several Kafka topics.
Stop the instance and define the partitions per topics manually. You
need to define as many partitions as Kafka nodes.
For more information, see Kafka documentation.
- Restart the instance.
You have installed several Talend Data Stewardship instances and configured them to work in cluster mode.