To install Talend Data Stewardship in
cluster mode, you need to make some modifications in the
<Data_Stewardship_Path>/tds/apache-tomcat/conf/data-stewardship.properties
configuration file.
To perform this installation, you need to install and configure as many instances of
Talend Data Stewardship and its dependencies as
necessary.
Procedure
-
Install a first Talend Data Stewardship
instance.
-
In the
<Data_Stewardship_Path>/tds/apache-tomcat/conf/data-stewardship.properties
file, edit the
spring.data.mongodb.host property to specify the hosts and
ports of the several MongoDB instances.
Use the following
syntax:
spring.data.mongodb.host=<host1>:<port1>,<host2>:<port2>,...,<hostN>
The
hosts and ports for the different URLs must be concatenated, except for the last
host, that will inherit the value of the
mongodb.port property.
For
example:
spring.data.mongodb.host=mongorep-mongodb-replica-1.mongorep-mongodbreplica.
default.svc.cluster.local:27017,
mongorep-mongodb-replica-0.mongorep-mongodbreplica.
default.svc.cluster.local:27017,
mongorep-mongodb-replica-2.mongorep-mongodbreplica.
default.svc.cluster.local:27017,
mongorep-mongodb-replica-3.mongorep-mongodbreplica.
default.svc.cluster.local
spring.data.mongodb.host=27017
-
Edit the properties specifying the hosts and ports for the
Kafka and ZooKeeper instances.
In the same way as the MongoDB URLs, the Kafka and ZooKeeper
hosts and ports must be concatenated, except for the last
port, that is inherited from the dedicated properties.
talend.kafka.brokers=host1:9092,host2:9092,host3
talend.kafka.port=9092
talend.zookeeper.nodes=host1:2181,host2:2181,host3
talend.zookeeper.port=2181
Specify also the below peer port parameters which
identify the host name with the port
number.
kafka.broker=host1:9092,host2:9092,host3:9092
schema.kafka.broker=host1:9092,host2:9092,host3:9092
-
To increase the session duration and reduce the risk of unexpected logouts, add
the following lines:
security.token.renew-after=600
security.token.invalid-after=3600
-
Repeat the above steps to install and configure other instances of Talend Data Stewardship.
Make sure to increment the values for the below parameters at
<Data_Stewardship_Path>/tds/apache-tomcat/conf/data-stewardship.properties
for each
Talend Data Stewardship instance
to have a unique property per
instance:
tds.dqDictionary.group=TDSCoreDqDictionaryGroup1
schema.dqDictionary.group=SchemaServiceDqDictionaryGroup1
-
Edit the
<Data_Stewardship_Path>/iam/apache-tomcat/clients/tds-client.json
files to add the redirection URLs in the
post_logout_redirect_uris and redirect_uris
fields specifying the load balancer ports.
Optionally, to access directly one of the Talend Data Stewardship instances add the
redirection URLs of the other instances in the fields.
-
Create partitions for Kafka topics in each Talend Data Stewardship instance:
-
Launch a Talend Data Stewardship
instance. This automatically creates several Kafka topics.
-
Stop the instance and define the partitions per topics manually. You
need to define as many partitions as Kafka nodes.
-
Restart the instance.
Results
You have installed several Talend Data Stewardship instances and configured them
to work in cluster mode.