Talend Data Stewardship in cluster mode

Talend Real-time Big Data Platform Installation Guide for Linux

EnrichVersion
6.3
EnrichProdName
Talend Real-Time Big Data Platform
task
Installation and Upgrade

You can install several instances of Talend Data Stewardship in cluster mode if you want to benefit from a high availability and a better scalability with your product.

Clustering is the process of grouping together a set of similar physical systems in order to ensure a level of operational continuity and minimize the risk of unplanned downtime, in particular by taking advantage of load balancing and failover features.

This documentation provides the procedures to set up a cluster for Talend Data Stewardship, and optionally Talend Dictionary Service.

Architecture of Talend Data Stewardship in cluster mode

The following diagram illustrates the architecture behind Talend Data Stewardship and Talend Dictionary Service when set up in cluster mode.

This architecture is composed of several functional blocks:

  • A Load Balancer, that distributes the workload from the different users accessing the Talend Data Stewardship Web application at the same time as well as the Talend Dictionary Service server.

  • The Talend Data Stewardship instances.

  • The Talend Dictionary Service instances that you can optionally install if you want to add, remove, or edit the semantic types used on data in Talend Data Stewardship.

  • A block containing the various components necessary for Talend Data Stewardship and Talend Dictionary Service to work, namely several instances of MongoDB for storage, Kafka and Zookeeper for messaging, and an instance of Talend Administration Center to manage authorizations.

Installing Talend Data Stewardship in cluster mode

To install Talend Data Stewardship in cluster mode, you need to make some modifications in the <Data_Stewardship_Path>/tds/apache-tomcat/conf/data-stewardship.properties configuration file.

To perform this installation, you need to install and configure as many instances of Talend Data Stewardship and its dependencies as necessary.

Prerequisites:

  • You have configured a Load Balancer for each module, namely Talend Data Stewardship and optionally Talend Dictionary Service.

  • You have configured MongoDB in cluster mode. For more information, see MongoDB documentation.

  • You have configured Kafka and Zookeeper in cluster mode. For more information, see Zookeeper documentation and Kafka documentation.

  1. Install a first Talend Data Stewardship instance.

    For more information on the installation procedure, see Installing and configuring Talend Data Stewardship .

  2. In the <Data_Stewardship_Path>/tds/apache-tomcat/conf/data-stewardship.properties file, edit the mongodb.host property to specify the hosts and ports of the several MongoDB instances.

    Use the following syntax:

    spring.data.mongodb.host=<host1>:<port1>,<host2>:<port2>,…,<hostN>

    The hosts and ports for the different URLs must be concatenated, except for the last host, that will inherit the value of the mongodb.port property. For example:

    spring.data.mongodb.host=mongorep-mongodb-replica-1.mongorep-mongodbreplica.
    default.svc.cluster.local:27017,
    mongorep-mongodb-replica-0.mongorep-mongodbreplica.
    default.svc.cluster.local:27017,
    mongorep-mongodb-replica-2.mongorep-mongodbreplica.
    default.svc.cluster.local:27017,
    mongorep-mongodb-replica-3.mongorep-mongodbreplica.
    default.svc.cluster.local
    spring.data.mongodb.host=27017
  3. Edit the properties specifying the hosts and ports for the Kafka and Zookeeper instances.

    In the same way as the MongoDB URLs, the Kafka and Zookeeper hosts and ports must be concatenated, except for the last port, that is inherited from the dedicated properties.

    talend.kafka.brokers=host1:9092,host2:9092,host3
    talend.kafka.port=9092
    talend.zookeeper.nodes=host1:2181,host2:2181,host3
    talend.zookeeper.port=2181

    Specify also the below peer port parameters which identify the host name with the port number.

    kafka.broker=host1:9092,host2:9092,host3:9092
    schema.kafka.broker=host1:9092,host2:9092,host3:9092
    
  4. To increase the session duration and reduce the risk of unexpected logouts, add the following lines:

    security.token.renew-after=600
    security.token.invalid-after=3600
  5. Repeat the above steps to install and configure other instances of Talend Data Stewardship. Increment the value in the service.instance.id parameter at <Data_Stewardship_Path>/tds/apache-tomcat/conf/data-stewardship.properties to use a unique identifier per instance.

  6. Create partitions for Kafka topics in each Talend Data Stewardship instance:

    1. Launch a Talend Data Stewardship instance. This automatically creates several Kafka topics.

    2. Stop the instance and define the partitions per topics manually. You need to define as many partitions as Kafka nodes.

      For further information, see Kafka documentation.

    3. Restart the instance.

    You have installed several Talend Data Stewardship instances and configured them to work in cluster mode.

Installing Talend Dictionary Service in cluster mode

You can optionally install Talend Dictionary Service in cluster mode, to add, remove, or edit the semantic types used on data in Talend Data Stewardship.

To install Talend Dictionary Service in cluster mode, you need to modify the <Tomcat>/conf/data-quality.properties configuration file.

To perform this installation, you need to install and configure as many instances of Talend Dictionary Service, and its dependencies, as necessary.

Prerequisites:

  1. Install a first Talend Dictionary Service instance.

    For more information on the installation procedure, see Installing and configuring Talend Dictionary Service .

  2. In the <Tomcat>/conf/data-quality.properties file, edit the mongodb.host property to specify the hosts and ports of the several MongoDB instances.

    Use the following syntax:

    mongodb.host=<host1>:<port1>,<host2>:<port2>,…,<hostN>

    The hosts and ports for the different URLs must be concatenated, except for the last host, that will inherit the value of the mongodb.port property. For example:

    mongodb.host=mongorep-mongodb-replica-1.mongorep-mongodbreplica.
    default.svc.cluster.local:27017,
    mongorep-mongodb-replica-0.mongorep-mongodbreplica.
    default.svc.cluster.local:27017,
    mongorep-mongodb-replica-2.mongorep-mongodbreplica.
    default.svc.cluster.local:27017,
    mongorep-mongodb-replica-3.mongorep-mongodbreplica.
    default.svc.cluster.local
    mongodb.port=27017
  3. Edit the properties specifying the hosts and ports for the Kafka and Zookeeper instances.

    In the same way as the MongoDB URLs, the Kafka and Zookeeper hosts and ports must be concatenated, except for the last port, that is inherited from the dedicated properties.

    spring.cloud.stream.kafka.binder.brokers=host1:9092,host2:9092,host3
    spring.cloud.stream.kafka.binder.zkNodes=host1:2181,host2:2181,host3
    spring.cloud.stream.kafka.binder.defaultBrokerPort=9092
    spring.cloud.stream.kafka.binder.defaultZkPort=2181
  4. Repeat this installation and configuration procedure for each instance of Talend Dictionary Service that you want to install.

You have installed several Talend Dictionary Service instances and configured them to work in cluster mode.

Talend Data Stewardship cluster mode limitations

When you install Talend Data Stewardship in cluster mode, unexpected logouts from the interface may occasionally happen, even if the risk is minimal. See the corresponding Jira ticket https://jira.talendforge.org/browse/TDS-1974.

Also you may need to restart Talend Data Stewardship so that new Kafka consumer groups are taken into account. See the corresponding Jira ticket https://jira.talendforge.org/browse/TDS-1975.