Installing and Configuring Talend server modules

Talend Data Management Platform Installation Guide for Linux

EnrichVersion
6.2
EnrichProdName
Talend Data Management Platform
task
Installation and Upgrade

The following pages provide installation and configuration procedures for your Talend server modules.

Installing and configuring the Nexus artifact repository

Talend Administration Center is provided together with Nexus artifact repository. It is used to store software updates and Data Integration Job artifacts.

This tool is used for the Software Update feature and its instance holds the talend-updates repository where the updates are retrieved by the user.

It can also be used as a catalog for the Jobs created from Talend Studio or any other Java IDE. For this, two repositories are available: repo-snapshot for development purposes and repo-release for production purposes.

This instance is embedded in the .zip file of Talend Administration Center Web application and it allows you to store artifacts designed from Talend Studio or any other Java IDE and ready to be deployed and executed in an execution server. For more information, see the Talend Administration Center User Guide.

So when unzipping Talend Administration Center zip file, you will find an archive file called Artifact-Repository-Nexus-VA.B.C.D.E containing a ready-to-be-used Nexus artifact repository.

Nexus is based on Sonatype Nexus. For more information on how to use it, see Artifact Repository and Sonatype Nexus's documentation on http://www.sonatype.org/nexus.

To install and configure Nexus artifact repository, see the following procedures:

For more information on how to configure Nexus in Talend Runtime, see Configuring Nexus in Talend Runtime

Installing the Nexus artifact repository

To install the Nexus artifact repository, proceed as follow:

  1. Unzip the Artifact-Repository-Nexus-VA.B.C.D.E archive file in a dedicated folder.

  2. Add the execution rights to the relevant files using the following commands:

    $ chmod 755 bin/nexus
    $ chmod 755 bin/jsw/linux-ppc-64/*
    $ chmod 755 bin/jsw/linux-x86-32/*
    $ chmod 755 bin/jsw/linux-x86-64/*
    $ chmod 755 bin/jsw/macosx-universal-32/*
    $ chmod 755 bin/jsw/macosx-universal-64/*
  3. Launch the Nexus instance using the command nexus.sh console. If you installed Nexus as a service, run the start command to launch it.

  4. Log in the Nexus Web application (default login information: admin/Talend123). Make sure you have the relevant rights to access the releases and snapshots repositories.

Configuring the Software Update repository in Talend Administration Center

Once you installed the Nexus artifact repository and started it, you can configure it to use Talend Software Update.

Once you have launched and configured the Software Update repository, go to the Configuration page of Talend Administration Center and fill in the following information in the Software Update group:

  • Talend update url : Location URL to the Talend remote repository from which software updates are retrieved, this field is filled by default.

  • Talend update username et Talend update password : Type in the credentials of the software update repository user that you received from Talend.

  • Local repository url : Type in the location URL to the repository where software updates are stored. By default, it is http://localhost:8081/nexus/.

  • Local deployment username and Local deployment password : Type in the credentials of the user with deployment rights to the local repository. By default, it is talend-updates-admin/talend-updates-admin.

  • Local reader username and Local reader password : Type in the credentials of the user with read rights to the local repository. By default, no credentials are required but you are free to define them if you want to disable public access to the repository.

  • Local repository ID : Type in the ID of the repository in which software updates are published. By default, it is talend-updates.

In the Software Update page of Talend Administration Center, you can now see the versions and patches available and download them according to your needs.

Configuring Nexus in Talend Administration Center

Once you have launched the Nexus artifact repository, go to the Configuration page of Talend Administration Center and do the following:

  • Fill in the following information in the Artifact Repository node:

    • Artifact repository type: select NEXUS.

    • Nexus url: Type in the location URL to your Nexus Artifact repository, http://localhost:8081/nexus/ for example.

      Note

      "http://localhost:8081/nexus" is only given as example. Depending on your configuration, you may have to replace <localhost> with the IP address of the Web server and <8081> with the port that is used for your repository instance.

    • Nexus username: Type in the name of the repository user with Manager role. By default, it is admin.

    • Nexus password: Type in the password of the repository user with Manager role. By default, it is Talend123.

    • Nexus Default Release Repo: Type in the Nexus artifact repository's Release repository name. By default, it is releases.

    • Nexus Default Snapshot Repo: Type in the Nexus artifact repository's Snapshot repository name. By default, it is snapshots.

    • Nexus Default Group ID: Type in the name of the group in which to publish your Jobs, Service and Route artifacts. By default, it is org.example.

From the Job Conductor pages of Talend Administration Center, you can retrieve all the artifacts published in the two repositories to configure their execution in your execution server. For more information, see the Talend Administration Center User Guide.

Installing and configuring your JobServers

The execution servers allow you to execute the Jobs (processes) developed with Talend Studio from the Talend Administration Center web application.

Note that you can also use Talend Runtime servers to deploy and execute Jobs tasks if these Jobs are linked with Services or Routes, but these servers are especially used to deploy and execute Services, Routes, or even generic OSGi features when you are using our service-oriented products. For more information about the installation of Talend Runtime, see Installing Talend Runtime.

To install and configure your JobServers, see the following procedures:

Installing your JobServers

In order to install your JobServers, proceed as follows:

Unzip the archive file

  1. First select the servers that will be used to execute the Jobs developed with Talend Studio.

  2. Then, on each server, uncompress the archive file containing the JobServer application matching your version of Talend Studio.

    The archive file name for example reads: Talend-JobServer-YYYYMMDD_HHmm-VA.B.C.zip

  3. In the uncompressed file you need to configure the file TalendJobServer.properties that you can find in the directory <root>/conf/ where <root> is the JobServer path.

    For example, if you want to change the directory where the JobServer stores its data, change the org.talend.remote.jobserver.commons.config.JobServerConfiguration.ROOT_PATH parameter.

  4. Modify the installation directory of JobServer and check that the 8000, 8001 and 8888 ports are available.

Enable user authentication

  1. To enable user authentication on JobServer, you need to define one or more lines of username and password pairs in the file users.csv that you can find in the directory <root>/conf/ where <root> is the JobServer path.

  2. In the directory you have unzipped, you will find the files start_rs.sh and stop_rs.sh that will let you respectively start and stop the Job server.

Note

You may need to change the java.library path in order to load the correct native library for your system. In this case, adapt the variable MY_JSYSMON_LIB_DIR in the script start_rs.sh.

JobServer is an application that allows a system installed on the same network as the Web application to declare itself as an execution server. These systems must obviously have a working JVM. For more information about the prerequisites of JobServer, see Software requirements.

Information about JobServer resources and load balancing :

Once you have declared these execution servers in the Servers page of the Talend Administration Center Web application, their resources (CPU, RAM, etc.) are displayed. For more information on how to do this, see your Talend Administration Center User Guide.

For some operating systems, the CPU information may not be available. You can test your system by setting up the following variable as true:

org.talend.monitoring.jmx.api.OsInfoRetriever.FORCE_LOAD in the file TalendJobServer.properties.

For users working in cluster mode, note that the ranking of servers to be used for load balancing is based on indicators, whose bounds (such as free disk space limits) and weight are defined in the file: monitoring_client.properties which is located in <ApplicationPath>\WEB-INF\lib\org.talend.monitoring.client-A.B.C.jar. These values can be edited according to your needs. For more information, see Configuring the indicators which determine which server to be used for load balancing.

Configuring the JVM for your JobServer (optional)

The JobServer application provided by Talend allows you to choose another JVM than the one used by default to launch your Jobs.

To change the Job launcher path, proceed as follows:

  1. Go to the directory <root>/conf/, where <root> is the JobServer path, and open the TalendJobServer.properties file to edit it.

  2. In the line dedicated to the Job launcher path, add the path to your java executable after the equal sign.

    # Set the executable path of the binary which will run the job, for example: /usr/bin/java/java or "c:\\Program Files\\Java\\bin\\java.exe"
    org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_LAUNCHER_PATH=C:\Program Files\Java\jre1.8.0_65\bin\java.exe

    Note

    The use of quotes is only necessary when your path contains spaces, as shown in the capture. Otherwise, type in the path without quotes.

  3. Save your changes and close the file.

The next time you launch JobServer, the java executable used will be the one you have previously set in the TalendJobServer.properties file.

Configuring the SSL Keystore (optional)

You are also able to choose another Keystore if needed.

To override the existing Keystore file, you have to:

  • generate a new Keystore with the utility tool called Keytool (Key and Certificate Management Tool);

  • set the new Keystore location;

  • enable the SSL Keystore at server side.

To generate a Keystore

  1. Open a command prompt and change directory to <root>/keystores where <root> is the JobServer path.

  2. Type in the following:

    keytool -genkey -keystore <myKeystoreName> -keyalg RSA

    where <myKeystoreName> refers to the name of the Keystore you are creating.

  3. Enter the password for your Keystore twice, then enter the other optional information, such as your name, the name of your organization, your state etc., if needed.

  4. Type in yes to confirm your information.

  5. Type in the password you have previously defined. The new Keystore file has been created in <root>/keystores.

To set the location of the new Keystore

To set the new Keystore location, you can either edit the JAVA_OPTS environment variable or edit the launching script (start_rs.sh) of the Jobserver.

  1. To edit the JAVA_OPTS environment variable, add:

    -Djavax.net.ssl.keyStore=/<myDirectory>/<myKeystore>

    -Djavax.net.ssl.keyStorePassword=<myPassword>

    to your JAVA_OPTS environment variable, where <myDirectory> is the installation directory of your Keystore, <myKeystore> is the name of your Keystore and <myPassword> is the password you have previously defined for your Keystore.

    Note

    If you have not created the JAVA_OPTS environment variable yet, you have to create it before completing this procedure.

    OR

  2. To edit the launching script, open the start_rs.sh to edit it.

  3. As shown in the capture, add

    -Djavax.net.ssl.keyStore=/<myDirectory>/<myKeystore>

    -Djavax.net.ssl.keyStorePassword=<myPassword>

    to the JVM arguments location, where <myDirectory> is the installation directory of your Keystore, <myKeystore> is the name of your Keystore and <myPassword> is the password you have previously defined for your Keystore.

To configure the service

Now you just have to enable Secure Sockets Layer as described in Enabling the SSL encryption in Talend Runtime.

Configuring user impersonation for JobServer

The Talend Administration Center web application allows you to run tasks as different UNIX system users, through the Run As option. To avoid errors when starting the task on the server, you need first to:

  • give specific permissions to some server directories.

  • give necessary authorizations to the directories and files created by the JobServer by configuring the umask.

  • define the Operating System users allowed to run tasks from the server.

For more information on this feature, see the Talend Administration Center User Guide.

Setting the server directory permissions

Prerequisite: If you have already started Jobs from this server, it is recommended to remove the directory <jobserver_path>/TalendJobServerFiles to avoid unexpected authorizations on already deployed Jobs or cached files.

  1. Add each user allowed to run tasks (for example, user called subuser) to the 'root' group as well as to the group of the user who owns the parent directories of JobServer (for example, group of the user called myuser), such as:

    > sudo usermod -a -G myuser_group subuser
    > sudo usermod -a -G root subuser
  2. Give the permissions execute to myuser_group in the following directories by executing the command chmod g+rx /<directory_path>:

    /DIRECTORY_1

    /DIRECTORY_1/DIRECTORY_2

    /DIRECTORY_1/DIRECTORY_2/Talend-JobServer

    /DIRECTORY_1/DIRECTORY_2/Talend-JobServer/cache

    /DIRECTORY_1/DIRECTORY_2/Talend-JobServer/cache/lib

    /DIRECTORY_1/DIRECTORY_2/Talend-JobServer/repository

    Note that the read authorization for the group is only required for deployed files.

Configuring the umask of the user which launches the JobServer

  • Set the user profile with the following umask: umask u=rwx,g=rx,o=

    which is the same as umask 0027

    This configuration will create:

    • directories with group authorization equal to r-x

    • files with group authorization equal to r--

    • no authorizations for others

Defining the list of users allowed to run tasks as different users

  1. Open the following file: <jobserver_path>/conf/TalendJobServer.properties

  2. Edit the org.talend.remote.jobserver.server.TalendJobServer.RUN_AS_WHITELIST value and add all the users you need.

    Note that spaces as well as commas are valid separators for user name values in this file.

Starting the JobServer

  • Start the JobServer using this command: sudo sh start_rs.sh

    Note that if you do not use sudo, the Jobs will hang because a password will be required at JobServer side.

Disabling some SSL ciphers (optional)

SSL ciphers are encryption algorithms that are used to establish a secure communication. Some cipher suites offer a lower level of security than others, and you may want to disable these ciphers. To do so:

  1. Go to the directory <root>/conf/ and open the TalendJobServer.properties file.

  2. Add to the following parameter the list of ciphers that you want to disable:

    org.talend.remote.jobserver.server.TalendJobServer.DISABLED_CIPHER_SUITES

    Here is the list of the ciphers supported by JobServer:

    TLS_KRB5_WITH_3DES_EDE_CBC_MD5
    TLS_KRB5_WITH_RC4_128_SHA
    SSL_DH_anon_WITH_DES_CBC_SHA
    TLS_DH_anon_WITH_AES_128_CBC_SHA
    TLS_DHE_RSA_WITH_AES_128_CBC_SHA
    SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
    SSL_RSA_EXPORT_WITH_RC4_40_MD5
    SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
    TLS_KRB5_WITH_3DES_EDE_CBC_SHA
    SSL_RSA_WITH_RC4_128_SHA
    TLS_KRB5_WITH_DES_CBC_MD5
    TLS_KRB5_EXPORT_WITH_RC4_40_MD5
    TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5
    SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
    TLS_KRB5_EXPORT_WITH_RC4_40_SHA
    SSL_DH_anon_EXPORT_WITH_RC4_40_MD5
    SSL_DHE_DSS_WITH_DES_CBC_SHA
    TLS_KRB5_WITH_DES_CBC_SHA
    SSL_RSA_WITH_NULL_MD5
    SSL_DH_anon_WITH_3DES_EDE_CBC_SHA
    TLS_RSA_WITH_AES_128_CBC_SHA
    SSL_DHE_RSA_WITH_DES_CBC_SHA
    TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA
    SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA
    SSL_RSA_WITH_NULL_SHA
    TLS_KRB5_WITH_RC4_128_MD5
    SSL_RSA_WITH_DES_CBC_SHA
    TLS_EMPTY_RENEGOTIATION_INFO_SCSV
    SSL_RSA_EXPORT_WITH_DES40_CBC_SHA
    SSL_DH_anon_WITH_RC4_128_MD5
    SSL_RSA_WITH_RC4_128_MD5
    TLS_DHE_DSS_WITH_AES_128_CBC_SHA
    SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
    SSL_RSA_WITH_3DES_EDE_CBC_SHA

Installing Talend Runtime

Note

According to the solution you have subscribed to, Talend recommend you to use:

  • JobServer, if you have subscribed to a Data Integration solution;

  • Talend Runtime, if you have subscribed to both a Data Integration and a ESB solution.

    However, if you are willing to use both Talend Runtime and JobServer on the same machine, you are required to change the port numbers because, by default, both servers are using the same ports.

Talend Runtime is an OSGi container, based on Apache Karaf, allowing you to deploy and execute various components and applications inside its deploy folder.

To install and configure your Talend Runtime, see the following procedures:

Installing the Talend Runtime containers

You need now to define on which server(s) you will install Talend Runtime.

  1. First select the servers that will be used for the execution.

  2. Then, on each server, unzip the archive file containing the Talend Runtime application matching your release version of Talend.

    The archive file name for example reads: Talend-Runtime-V6.2.0.zip

  3. In the unzipped file you might need to configure the files org.ops4j.pax.web.cfg to change the HTTP listening port.

  4. To launch Talend Runtime, browse to the bin directory and run the trun file.

Now we simply have to declare these runtime instances in the Web application and their resources (CPU, RAM, etc.) should become available. To do this:

  1. Go to the Servers page of Talend Administration Center.

    Warning

    Only users that have Operation Manager role and rights can have a read-write access to this page. For more information on access rights, see your Talend Administration Center User Guide. So, you have to connect to Talend Administration Center as an Operation Manager to be able to configure your servers.

  2. And define the server as follows:

    Label

    TestingServer

    Description

    Type in the description of server.

    Host

    localhost

    Command port

    8000

    File transfer port

    8001

    Monitoring port

    8888

    Timeout on unknown status(s)

    120

    Username

    Type in the username for user authentication to access a Job server.

    Password

    Type in the password for user authentication to access a Job server.

    Active

    Select/clear the check box to activate/deactivate this server

    Use SSL

    Select/clear the check box to use or not your own SSL Keystore to encrypt the data prior to transmission.

    For more information about how to enable SSL, see Enabling the SSL encryption in Talend Runtime.

    Talend Runtime

    By default, servers created are Job servers.

    To deploy and execute your Jobs tasks into Talend Runtime, select the Talend Runtime check box. The following fields will display: Mgmt-Server port, Mgmt-Reg port, Admin Console port and Instance.

    Mgmt-Server port RMI Server Port (44444 by default). This field is mandatory.
    Mgmt-Reg port RMI Registry Port (1099 by default). This field is mandatory.
    Admin Console port Port of the Administration Web Console (8040 by default). This field is mandatory and allows to activate the Admin server button allowing you to access the Administration Web console.
    Instance Type in the name of the container instance in which you will deploy and execute your Jobs tasks, trun by default.

    This corresponds to the configuration of a Talend Runtime on the system that hosts the Web application. For any other system, the Host field should contain the IP address of the system. Check also that the ports 8000, 8001 and 8888 are available. These ports must be the same as defined in the TalendJobServer.properties defined above. Note that if no username and password pairs are defined in the file users.csv in the directory <root>/conf/ where <root> is the JobServer path, then you do not have to set the Username and the Password.

  3. Click the Servers page again so that the Talend Runtime servers appear with their properties.

Enabling the SSL encryption in Talend Runtime

The execution servers provided by Talend allows you to encrypt data prior to transmission via an existing SSL Keystore. To enable Secure Sockets Layer (SSL) at server side in order to establish an encrypted link between the Jobserver and its clients, proceed as follows:

  1. If you want to configure Talend Runtime, go to the etc directory and open the org.talend.remote.jobserver.server.cfg file to edit it.

    If you want to configure the JobServer, go to the <root>/conf/ directory and open the TalendJobServer.properties file to edit it.

  2. Edit the following line

    org.talend.remote.jobserver.server.TalendJobServer.USE_SSL=false

    and replace false with true.

    The next time you launch your execution server, the SSL protocol will be used to secure the communication between servers and clients.

Note

From Talend Administration Center, you have to select the Use SSL check box to enable the encryption.

Configuring Nexus in Talend Runtime

The default Nexus artifact repository URL is described in the etc/org.ops4j.pax.url.mvn.cfg file.

If your artifact repository has been installed on another URL, edit the org.ops4j.pax.url.mvn.repositories part of the file.

Installing the Talend Activity Monitoring Console web application

The Talend Activity Monitoring Console Web application we are talking about in this section is available in Talend Administration Center. However, the AMC is also a perspective available in Talend Studio. For more information about this perspective, see the Talend Activity Monitoring Console User Guide.

For more information on Talend Activity Monitoring Console, see Talend Activity Monitoring Console log database.

To install the Talend Activity Monitoring Console, follow these procedures:

Deploying the Talend Activity Monitoring Console Web application
  1. Unzip the Talend-AMC_Web-YYYYYYYY_YYYY-VA.B.C.zip archive file containing the amc.war file on the same machine as Talend Administration Center Web application.

  2. Paste the amc.war file in the same webapps folder as the one where Talend Administration Center is located, for example <TomcatPath>/webapps/.

  3. Restart your Web application server.

If you want to install Talend Activity Monitoring Console on other Web application server than the one where Talend Administration Center is installed, follow the same procedure as Deploying Talend Administration Center on an application server.

If you do not want Talend Activity Monitoring Console to use the default H2 database, see Installing database drivers in your Web application server.

Configuring the Talend Activity Monitoring Console in Talend Administration Center

In the Talend Administration Center web application, you have to set up the link to the Talend Activity Monitoring Console.

  • To do so, specify the following information on the Monitoring group of the Configuration page:

    • AMC url: type in the URL address of the Talend Activity Monitoring Console application, http://localhost:8080/amc/rap?startup=amc for example.

      Warning

      http://localhost:8080/amc/rap?startup=amc is only given as example. Depending on your configuration, you may have to replace <localhost> with the IP address of the Web server and <8080> with the actual port of the <AMC>.

    You also have to configure three database tables that will store statistics and log information.

    For more information about this application, see the Talend Activity Monitoring Console User Guide.

Installing and configuring the Drools Business Rules Management System (BRMS)

To install and configure the Drools Business Rules Management System (BRMS) on your machine, follow these procedures:

Installing manually the Drools Business Rules Management System (BRMS)

To manually install the Drools Business Rules Management System (BRMS), proceed as follows:

  1. Download the Talend-BRMS-YYYYMMDD_HHmm-VA.B.C.zip file and unzip it.

  2. Stop your Tomcat server.

  3. Go to the Talend-BRMS-Webapp-A.B.C folder.

  4. Copy the talend-brms-A.B.C.war file to <TomcatPath>/webapps.

  5. Unzip the WorkbenchInstallationResources.zip file.

  6. Copy the content of the WorkbenchInstallationResources/lib folder to <TomcatPath>/lib.

  7. Got to the WorkbenchInstallationResources/conf folder.

  8. Copy the btm-config.properties and resources.properties files to <TomcatPath>/conf.

  9. Create a backup copy of your <TomcatPath>/bin/setenv.sh file.

  10. Copy the WorkbenchInstallationResources/bin/setenv.sh file to <TomcatPath>/bin.

  11. Restart Tomcat to deploy the Drools Business Rules Management System (BRMS).

Configuring the Drools Business Rules Management System (BRMS) in Talend Administration Center

Before being able to use Drools BRMS in Talend Administration Center web application, you will need to configure its URL in the Configuration page. For more information about configuring the Web application, first read Installing and configuring Talend Administration Center.

Then after you access Talend Administration Center's Configuration page, set Drools URL, http://10.42.10.84:8080/kie-drools-wb/ for example.

For more information about the actual data contained on the Configuration page of Talend Administration Center, check out the Talend Administration Center User Guide.

When Drools Guvnor URL is correctly set up, the application can be accessed through the menu tree view of Talend Administration Center:

Note

If you have used Drools with older Talend product releases and you do not want to use the latest Drools version installed with Talend Administration Center, you must upgrade your Drools repository before being able to use it with the current release. For more information, check the article Migrating Drools repository from version 5.x to version 6.0.