Configuring parameters according to your system - 6.3

Talend Administration Center User Guide

EnrichVersion
6.3
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend Real-Time Big Data Platform
task
Administration and Monitoring
Deployment
EnrichPlatform
Talend Administration Center

The parameters described in the following sections do not follow the same order as the application interface (which is alphabetical) because they are sorted in order of importance, from essential to optional. It it thus recommended to follow this order when configuring your application.

Setting up SVN or Git parameters

Your project metadata is either stored in Subversion or Git. If you are using a SVN/Git repository, you can modify the location URL, user and password from Talend Administration Center.

When you will connect to a remote repository in the Studio, you will have to enter the URL of your Talend Administration Center in the Web-app URL field so that the Studio can retrieve these parameters.

Prerequisites:

  • the repository is on the Subversion/Git server and you have copied its URL.

    For Git users:

    • it is recommended to check via git bash that the repository is accessible, that you can push your changes on it and, if you use SSH, that the host is known.

    • all branches/tags are visible for all projects on the same Git repository. Therefore, if you create some branches/tags for one project, all other projects on the same Git repository will have the same list of branches and tags.

    • if you want to create projects in multiple Git repositories (not only the one set on the Configuration page), you need to add a parameter in a configuration file. For more information, see the Talend Installation Guide.

  • the SVN/Git username and password correspond to an existing user in Subversion/Git.

    For Git users: note that the SSH key passphrase feature is not supported in this release of Talend Administration Center.

  1. On the Menu tree view, click Configuration.

  2. Click the Svn or Git group to display its parameters.

    The following parameters are editable according to your use:

    Parameter

    Value

    Branches whitelist

    Select True or False in the list.

    This function allows you to filter the project on the defined SVN/Git branches or tags in order to reduce the use of disk resources and improve performances. For more information, see To create a Git or SVN Branch whitelist.

    Note

    This configuration does not filter the branches that are available for this project in Talend Studio. All branches will still be shown in Talend Studio.

    Server location url /

    Git server url

    Type in the location URL to your Subversion/Git server repository.

    Username

    Type in the name of the Subversion/Git user.

    Password

    Type in the password of the Subversion/Git user.

    Commit Log Pattern

    Define the Subversion/Git commit log pattern according to your log format convention. For more information about commit log patterns, see the Talend Studio User Guide.

    The log pattern will be automatically applied to all commit logs, except user logs if the Svn user log option is activated (SVN only). For more information about this option, see Adding a project.

    Example: the pattern start {0} end will add the word start before the log message body and end after the log message body.

    The default pattern is {0}, namely no leading or trailing information is added to any log message.

    These parameters are usually set up when installing the product, so, for more information, see the relevant section of Talend Installation Guide.

Setting up the user library location

Some specific third-party Java libraries or database drivers (.jar files), known as external modules, can be required by some components or by some connection wizards. Due to license restrictions, may not be able to ship certain external modules within Talend Studio but allows you to download, install and store them in a shared repository (either SVN, or Nexus which is delivered with the Talend Administration Center archive file).

Note that, if you are working in collaborative mode and you do not fill these parameters, the external libraries that you have downloaded at Studio start-up will not be shared and thus the other users of Talend Administration Center will not be able to use CommandLine to generate their Jobs. Additionally, if you migrate to a more recent version of Talend Administration Center, those required libraries will not be backed up.

Finally, note that for users with Designer role to have access to user libraries configuration, they need to have the corresponding Configuration visualization right selected on the Rights management page. For more information, see Managing rights associated with roles.

For more information about these external libraries, see the Talend Installation Guide.

  1. On the Menu tree view, click Configuration.

  2. Click the User Libraries group to display its parameters.

    The following parameters are editable according to your use:

    Parameter

    Value

    Libraries Repository

    Select the repository where the external libraries will be stored. It is either NEXUS (recommended) or SVN.

    Library location url

    Type in the location URL to the Nexus repository/SVN directory where the external libraries downloaded at Studio start-up are stored. By default, it is http://localhost:8081/nexus on Nexus.

    IMPORTANT:

    • As the Studio will connect to this repository via the URL you specify in this field, you should replace <localhost> with the host name of the machine where Nexus is installed and make sure that the URL is accessible from the Studio and/or CommandLine.

    • If you use Subversion, this directory must be created outside the repository in which your projects are stored.

    Library username

    Type in the name of the Nexus/Subversion user which has access to the libraries repository. By default, it is talend-custom-libs-admin on Nexus.

    Library password

    Type in the password of the Nexus/Subversion user which has access to the libraries repository. By default, it is talend-custom-libs-admin on Nexus.

    Repository ID Type in the repository ID in which libraries will be stored. By default, it is talend-custom-libs on Nexus.

Setting up the CommandLines' parameters

From Talend Administration Center, you can set the parameters of two Commandlines: host, port, archive path and the user component path. To ensure failover, the second CommandLine is used when the first one is down. To set the parameters of the CommandLines, complete the following:

  1. On the Menu tree view, click Configuration.

  2. Click the CommandLine group to display the parameters.

    The following parameters are editable as required:

    Parameter

    Value

    Host

    Type in the IP address of the CommandLine.

    Port

    Type in the port number on which the CommandLine will be queried.

    Job generation folder

    Type in the path to the folder where you want your jobs to be generated.

    User component path

    If you are using user components, copy them in a folder on your CommandLine server, then type in the path to this folder in this field.

    If you are not using user components, leave this field blank.

    Version

    This field allows you to check whether the CommandLine version is the same as the database version.

    Note

    A message will display in the CommandLine header page, indicating to which CommandLine you are connected and if your CommandLines are down.

Setting up the Job Conductor parameters

When deploying and executing tasks in Talend Administration Center, multiple files are created and stored locally. These files include the Job archives, execution, recovery and deployment logs. From Talend Administration Center, you can modify the path to those log files. To do so, proceed as follows:

  1. On the Menu tree view, click Configuration.

  2. Click the Job Conductor group to display its parameters.

The parameters are the following:

Parameter

Value

Generated jobs folder

Type in the path to the folder that has been created to store the Job execution archives, grouped by execution tasks. This folder can not be removed. Otherwise, the tasks can not be executed.

Task logs folder

Type in the path to the folder that has been created to store the Job execution logs, grouped by execution tasks. The logs in this folder are used to be downloaded as execution log on UI and/or for manual check of the Job execution. It is recommended not to remove this folder either.

Number of execution logs

Enter how many execution log files you want to store in each folder created for the Jobs. This value prevails over the value of the 'Storage period for log files' field. Note that this limitation applies only to execution log files. The folders also contain resuming log files as well, therefore the file count in the folders is twice the value of the limit set in the Number of execution logs parameter.

Storage period for log files

Enter how long the log files of all Jobs will be kept, in days, before they are removed.

Maximum number of generated jobs

Enter how many generated Job archives you want to store in the dedicated folder. This value prevails over the value of the 'Storage period for generated jobs' field.

Storage period for generated jobs

Enter how long the generated Job archives will be kept, in days, before they are removed.

Quartz servlet

Shows the status of the Job Conductor. It can be Initialized or Not initialized.

Setting up the ESB parameters

If your license includes the ESB option, you need to configure several parameters to make the best use of the corresponding features.

Setting up the ESB Service Locator and Service Activity Monitoring parameters

From Talend Administration Center, you can set the parameters of Service Locator and Service Activity Monitoring.

  • Service Locator provides service consumers with a mechanism to discover service endpoints at runtime.

  • Service Activity Monitoring is a monitoring server for your services.

  1. On the Menu, click Configuration.

  2. Click the ESB Service Locator and SAM group to display the parameters.

    The following parameters are editable as required:

    Parameter

    Value

    Service Locator Server(s)

    Type in the URL to the Service Locator server(s) and the credentials to access the server(s) when authentication is enabled. If there are multiple servers, separate them with a comma, for example: 192.168.0.1:2181,192.168.0.2:2181;user=tesb;password=tesb

    Service Activity Monitoring Server

    Type in the URL to the Service Activity Monitoring server.

    Authentication Type

    The authentication required on the Service Activity Monitoring server end. The types available are None, and Basic.

    Username

    Type in the username to connect to the Service Activity Monitoring server if the authentication type is Basic.

    Password

    Type in the password to connect to the Service Activity Monitoring server if the authentication type is Basic.

Setting up the ESB Identity and Access Management

To be able to authorize the Resources to the Roles and Users created in the Talend Identity Management Service from the Authorization page, you need to have properly installed and configured it, and set up the XACML Policy registry.

For more information about the installation of Talend Identity Management Service, see the Talend Installation Guide. For how to use Talend Identity Management Service, and how to set up the XACML Policy registry, refer to Talend ESB Infrastructure Services Configuration Guide.

  1. On the Menu tree view, click Configuration.

  2. Click the ESB Identity and Access Management group to display its parameters.

    The parameters are the following:

    Parameter

    Value

    Identity Service URL

    Type in the location URL to your Identity Service. The default is http://localhost:8080/syncope-console.

    Identity Service Rest URL

    Type in the Rest interface URL of your Identity Service for the Administration Center to get the user data. The default is http://localhost:8080/syncope/cxf.

    Authentication Type

    The authentication type is Basic by default.

    Identity User

    Type in the username to connect to your Identity Service. The default is admin.

    Identity Password

    Type in the password to connect to your Identity Service. The default is password.

    Authorization (PAP) URL

    The URL of the XACML Policy Registry to retrieve policies from. The default is http://localhost:8040/services/XacmlRegistryAtom.

Setting up the ESB Service Registry

To be able to manage services, policies and their relations in the Service Registry page, you need to have properly set up the registry and the Service Registry feature. For how to set up the Service Registry, refer to Talend ESB Infrastructure Services Configuration Guide.

  1. On the Menu tree view, click Configuration.

  2. Click the ESB Service Registry group to display its parameters.

    The parameters are the following:

    Parameter

    Value

    Service Registry API

    The URL of the Service Registry interface. The default is http://localhost:8040/services/registry/admin.

    Authentication Type

    The authentication required on the Service Registry server end. The types available are None, and Basic.

    Username

    Type in the username to connect to the Service Registry if the authentication type is Basic.

    Password

    Type in the password to connect to the Service Registry if the authentication type is Basic.

    Lookup Service

    The URL of the Lookup Service interface. The default is http://localhost:8040/services/registry/lookup.

    Authentication Type

    The authentication required by the Lookup Service. The types available are None, and Basic.

    Username

    Type in the username to connect to the Lookup Service if the authentication type is Basic.

    Password

    Type in the password to connect to the Lookup Service if the authentication type is Basic.

Setting up the ESB Provisioning Service

To be able to distribute features and resources, and configure your Talend Runtimes more easily via the Provisioning page, you need to have properly set up the Provisioning Service feature. For how to set up the Provisioning Service, refer to Talend ESB Infrastructure Services Configuration Guide.

  1. On the Menu tree view, click Configuration.

  2. Click the ESB Provisioning Service group to display its parameters.

    The parameters are the following:

    Parameter

    Value

    Provisioning Service API

    The URL of the Provisioning interface. The default is http://localhost:8040/services/provision/admin.

    Authentication Type

    The authentication required on the Provisioning Service server end. The types available are None, and Basic.

    Username

    Type in the username to connect to the Provisioning Service if the authentication type is Basic.

    Password

    Type in the password to connect to the Provisioning Service if the authentication type is Basic.

Setting up the Artifact Repository parameters

To retrieve all the artifacts published in the artifact repository and configure their execution, you have to connect your Talend Administration Center to the Nexus artifact repository.

For more information about how to install the artifact repository, see the Talend Installation Guide.

Nexus is based on Sonatype Nexus. For more information on how to use it, see Sonatype Nexus's documentation on http://www.sonatype.org/nexus.

  1. On the Menu tree view, click Configuration.

  2. Click the Artifact Repository group to display its parameters.

    The parameters are the following:

    Parameter

    Value

    Artifact repository type

    Select the type of artifact repository (NEXUS). This artifact repository is embedded within the Talend Administration Center archive.

    For more information on how to configure this artifact repository in the Studio and in Talend Runtime, see the Talend Installation Guide.

    Nexus/Artifact repository url

    Type in the location URL to your repository, http://127.0.0.1:8081/nexus by default.

    Nexus/Artifact repository username

    Type in the username to connect to your repository. By default, it is admin for the Nexus artifact repository.

    Nexus/Artifact repository password

    Type in the password to connect to your repository. By default, it is Talend123 for the Nexus artifact repository.

    Nexus/Artifact Default Release Repo

    Type in the name of the repository into which to publish the Release version of your artifact item by default. By default, it is releases for the Nexus artifact repository.

    Nexus/Artifact Default Snapshot Repo

    Type in the name of the repository into which to publish the Snapshot version of your artifact item by default. By default, it is snapshots for the Nexus artifact repository.

    Nexus/Artifact Default Group ID

    Type in the name of the group in which to publish your artifact item by default.

Setting up the link to the Monitoring modules

From Talend Administration Center, you can access the monitoring modules that allow you to monitor your Jobs and projects (Activity Monitoring Console page) and to display the logged events (Logging page).

The Talend Activity Monitoring Console application is integrated both within the Studio and within the Web application. For more information about this application and its operation, see the Talend Activity Monitoring Console User Guide.

To set up the parameters of these modules, proceed as follows:

  1. In the Menu tree view, click Configuration.

  2. Click the Monitoring node to display the parameters.

The following parameter is editable according to your needs:

Parameter

Value

AMC url

Type in the URL address of the Activity Monitoring Console application, http://localhost:8080/amc for example.

Warning

"http://localhost:8080/amc" is only given as example. Depending on your configuration, you may have to replace <localhost> with the IP address of the Web server application and <8080> with its actual port.

Kibana url

Type in the URL address of the Kibana application, http://localhost:8080/kibana for example.

Warning

"http://localhost:8080/kibana" is only given as example. Depending on your configuration, you may have to replace <localhost> with the IP address of the Web server application and <8080> with its actual port.

Setting up the Logging parameters

In Talend Administration Center, there are two types of collected logs:

  • the technical logs, that gather technical events like the status of connections and servers, and so on.

  • the business logs, that gather all users' actions, like the creation, connection, deletion, and edition of a user, a Job and so on.

Those logs are collected by Logstash and displayed on the Logging page thanks to the Kibana Web application whose URL needs to be configured in the Monitoring node of the Configuration page as explained in Setting up the link to the Monitoring modules. For more information about the extended logging module, see Displaying log events.

Note

Kibana is not supported on IE11.

You can modify the log file paths and threshold. To do so, complete the following:

  1. On the Menu tree view, click Configuration.

  2. Click the Logging group to display its parameters.

The parameters are editable according to your use:

Parameter

Value

Technical log file path

Type in the path to the technical log file of Talend Administration Center.

The specified folder must exist. For example, if you enter c:/logs/technical.log, the technical.log file will be created in the existing c:/logs folder. The log file contains technical information logged by classes on the Talend Administration Center runtime, for example task execution start information and so on. You can safely remove the log file.

Note

If no path has been set or if the path is invalid, the technical logs will be appended in Tomcat logs.

Technical log threshold

Select the level of logs you want to append between DEBUG, INFO, WARN, and ERROR from the drop-down list.

Business log file path

Type in the path to the business log file of Talend Administration Center.

The specified folder must exist. For example, if you enter c:/logs/business.log, the business.log file will be created in the existing c:/logs folder. The log file contains business information logged by classes on the runtime, for example task or user create information and so on. You can safely remove the log file.

Note

If no path has been set or if the path is invalid, the business logs will not be recorded.

Logstash host and port

Type in the host and port corresponding to the Logstash instance. By default, it is localhost:8050. This field automatically turns green when the Talend Logserver is launched.

You can also do this manually by editing the log4j.xml file. For information related to this Log4j file, see the relevant section in the Talend Installation Guide and for more information on how to enable and customize the log4j feature from the Studio, see the Talend Studio User Guide.

Setting up an LDAP(S) protocol

If you use an LDAP or LDAPS directory to list your resources, in particular your personnel, you can use this directory to authenticate the Administration Center users. For more information on how to configure the keystore for LDAPS, see the Talend Installation Guide.

  1. On the Menu, click Configuration and then click the LDAP group to display the parameters. Note that the data displayed in the capture below is used as an example, and that the field values must correspond to the values defined in your LDAP configuration.

  2. Set the parameters as the following:

    Parameter

    Value

    UseLDAPAuthentication

    Select true to activate LDAP authentication or false to deactivate it from the drop-down list.

    Enable LDAPS

    Select true to activate LDAP over SSL (LDAPS) authentication or false to deactivate it from the drop-down list. For more information on how to configure the keystore for LDAPS, see the Talend Installation Guide.

    Host

    Type in the IP address and the host of the LDAP server.

    Port

    Type in the port of the LDAP server.

    Ldap principal DN prefix

    Type in the login of the user who has the right to search on the specified root or branch.

    Admin password

    Type in the password of the user who has the right to search on the specified root or branch.

    Login field

    Type in the name of the attribute representing the data that corresponds to the user's login.

    The name must correspond to one of the LDAP fields defined in your LDAP configuration and the value must not be empty.

    Email field

    Type in the name of the attribute representing the data that corresponds to the user's Email.

    The name must correspond to one of the LDAP fields defined in your LDAP configuration and the value must not be empty.

    Firstname field

    Type in the name of the attribute representing the data that corresponds to the user's Firstname.

    The name must correspond to one of the LDAP fields defined in your LDAP configuration and the value must not be empty.

    Lastname field

    Type in the name of the attribute representing the data that corresponds to the user's Lastname.

    The name must correspond to one of the LDAP fields defined in your LDAP configuration and the value must not be empty.

    SvnLogin field /

    GitLogin field

    Type in name of the attribute representing the data that corresponds to the user's SVN or GIT login. This field must correspond to the field defined in your LDAP configuration.

    If you use the LDAP system to handle the SVN and Git credentials, the recommended way is to edit these credentials through LDAP and not through the Users page as Talend Administration Center will automatically retrieve the changes done on these credentials.

    SvnPassword field /

    GitPassword field

    Type in name of the attribute representing the data that corresponds to the user's SVN or GIT password. This field must correspond to the field defined in your LDAP configuration.

    If you use the LDAP system to handle the SVN and Git credentials, the recommended way is to edit these credentials through LDAP and not through the Users page as Talend Administration Center will automatically retrieve the changes done on these credentials.

    For more information on how to manage encryption of the SVN password in LDAP for Talend Administration Center, see the documentation on Talend Help Center.

Once the LDAP authentication is activated, the creation of the user account in the Users page is modified. From now on, only the Distinguished name is required as the other information about the user is retrieved from the LDAP directory. This way, user's login and password used to connect to Talend Administration Center or to Talend Studio are those defined in the LDAP directory.

For more information on the creation of a user account using LDAP authentication system, see How to add a user with LDAP.

Setting up an SMTP protocol

On Talend Administration Center, you can get notified when an event occurs. To realize this feature, you should first set up the parameters of the email system which will send notifications to you.

  1. On the Menu tree view, click Configuration.

  2. Click the SMTP group to display the parameters.

The following parameters are editable according to your use:

Parameter

Value

Use SMTP

Select true to activate SMTP or false to deactivate it.

Host

Type in the IP address of your mail server.

Port

Type in the port of your mail server.

Require SSL

Select true if your email server uses an SSL authentication.

Username

Type in the email address that will be displayed as the sender information in the notification received by the recipients. Whether this field is optional depends on your local configuration of SMTP authentication.

An anonymous user may be used to send notifications. By default, the first admin email address will be used.

Password

If your email server requires an authentication, type in the password corresponding to the mailbox that will send the notification. If your message server is anonymous, this field is optional.

From Address (optional)

Type in the SMTP From email address, admin@company.com for example. By default, if the field is empty, it is the value of the SMTP User Name that will be used.

Once the SMTP parameters are set up, you can configure the notifications that will be sent, from the Notification page. For more information on the notifications, see Managing notifications.

Note

If one of the SMTP parameters is down, a warning will display at the top of the Notification page to inform the user that the notification might not work due to an SMTP server connection error.

Setting up Software Update parameters

From Talend Administration Center, you can edit the Software Update parameters.

The following parameters are editable according to your needs:

Parameter

Value

Talend update url

Location URL to the Talend remote repository from which software updates are retrieved, this field is filled by default.

Talend update username and Talend update password

Type in the credentials of the software update repository user that you received from Talend.

Local repository url

Type in the location URL to the repository where software updates are stored. By default, it is http://localhost:8081/nexus/.

Local deployment username and Local deployment password

Type in the credentials of the user with deployment rights to the local repository. By default, it is talend-updates-admin/talend-updates-admin.

Local reader username and Local reader passwordType in the credentials of the user with read rights to the local repository. By default, no credentials are required but you are free to define them if you want to disable public access to the repository.
Local repository ID Type in the ID of the repository in which software updates are published. By default, it is talend-updates.

These parameters are usually set up when installing manually the software update repository. For more information, see the Talend Installation Guide.

For more information on checking updates via the artifact repository, see Checking for updates.

Enabling the 'Force Logout' option

In order to force the logging out of a user who is already connected to a session with its credentials, you need to enable this option. For more information, see Logging in when your login is already in use.

  1. On the Menu tree view, click Configuration.

  2. Click the Login Page to display the Enable Force logout parameter, and select the true value if you want to enable the option, or false if you want to disable it.

Enabling SSO

You have the possibility to enable unified sign-on and authentication (SSO) to access Talend Administration Center through different Identity provider systems (IdP) and to manage the roles and project types of the application users via the IdPs. For more information on how to implement SSO at the Identity Provider side (Okta, SiteMinder, etc.), see the Talend Installation Guide.

  1. From the configuration page, expand the SSO node.

  2. If SSO has not been enabled yet, select true in the Use SSO Login field.

  3. Click Launch Upload in the IDP metadata field and upload the Identity Provider metadata file you have previously downloaded from the Identity Provider system.

  4. In the Service Provider Entity ID field, enter the Entity ID of your Service Provider (available in the configuration of the IdP), ssologin for example.

  5. Select your Identity Provider System in the corresponding list.

    • If your provider is Okta: enter the corresponding Okta administrator Organization URL, as well as the Okta App Embed link which is the link used to sign into Talend Administration Center from a portal outside of Okta (can be found in Okta configuration).

    • If your provider is SiteMinder: enter the corresponding SiteMinder administrator SiteMinder SSO Service URL, http://<host>/affwebservices/public/saml2sso?SPID=<SPEntityName> for example.

  6. In the Use Role Mapping field, map the application user roles with the roles defined in the Identity Provider system.

    Once you have defined roles at the Identity Provider side, you will not be able to edit the user roles from Talend Administration Center.

    Fill in the role fields with the corresponding SAML role attributes previously set in the Identity Provider system.

    Ex: Talend Administration Center Roles > Administrator = tac_admin; Operation Manager = tac_om

    Ex: Talend Data Preparation Roles > Administrator = dp_admin; Data Preparator = dp_dp

    Ex: Talend Data Stewardship Roles > Data Steward = tds_ds

    The roles set in the Identity Provider will override the roles set in Talend Administration Center.