Configuring Talend logging modules with an external Elastic stack

Talend Real-time Big Data Platform Installation Guide for Linux

EnrichVersion
6.4
EnrichProdName
Talend Real-Time Big Data Platform
task
Installation and Upgrade

The following sections explain how to configure your external Elastic for Data Integration, ESB or MDM.

Configuring Logstash and Elasticsearch in an external Elastic stack

If you want to use your external ELK stack to collect and classify logs from Data Integration, ESB or MDM, you need to perform the following configuration on your Logstash and Elasticsearch:

  1. Create/edit your Logstash configuration file as follows:

    • In the input section, you add the configuration ports for Talend components.

    • In the output section, you configure Elasticsearch as output.

      Note

      The options user and password should match your Elasticsearch credentials.

      If you are not using a true Certificate Authority (CA), set the option ssl_certificate_verification to false.

    input {
      log4j {
        port => 8050
        type => "TAC"
      }
      log4j {
        port => 8052
        type => "Components"
      }
      log4j {
        port => 8053
        type => "MDM-Server"
      }
      log4j {
        port => 8054
        type => "MDM-Match"
      }
      log4j {
        port => 8055
        type => "JobServer"
      }
    }
     
    output {
      elasticsearch { 
          hosts => ["https://ELK_HOST:9200"] 
          ssl => true
          ssl_certificate_verification => true
          user => "<es_user>"
          password => "<es_password>"
      } 
    }
  2. Before starting Elasticsearch, open a shell/prompt and configure the ES_JAVA_OPTS environment variable as follows:

    • export ES_JAVA_OPTS="-Dmapper.allow_dots_in_name=true".

  3. Start Elasticsearch.

  4. Restart Logstash with the .conf configuration file:

    • ${LOGSTASH_HOME}/bin/logstash -f ${LOGSTASH_CONFIG_FILE}.

Importing Talend Kibana dashboard templates: prerequisites

If you wish to import Talend dashboards in Kibana, you must have:

  • Imported the Logstash templates in Elasticsearch using the following commands:

    Note

    In the commands below, you must replace ELK_HOST by the Elasticsearch host. The option es_user corresponds to the Elasticsearch user.

    • If you are using a true Certificate Authority (CA):

      curl -u es_user -s -XPUT https://ELK_HOST:9200/_template/template_logstash -d @template_logstash.json
      curl -u es_user -s -XPUT https://ELK_HOST:9200/.kibana/index-pattern/logstash-*  -d @dashboards/index-pattern/logstash.json
      curl -u es_user -s -XPUT https://ELK_HOST:9200/_template/template_esb -d @template_talendesb.json
      curl -u es_user -s -XPUT https://ELK_HOST:9200/.kibana/index-pattern/talendesb-*  -d @dashboards/index-pattern/talendesb.json
    • If you are using self-signed certificates add --insecure option:

      curl --insecure -u es_user -s -XPUT https://ELK_HOST:9200/_template/template_logstash -d @template_logstash.json
      curl --insecure -u es_user -s -XPUT https://ELK_HOST:9200/.kibana/index-pattern/logstash-*  -d @dashboards/index-pattern/logstash.json
      curl --insecure -u es_user -s -XPUT https://ELK_HOST:9200/_template/template_esb -d @template_talendesb.json
      curl --insecure -u es_user -s -XPUT https://ELK_HOST:9200/.kibana/index-pattern/talendesb-*  -d @dashboards/index-pattern/talendesb.json
  • Created a logstash-* index pattern:

    curl -u es_user -s -XPUT https://ELK_HOST:9200/.kibana/config/4.6.1 -d "{\"defaultIndex\" : \"logstash-*\"}" 

  • Created a talendesb-* index pattern:

    curl -u es_user -s -XPUT https://ELK_HOST:9200/.kibana/config/4.6.1 -d "{\"defaultIndex\" : \"talendesb-*\"}"

Importing Talend Kibana dashboard templates in an external Elastic stack

Talend provides the following Kibana dashboard templates as part of the open-source Elastic stack shipped with the Talend Log Server. You can also import these dashboard templates in Kibana.

  1. Start Kibana.

  2. Go to Objects, under the Settings tab.

  3. Within each sub-tab, import the templates using the Import/Export buttons:

    Kibana UI tab

    Location of file(s) to upload

    Dashboards

    Under Talend-LogServer/dashboards/dashboard:

    • default.json

    Searches

    Under Talend-LogServer/dashboards/search:

    • TABLE.json

    Visualizations

    Under Talend-LogServer/dashboards/visualization:

    • FILTER-BY-SEVERITY.json

    • FILTER-BY-SOURCE.json

    • GROUP-BY-SEVERITY.json

    • GROUP-BY-SOURCE.json

    • Middleware-Navigation.json

    • TIMELINE.json

    Kibana UI tab

    Location of file(s) to upload

    Dashboards

    Under Talend-LogServer/dashboards/dashboard:

    • ESB-Events.json

    • ESB-Locator-Endpoints.json

    • ESB-SAM.json

    Searches

    Under Talend-LogServer/dashboards/search:

    • All-Events-ESB-Locator-Endpoints.json

    • All-Evetns-ESB-SAM.json

    • DOCUMENTS-ESB-Events.json

    Visualizations

    Under Talend-LogServer/dashboards/visualization:

    • DOCUMENT-TYPES-CHART-ESB-Events.json

    • DOCUMENT-TYPES-TABLE-ESB-Events.json

    • ENDPOINT-AVAILABILITY-ESB-Locator-Endpoints.json

    • ENDPOINTS-PER-SERVICES-ESB-Locator-Endpoints.json

    • HISTOGRAM-ESB-Events.json

    • OFFLINE-ENDPOINTS-ESB-Locator-Endpoints.json

    • OFFLINE-SERVICES-ESB-Locator-Endpoints.json

    • REMOTE-CONSUMER-ESB-SAM.json

    • SERVICE-AVAILABILITY-ESB-Locator-Endpoints.json

    • SERVICE-REQUESTS-ESB-SAM.json

    • SERVICE-RESPONSES-ESB-SAM.json

    • TOP-10-ENDPOINTS-ESB-Locator-Endpoints.json

    • TOP-SERVICES-ESB-SAM.json