Configuring Logstash and Elasticsearch in an external Elastic stack - 6.5

Talend Real-Time Big Data Platform Installation Guide for Windows

EnrichVersion
6.5
EnrichProdName
Talend Real-Time Big Data Platform
task
Installation and Upgrade
EnrichPlatform
Talend Activity Monitoring Console
Talend Administration Center
Talend Artifact Repository
Talend CommandLine
Talend Data Preparation
Talend Data Stewardship
Talend DQ Portal
Talend ESB
Talend Identity and Access Management
Talend Installer
Talend JobServer
Talend Log Server
Talend Repository Manager
Talend Runtime
Talend SAP RFC Server
Talend Studio
You can use your external ELK stack to collect and classify logs from Data Integration, ESB or MDM by configuring your Logstash and Elasticsearch.

Procedure

  1. Create/edit your Logstash configuration file.
  2. In the input section, you add the configuration ports for Talend components.
  3. In the output section, you configure Elasticsearch as output.
    input {
      log4j {
        port => 8050
        type => "TAC"
      }
      log4j {
        port => 8052
        type => "Components"
      }
      log4j {
        port => 8053
        type => "MDM-Server"
      }
      log4j {
        port => 8054
        type => "MDM-Match"
      }
      log4j {
        port => 8055
        type => "JobServer"
      }
    }
     
    output {
      elasticsearch { 
          hosts => ["https://ELK_HOST:9200"] 
          ssl => true
          ssl_certificate_verification => true
          user => "<es_user>"
          password => "<es_password>"
      } 
    }

    The options user and password should match your Elasticsearch credentials.

    If you are not using a true Certificate Authority (CA), set the option ssl_certificate_verification to false.

  4. Before starting Elasticsearch, open a prompt and configure the ES_JAVA_OPTS environment variable as follows:
    SET ES_JAVA_OPTS="-Dmapper.allow_dots_in_name=true"
  5. Start Elasticsearch.
  6. Restart Logstash with the .conf configuration file:
    ${LOGSTASH_HOME}\bin\logstash -f ${LOGSTASH_CONFIG_FILE}