Configure Logstash to use Transport Layer Security (TLS) encryption - 7.0

Talend Data Fabric Installation Guide for Linux

EnrichVersion
7.0
EnrichProdName
Talend Data Fabric
task
Installation and Upgrade
EnrichPlatform
Talend Activity Monitoring Console
Talend Administration Center
Talend Artifact Repository
Talend CommandLine
Talend Data Preparation
Talend Data Stewardship
Talend DQ Portal
Talend ESB
Talend Identity and Access Management
Talend Installer
Talend JobServer
Talend Log Server
Talend MDM Server
Talend MDM Web UI
Talend Repository Manager
Talend Runtime
Talend SAP RFC Server
Talend Studio

Procedure

  1. Enable TLS on Logstash:
    1. Create a certs subdirectory under the Logstash config folder.
    2. Copy the ca/ca.crt, the public certificate and the private key of the node to the config/certs directory.
    3. Run the following command to convert the private key to the pkcs8 format:
    openssl pkcs8 -in config/certs/logstash.key -topk8 -nocrypt -out config/certs/logstash.pkcs8.key
    The password of the logstash_system user must correspond to the password generated in Configuring Transport Layer Security (TLS/SSL) in Elasticsearch.
  2. Edit the config/logstash.yml file as follows:
    node.name: logstash.local
    xpack.monitoring.elasticsearch.username: logstash_system
    xpack.monitoring.elasticsearch.password: 'TalendELK'
    xpack.monitoring.elasticsearch.url: https://node1.local:9200
    xpack.monitoring.elasticsearch.ssl.ca: config/certs/ca.crt
    The password of the logstash_system user must correspond to the password generated in Configuring Transport Layer Security (TLS/SSL) in Elasticsearch.
  3. Edit the logstash-talend.conf file as follows:
    input {
      beats {
        port => 5044
        ssl => true
        ssl_key => '/config/certs/logstash.pkcs8.key'
        ssl_certificate => '/config/certs/logstash.crt'
      }
      http {
        response_headers => {
            "Access-Control-Allow-Origin" => "*"
            "Access-Control-Allow-Headers" => "Content-Type, Access-Control-Allow-Headers, Authorization, X-Requested-With"
            "Access-Control-Allow-Methods" => "*"
            "Access-Control-Allow-Credentials" => "*"
        }
        codec => "json"
        port => 8057
        type => "Audit"
      }
    }
    filter {
        if [type] == "Audit" {
            json { source => "message" }
            mutate {
                rename => {
                    "severity" => "priority"
                    "logMessage" => "message"
                }
            }
            mutate { add_field => { "esIndex" => "talendaudit-%{+YYYY.MM.dd}" } }
        } else {
            grok {
                match => { "message" => "%{URIHOST:agentTimestamp} %{HAPROXYTIME:time} %{DATA:priority} %{SYSLOG5424SD:method} %{JAVACLASS:logger_name} %{GREEDYDATA:log_message}" }
            }
            if [log_message] {
                mutate { update => { "message" => "%{log_message}" } }
            }
            mutate { add_field => { "esIndex" => "logstash-%{+YYYY.MM.dd}" } }
            if [app_id] {
                mutate { rename => { "app_id" => "application" } }
            }
        }
        mutate {
            remove_field   => [ "beats_input_codec_plain_applied", "offset", "beat[name]", "app_id", "beat[hostname]", "host", "tags"  ]
            remove_tag   => [ "beats_input_codec_plain_applied" ]
        }
    }
    output {
      elasticsearch {
        hosts => ["https://node1.local:9200"]
        cacert => 'config/certs/ca/ca.crt'
        user => 'logstash_writer'
        password => 'TalendELK'
        index => "%{esIndex}"
      }
    }
  4. Run the following command to start Logstash with the right configuration file:
    bin/logstash -f  ../logstash-talend.conf