Configuration of ElasticSearch Synchronization - 7.1

Talend ESB Infrastructure Services Configuration Guide

EnrichVersion
7.1
EnrichProdName
Talend Data Fabric
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for ESB
Talend Real-Time Big Data Platform
EnrichPlatform
Talend ESB
Talend Runtime
task
Design and Development
Installation and Upgrade

Synchronization of metadata with ElasticSearch is configured by the following files in directory <TalendRuntimePath>\etc:

  • org.talend.esb.registry.el.sync.server.cfg: This file controls activation of the ElasticSearch access by the service registry server.

    The el.enable property must be set true after ElasticSearch access has been properly configured in order to really enable it.

  • org.talend.esb.registry.el.sync.client.cfg: This file contains access information for ElasticSearch including credentials and SSL keystore information.

    Property name

    Description

    el.host Hostname of ElasticSearch client node.
    el.port HTTP or HTTPS Port of ElasticSearch client node.
    el.ssl.enable Enable or disable HTTPS communication.
    el.authentication.enable Enable or disable authentication settings.
    el.username Username for BASIC authentication method.
    el.password Password for BASIC authentication method.
    el.index.name Elasticsearch index name for synchronization.
    el.ssl.keystore.password SSL Keystore password.
    el.ssl.keystore.key.password SSL Keystore Private key password.
    el.ssl.keystore.path SSL Keystore with private key.
    el.ssl.keystore.type SSL Keystore type.
    el.ssl.truststore.password SSL Truststore password.
    el.ssl.truststore.path SSL Truststore with Elasticsearch server certificate.
    el.ssl.truststore.type SSL Truststore type.
    el.ssl.enable.cn.check Enable or disable hostname verification.
  • org.talend.esb.registry.server.metadata.xsd: This file is the schema XSD which is used to validate the input metadata XML.

    <?xml version="1.0" encoding="UTF-8"?>
    <schema
        targetNamespace="http://metadata.extension.registry.esb.talend.org/v1"
        elementFormDefault="qualified" xmlns="http://www.w3.org/2001/XMLSchema"
        xmlns:tns="http://metadata.extension.registry.esb.talend.org/v1"
        xmlns:xs="http://www.w3.org/2001/XMLSchema">
     
        <element name="metadata" type="tns:metadata"></element>
     
        <complexType name="metadata">
            <sequence>
            <element name="ServiceType" type="string"></element>
                <element name="ServiceStatus">
                    <simpleType>
                        <restriction base="string">
                            <enumeration value="active"></enumeration>
                            <enumeration value="inactive"></enumeration>
                        </restriction>
                    </simpleType>
                </element>
                <element name="ServiceVersion" type="string"></element>
                <element name="CreationDate">
                    <simpleType>
                        <restriction base="date">
                            <pattern value="[^:Z]*"></pattern>
                        </restriction>
                    </simpleType>
                </element>
                <element name="Description" type="string"></element>
                <element name="Usage">
                    <simpleType>
                        <restriction base="string">
                            <enumeration value="internal"></enumeration>
                            <enumeration value="external"></enumeration>
                        </restriction>
                    </simpleType>
                </element>
                <element name="ServiceOwner" type="string"></element>
                <element name="ServiceDesigner" type="string"></element>
                <element name="ServiceDevelopmentTeam" type="string"></element>
                <element name="Projectname" type="string"></element>
                <element name="ApplicationName" type="string"></element>
                <element name="ServiceMaintenanceOwner" type="string"></element>
                <element name="ServiceMaintenanceContact" type="string"></element>
                <element name="Tags" type="string" maxOccurs="unbounded"
                    minOccurs="0">
                </element>
            </sequence>
        </complexType>
    </schema>
  • org.talend.esb.registry.el.sync.template.json: This file contains a JSON template with information for preparing and indexing of the metadata for search in ElasticSearch. This template must be consistent with the schema of the metadata.

    {
      "order" : 1,
      "template" : "talendregistry",
      "settings" : {
        "index.number_of_replicas" : "1",
        "index.number_of_shards" : "1",
        "analysis" : {
          "analyzer" : {
            "servicename" : {
              "type" : "pattern",
              "pattern" : "\\{.*\\}",
              "lowercase" : false
            },
            "namespace" : {
              "type" : "pattern",
              "pattern" : "(.*\\{)|(\\}.*)",
              "lowercase" : false
            }
          }
        }
      },
      "mappings" : {
        "metadata" : {
          "properties" : {
            "metadataExtension" : {
              "properties" : {
                "@schemaLocation" : {
                  "type" : "string",
                  "index" : "not_analyzed"
                },
                "ServiceName" : {
                  "type" : "string",
                  "analyzer" : "servicename",
                  "fields" : {
                    "raw" : {
                      "index" : "not_analyzed",
                      "type" : "string"
                    },
                    "namespace" : {
                      "index" : "analyzed",
                      "analyzer" : "namespace",
                      "type" : "string"
                    }
                  }
                },
                "metadata" : {
                  "properties" : {
                    "CustomServiceType" : {
                      "type" : "string",
                      "index" : "not_analyzed"
                    },
                    "CustomCreationDate" : {
                      "type" : "date",
                      "format" : "yyyy-MM-dd"
                    },
                    "CustomDescription" : {
                      "type" : "string",
                      "index" : "analyzed"
                    },
                    "CustomServiceOwner" : {
                      "type" : "string",
                      "index" : "analyzed",
                      "fields" : {
                        "raw" : {
                          "index" : "not_analyzed",
                          "type" : "string"
                        }
                      }
                    },
                  }
                }
              }
            }
          }
        }
      }
    }

Most of the template file needs to be taken over as it is. Only the content of the innermost metadata properties that starts with Custom need to be replaced by the properties matching the elements from the corresponding custom schema.

For the present example, matches are as follows:

  • CustomServiceType: a simple string which should be indexed for search as it is without cutting it into further details.

  • CustomCreationDate: a stringified date where the template contains the format for converting the string back into a date.

  • CustomDescription: a string which should be cut into words for full-text search.

  • CustomServiceOwner: a string which should be cut into words for full-text search, but at the same time the raw string should be searchable as a whole.