Migration of Service Registry Metadata after Custom Schema Update - 8.0

Talend ESB Infrastructure Services Configuration Guide

Version
8.0
Language
English
Product
Talend Data Fabric
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
Module
Talend ESB
Talend Runtime
Content
Design and Development
Installation and Upgrade
Last publication date
2024-03-13

Generally, the metadata schema is supposed to be created before any metadata are uploaded to the Service Registry. But you can still modify the original metadata schema after service metadata are uploaded to the Service Registry.

For a compatible schema change, proceed as follows:

  1. Add optional elements to the schema definition and the ElasticSearch template like the following:

    • In the <TalendRuntimePath>/etc/org.talend.esb.registry.server.metadata.xsd file, add

      ...
      <element name="CustomNewField" type="string" minOccurs="0"></element>
      ...
    • In the <TalendRuntimePath>/etc/org.talend.esb.registry.el.sync.template.json file, add

      ...
          "CustomNewField" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
      ...
  2. Copy the changed schema and ElasticSearch template to the <TalendRuntimePath>/etc folder to replace the old files.

  3. Restart the Runtime container, or at least run the following commands to restart the Service Registry:

    tesb:stop-registry
    tesb:start-registry
  4. Re-synchronize ElasticSearch with the new template:

    tregistry:sync metadata

For incompatible schema changes, a full-fledged migration is needed as described in the following steps:

  1. Export the existing metadata:

    tregistry:export metadata <PATH>/metadata_export.xml
  2. List all the metadata:

    tregistry:list metadata
  3. Delete each metadata, for example:

    tregistry:delete metadata '{http://www.talend.org/service/}DemoService'
  4. Update the schema configuration <TalendRuntimePath>/etc/org.talend.esb.registry.server.metadata.xsd and index template <TalendRuntimePath>/etc/org.talend.esb.registry.el.sync.template.json to the new files.

  5. Restart the Runtime container, or at least run the following commands to restart the Service Registry:

    tesb:stop-registry
    tesb:start-registry
  6. Re-synchronize the Service Registry to enable the new index template. This will result in dropping and recreating the index in Elasticsearch.

    tregistry:sync metadata
  7. Update the old metadata to fit the new structure in <PATH>/metadata_export.xml.

  8. Import the updated metadata:

    tregistry:import metadata <PATH>/ metadata_export.xml
  9. Check Elasticsearch for the updated data, for example:

    tregistry:search metadata '{ "query": { "term": { "metadataExtension.ServiceName.raw" : "{http://www.talend.org/service/}DemoService" }}}'

Generally, the metadata schema is supposed to be created before any metadata are uploaded to the Service Registry. But you can still modify the original metadata schema after service metadata are uploaded to the Service Registry.

For a compatible schema change, proceed as follows:

  1. Add optional elements to the schema definition and the ElasticSearch template like the following:

    • In the <TalendRuntimePath>/etc/org.talend.esb.registry.server.metadata.xsd file, add

      ...
      <element name="CustomNewField" type="string" minOccurs="0"></element>
      ...
    • In the <TalendRuntimePath>/etc/org.talend.esb.registry.el.sync.template.json file, add

      ...
          "CustomNewField" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
      ...
  2. Copy the changed schema and ElasticSearch template to the <TalendRuntimePath>/etc folder to replace the old files.

  3. Restart the Runtime container, or at least run the following commands to restart the Service Registry:

    tesb:stop-registry
    tesb:start-registry
  4. Re-synchronize ElasticSearch with the new template:

    tregistry:sync metadata

For incompatible schema changes, a full-fledged migration is needed as described in the following steps:

  1. Export the existing metadata:

    tregistry:export metadata <PATH>/metadata_export.xml
  2. List all the metadata:

    tregistry:list metadata
  3. Delete each metadata, for example:

    tregistry:delete metadata '{http://www.talend.org/service/}DemoService'
  4. Update the schema configuration <TalendRuntimePath>/etc/org.talend.esb.registry.server.metadata.xsd and index template <TalendRuntimePath>/etc/org.talend.esb.registry.el.sync.template.json to the new files.

  5. Restart the Runtime container, or at least run the following commands to restart the Service Registry:

    tesb:stop-registry
    tesb:start-registry
  6. Re-synchronize the Service Registry to enable the new index template. This will result in dropping and recreating the index in Elasticsearch.

    tregistry:sync metadata
  7. Update the old metadata to fit the new structure in <PATH>/metadata_export.xml.

  8. Import the updated metadata:

    tregistry:import metadata <PATH>/ metadata_export.xml
  9. Check Elasticsearch for the updated data, for example:

    tregistry:search metadata '{ "query": { "term": { "metadataExtension.ServiceName.raw" : "{http://www.talend.org/service/}DemoService" }}}'