Reindex from a remote Elasticsearch cluster - 7.1

Talend Data Services Platform Migration Guide

EnrichVersion
7.1
EnrichProdName
Talend Data Services Platform
task
Installation and Upgrade
EnrichPlatform
Talend Activity Monitoring Console
Talend Administration Center
Talend Artifact Repository
Talend CommandLine
Talend Data Preparation
Talend Data Stewardship
Talend DQ Portal
Talend ESB
Talend Identity and Access Management
Talend JobServer
Talend Log Server
Talend Repository Manager
Talend Runtime
Talend SAP RFC Server
Talend Studio
You need to reindex your Logstash and Talend ESB indices.

Procedure

  1. Run the following command to see all the existing indices:
    curl -XGET 'oldhost9200/_cat/indices'
    Replace oldhost with the IP address or the hostname of the Elasticsearch cluster used in 6.5.x.
  2. Reindex your Logstash indices from the remote cluster to pull documents from the index used in 6.5.x into the index used in 7.0.x:
    curl -XPOST 'localhost:9200/_reindex?pretty' -H 'Content-Type: application/json' -d'
    {
      "source": {
        "remote": {
          "host": "http://oldhost:9200"
        },
        "index": "logstash-YYYY.MM.dd"
      },
      "dest": {
        "index": "logstash-YYYY.MM.dd"
      }
    }
    '
  3. Reindex your Talend ESB indices from the remote cluster to pull documents from the index used in 6.5.x into the index used in 7.0.x:
    curl -XPOST 'localhost:9200/_reindex?pretty' -H 'Content-Type: application/json' -d'
    {
      "source": {
        "remote": {
          "host": "http://oldhost:9200"
        },
        "index": "talendesb-YYYY.MM.dd"
      },
      "dest": {
        "index": "talendesb-YYYY.MM.dd"
      }
    }
    '
    Note:

    You must migrate all the logstash-YYYY.MM.dd and talendesb-YYYY.MM.dd indices one by one or choose only the indices you want to migrate.

    If you specify logstash-* or talendesb-* as the index name, you will get an error like the following one:

    "failures" : [
    {
    "index" : "logstash-*",
    "type" : "xxxxxxxx",
    "id" : "xxxxxxxxxxxxxx",
    "cause" : {
    "type" : "invalid_index_name_exception",
    "reason" : "Invalid index name [logstash-*], must not contain the following characters [ , \", *, \\, <, |, ,, >, /, ?]",
    "index_uuid" : "_na_",
    "index" : "logstash-*"
    },
    "status" : 400
    },
    {
    "index" : "logstash-*",
    "type" : "xxxxxx",
    "id" : "xxxxxxxxxxxxxxx",
    "cause" : {
    "type" : "invalid_index_name_exception",
    "reason" : "Invalid index name [logstash-*], must not contain the following characters [ , \", *, \\, <, |, ,, >, /, ?]",
    "index_uuid" : "_na_",
    "index" : "logstash-*"
    },
    "status" : 400
    }
    ]