Upgrade and Migration Best Practice

The following critical steps represent the best practice in application server upgrade or migration to a new machine (on prem or cloud)

Backup

As with any migration / upgrade process, it is critical to backup the underlying data:

1. the installation data and conf directories (see Understanding the Data Locations)
2. the repository database. (see Database Server Backup/Restore)

One of the most critical first step is to save disk space and speed up performance by performing a major cleanup of the repository:

1. Apply the latest MIMM and MIMB Cumulative patches (in the application server and all MIMB Harvesting Agents).
2. Make sure that all Configurations are NOT on auto update.
3. Stop any scheduled operations, including all automatic metadata harvesting and database maintenance (MANAGE > Schedules).
4. Delete all unused / test Directories, Configurations, Models, Mappings, etc. (MANAGE > Repository).
5. Delete as many versions as possible (e.g. retaining the last few versions) of the remaining critical Configurations (MANAGE > Repository).
6. Delete unused model versions (MANAGE > Repository: Repository object > Operations > Delete unused versions).
7. Run the database maintenance to purge all deleted objects from the database (MANAGE > Schedules: Run Database Maintenance). Note that each database maintenance run purges deleted models for only 2 hours, therefore the database maintenance has to run as many times as needed until the log no longer shows any models to delete.

To avoid surprises after the upgrade/migration, it is critical to make sure that you start with a stable environment. The most important aspect of that is to make sure all imports are working before any upgrade because the source may no longer be available, may not have been imported with the latest version of the bridge, or may simply not have been imported for a long time.

1. Apply the latest MIMM and MIMB Cumulative patches (in the application server and all MIMB Harvesting Agents).
2. Make sure that all Configurations are NOT on auto update.
3. Stop any scheduled operations, including all automatic metadata harvesting and database maintenance (MANAGE > Schedules).
4. Delete the model import cache in $MM_HOME/data/MIMB/cache).
5. Manual full import (no incremental harvesting) of all Models (one by one) with clear import cache and uncheck "Create new versions only when new import has changes".
6. Manual force build of all Configurations (one by one) with testing (connection stitching as needed for lineage).

If the Talend Data Catalog repository database server must be migrated to a new machine (e.g. from on prem to cloud), make sure you follow the proper the proper database vendor process (see Database Server Backup/Restore).

Note that the above process works well if you stay with the same database technology (e.g. PostgreSQL), but such database cannot be performed between database technologies, such as Oracle or SQL Server to PostgreSQL because the repository database implementations are different / optimized for each database technology. In such case, one can use the Talend Data Catalog Application's Backup/Restore format (directories of XML files) but there are known limitations as some content is not back up in that format.

Harvesting Agents

If the Application Server has to be migrated to a new machine (e.g. from on prem to cloud), Make sure to reconnect each MIMB harvesting agent to the new Application Server.

Note that such MIMB harvesting agents can remain on prem, while connecting to the new Application Server on cloud.