Stitching Models Together for Data Flow Tracing

Some external metadata Models may contain data movement source specifications and data movement rules. These are in turn imported into Talend Data Catalog . In many cases, these data movement source specifications may match up with another external metadata Model which was imported separately. Such data movement specification Models may then be added to a configuration and may be “stitched” together with that second Model, where one Model is the complete representation of a source that is defined in another with data movement specifications.

This is a mostly automated process which you perform once per new ETL/DI/BI model, in that Talend Data Catalog will propose the best connection matches it can find and identify the level of confidence (completeness of the match for stitching).

Steps

Ensure proper permissions

1. Sign in as a user with at least the Metadata Managementcapability object role assignment on the Configuration you are in.

View the configuration architecture

2. Go to the MANAGE > Configuration in the banner.
3. Click Diagram.

View the configuration list

4. Go to MANAGE > Configuration in the banner.

Identify stitching candidates

5. Select any model in the configuration with a warning and go to the Connections tab.

Stitch the connections

6. In the Connections tab, click the magic wand icon to Propose resolutions.
7. If necessary, double-click the row with a warning to resolve any ambiguities in the harvested connection definition.

8. If necessary, double-click a row and specify Select Manually to connect specific catalogs and schemas as defined in the data process model connection (model with a warning) and how that schema is defined in the data store selected.

9. Repeat for all other connections with a warning.

10. Click Build.

There are various options in the Build dialog.

UPDATE VERSIONS will update the configuration to ensure that they default or latest version of the contained models are the versions included in this configuration version. This insures that the newly imported model versions are included before rebuilding connections

REBUILD ALL CONNECTIONS option to rebuild all connections, even those already successfully resolved.

Example

You sign in as a user with at least the Metadata Managementcapability object role assignment on the Configuration you are in and go to the MANAGE > Configuration in the banner.

The configuration management feature is quite intelligent and already knows how to stitch the existing models together. Thus, we will need to begin with an clean configuration in a clean database to be sure it does not remember how to stitch and we can demonstrate the proposal and build process.

Import a Staging to Dimensional, Staging DW and Dimensional DW model.

• Connection Dimensional is not connected.
• Connection Staging is not connected.

These are the connections defined in the Talend DI model, the destination and the source. The goal of the configuration management process is to resolve these connections, otherwise referred to as stitching, so that lineage may be computed and presented. Just as in the actual DI tool and databases, the connections in Talend must match what is in the database exactly.

Refresh the browser to clear the caching information about what models are available and the Connections tab will then show (as the UI now understands that this is a DI model). Go to the Connections tab.

Then click on the magic wand icon for Propose Resolutions.

At this point, the proposals are simply connection resolution rules, but they have not been computed. In order to do so, click BUILD.

In this case, there is only one version of each of these models, so the UPDATE VERSIONS option is moot. However, click Yes for REBUILD ALL CONNECTIONS. Then click BUILD and refresh the browser.

Double-click the cell for each connection under the Approved header and they will be approved.

Explore Further

View Log

View Log presents the Log Message dialog for the selected connection. You may see any connection errors documented as log messages.

Stitching Report

Stitching Report presents a complete report on what was and was not stitched between the connection and the data store. It is presented as a flat list that you may drill down in (e.g., schema > table > column)

Stitching Connection Name Resolution

Connection names, those names used within a data integration (DI/ETL) or a business intelligence (BI) tool to reference data stores, are often not the same as the names for those same data stored as harvested in the repository. Because of this difference, you will see a different presentation in the lineage overview of the model with connection names, versus what you see in a data flow trace after stitching the connections to their data stores. This data connection name resolution is performed automatically as part of the stitching process and will even present the “proper” schema names (those from the data stare harvest) in the data lineage trace view.

Example

Go to the object page for the AP to Staging ETL process.

There are two database connections. Note, their names are shortened versions (spaces missing, etc.) of the data stores.

Click on the StagingDW connection:

<No name> is presented.

Now, go back up to the level of the entire ETL model and then click the Data Flow tab:

Again, you see the shortened names.

However, since we have stitched this model to the two data stores, and have the complete (proper) names for the database and schemas, we see these in the lineage trace. Go back to the Staging DW connection and navigate to the Vendor table object page and go to the Data Flow tab.