tFuzzyMatch - 6.1

Talend Components Reference Guide

EnrichVersion
6.1
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

tFuzzyMatch properties

Component family

Data Quality

 

Function

Compares a column from the main flow with a reference column from the lookup flow and outputs the main flow data displaying the distance

Purpose

Helps ensuring the data quality of any source data against a reference data source.

Basic settings

Schema and Edit schema

A schema is a row description, it defines the number of fields to be processed and passed on to the next component. The schema is either Built-in or stored remotely in the Repository.

Since version 5.6, both the Built-In mode and the Repository mode are available in any of the Talend solutions.

Two read-only columns, Value and Match are added to the output schema automatically.

 

 

Built-in: The schema will be created and stored locally for this component only. Related topic: see Talend Studio User Guide.

 

 

Repository: The schema already exists and is stored in the Repository, hence can be reused in various projects and job designs. Related topic: see Talend Studio User Guide.

 

Matching type

Select the relevant matching algorithm among:

Levenshtein: Based on the edit distance theory. It calculates the number of insertion, deletion or substitution required for an entry to match the reference entry.

Metaphone: Based on a phonetic algorithm for indexing entries by their pronunciation. It first loads the phonetics of all entries of the lookup reference and checks all entries of the main flow against the entries of the reference flow.

Double Metaphone: a new version of the Metaphone phonetic algorithm, that produces more accurate results than the original algorithm. It can return both a primary and a secondary code for a string. This accounts for some ambiguous cases as well as for multiple variants of surnames with common ancestry.

 

Min distance

(Levenshtein only) Set the minimum number of changes allowed to match the reference. If set to 0, only perfect matches are returned.

 

Max distance

(Levenshtein only) Set the maximum number of changes allowed to match the reference.

 

Matching column

Select the column of the main flow that needs to be checked against the reference (lookup) key column

 

Unique matching

Select this check box if you want to get the best match possible, in case several matches are available.

 

Matching item separator

In case several matches are available, all of them are displayed unless the unique match box is selected. Define the delimiter between all matches.

Global Variables

NB_LINE: the number of rows read by an input component or transferred to an output component. This is an After variable and it returns an integer.

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio User Guide.

Usage

This component is not startable (green background) and it requires two input components and an output component.

Log4j

If you are using a subscription-based version of the Studio, the activity of this component can be logged using the log4j feature. For more information on this feature, see Talend Studio User Guide.

For more information on the log4j logging levels, see the Apache documentation at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.

Scenario 1: Levenshtein distance of 0 in first names

This scenario describes a four-component Job aiming at checking the edit distance between the First Name column of an input file with the data of the reference input file. The output of this Levenshtein type check is displayed along with the content of the main flow on a table

Setting up the Job

  1. Drag and drop the following components from the Palette to the design workspace: tFileInputDelimited (x2), tFuzzyMatch, tLogRow.

  2. Link the first tFileInputDelimited component to the tFuzzyMatch component using a Row > Main connection.

  3. Link the second tFileInputDelimited component to the tFuzzyMatch using a Row > Main connection (which appears as a Lookup row on the design workspace).

  4. Link the tFuzzyMatch component to the standard output tLogRow using a Row > Main connection.

Configuring the components

  1. Define the first tFileInputDelimited in its Basic settings view. Browse the system to the input file to be analyzed.

  2. Define the schema of the component. In this example, the input schema has two columns, firstname and gender.

  3. Define the second tFileInputDelimited component the same way.

    Warning

    Make sure the reference column is set as key column in the schema of the lookup flow.

  4. Double-click the tFuzzyMatch component to open its Basic settings view, and check its schema.

    The Schema should match the Main input flow schema in order for the main flow to be checked against the reference.

    Note that two columns, Value and Matching, are added to the output schema. These are standard matching information and are read-only.

  5. Select the method to be used to check the incoming data. In this scenario, Levenshtein is the Matching type to be used.

  6. Then set the distance. In this method, the distance is the number of char changes (insertion, deletion or substitution) that needs to be carried out in order for the entry to fully match the reference.

    In this use case, we set both the minimum distance and the maximum distance to 0. This means only the exact matches will be output.

  7. Also, clear the Case sensitive check box.

  8. Check that the matching column and look up column are correctly selected.

  9. Leave the other parameters as default.

Executing the Job

  • Save the Job and press F6 to execute the Job.

As the edit distance has been set to 0 (min and max), the output shows the result of a regular join between the main flow and the lookup (reference) flow, hence only full matches with Value of 0 are displayed.

A more obvious example is with a minimum distance of 1 and a maximum distance of 2, see Scenario 2: Levenshtein distance of 1 or 2 in first names

Scenario 2: Levenshtein distance of 1 or 2 in first names

This scenario is based on the scenario described above. Only the minimum and maximum distance settings in the tFuzzyMatch component are modified, which will change the output displayed.

  1. In the Component view of the tFuzzyMatch, change the minimum distance from 0 to 1. This excludes straight away the exact matches (which would show a distance of 0).

  2. Change also the maximum distance to 2. The output will provide all matching entries showing a discrepancy of 2 characters at most.

    No other changes are required.

  3. Make sure the Matching item separator is defined, as several references might be matching the main flow entry.

  4. Save the new Job and press F6 to run it.

    As the edit distance has been set to 2, some entries of the main flow match more than one reference entry.

You can also use another method, the metaphone, to assess the distance between the main flow and the reference, which will be described in the next scenario.

Scenario 3: Metaphonic distance in first name

This scenario is based on the scenario described above.

  1. Change the Matching type to Metaphone. There is no minimum nor maximum distance to set as the matching method is based on the discrepancies with the phonetics of the reference.

  2. Save the Job and press F6. The phonetics value is displayed along with the possible matches.