tBlockedFuzzyJoin - 6.1

Talend Components Reference Guide

EnrichVersion
6.1
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

Warning

This component will be available in the Palette of Talend Studio on the condition that you have subscribed to one of the Talend Platform products.

tBlockedFuzzyJoin properties

Component family

Data Quality

This component is deprecated and hidden from the Palette by default, but it will continue to work in Jobs you import from older releases.

You can use the tRecordMatching component in place.

Function

tBlockedFuzzyJoin joins two tables by doing a fuzzy match on several columns. It compares columns from the main flow with reference columns from the lookup flow and outputs the match data, the possible match data and the rejected data.

Purpose

Helps ensuring the data quality of any source data against a reference data source.

Basic settings

Schema and Edit Schema

A schema is a row description, it defines the number of fields to be processed and passed on to the next component. The schema is either Built-in or stored remotely in the Repository.

Since version 5.6, both the Built-In mode and the Repository mode are available in any of the Talend solutions.

 

 

Built-in: You create and store the schema locally for this component only. Related topic: see Talend Studio User Guide .

 

 

Repository: You have already created and stored the schema in the Repository. You can reuse it in other projects and Job designs. Related topic: see Talend Studio User Guide .

 

Replace output column with lookup column if matches or possible matches

Select this check box to replace the output column with the lookup column in case of match or possible match values.

Key definition

Input key attribute

Select the column(s) from the main flow that needs to be checked against the reference (lookup) key column.

 

Lookup key attribute

Select the lookup key columns that you will use as a reference against which to compare the columns from the input flow.

 

Matching type

Select the relevant matching algorithm from the list:

Exact Match: matches each processed entry to all possible reference entries that have exactly the same value.

Levenshtein: Based on the edit distance theory. It calculates the number of insertion, deletion or substitution required for an entry to match the reference entry.

Metaphone: Based on a phonetic algorithm for indexing entries by their pronunciation. It first loads the phonetics of all entries of the lookup reference and checks all entries of the main flow against the entries of the reference flow.

Double Metaphone: a new version of the Metaphone phonetic algorithm, that produces more accurate results than the original algorithm. It can return both a primary and a secondary code for a string. This accounts for some ambiguous cases as well as for multiple variants of surnames with common ancestry.

 

Case sensitive

Select this check box to consider the letter case.

 

Min. distance

Only for Levenshtein. Set the minimum number of changes allowed to match the reference. If set to 0, only perfect matches(Exact Match) are returned.

Note

You can create and store context variables for the minimum and maximum distances and then have your Job to loop on these values in order to start from a low max number to match rows and go up to higher max number to match more possible rows. You can press Ctrl+Space to access the variable list and select the new context variables. For more information about context variables, see Talend Studio User Guide.

 

Max. distance

Only for Levenshtein. Set the maximum number of changes allowed to match the reference.

Advanced settings

tStatCatcher Statistics

Select this check box to collect log data at the component level.

Global Variables

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio User Guide.

Usage

This component is not startable (green background) and it requires two input components and one or more output components.

Limitation/prerequisite

n/a

Scenario: Doing a fuzzy match on two columns and outputting the match, possible match and non match values

This scenario describes a six-component Job that aims at:

  • matching each processed group number in the grp column against the entries that have exactly the same values in the reference input file,

  • checking the edit distance between the entries in the firstname column of an input file against those of the reference input file.

The outputs of these two matching types are written in three output files: the first for match values, the second for possible match values and the third for the values for which there are no matches in the lookup file.

In this scenario, we have already stored the main and reference input schemas in the Repository. For more information about storing schema metadata in the Repository, see Talend Studio User Guide.

The main input file contains four columns: grp, gender, firstname and count. The data in this input file have problems such as duplication, first names spelled differently or wrongly, different information for the same customer.

Setting up the Job

  1. In the Repository tree view, expand Metadata and the FileExcel node where you have stored the main input schemas and then drop it onto the design workspace.

    A tFileInputExcel component holding your schema displays on the workspace.

  2. Do the same with the reference input schema to display the reference tFileInputExcel on the workspace.

  3. Drop a tBlockedFuzzyJoin and tLogRow (x3) from the Palette onto the design workspace.

  4. Connect the main and reference input Excel files to tBlockedFuzzyJoin using Main links. The link between the reference input Excel file and tBlockedFuzzyJoin displays as a Lookup link on the design workspace.

  5. Connect tBlockedFuzzyJoin to the three tLogrow components using the Matches, Possible Matches and Non Matches links.

Configuring the input components

Double-click the main and reference input Excel file components to display their Basic settings views.

The capture below shows the properties of the main input file.

The capture below shows the properties of the reference input file.

The property fields for both tFileInputExcel components are automatically filled in. If you do not define your input schemas locally in the Repository, fill in the details manually after selecting Built-in in the Schema and Property Type fields.

Configuring the tBlockedFuzzyJoin component

  1. Double-click tBlockedFuzzyJoin to display its Basic settings view and define its properties.

  2. Click the Edit schema button to open a dialog box. Here you can define the data you want to pass to the output components.

    In this example we want to pass the four input columns to the output components in addition to the new column ref_firstname.

  3. Click OK to close the dialog box and proceed to the next step.

  4. In the Key definition area of the Basic settings view of tBlockedFuzzyJoin, click the plus button to add two columns to the list.

  5. Select the input columns and the output columns you want to do the fuzzy matching on from the Input key attribute and Lookup key attribute lists respectively, grp and firstname in this example.

  6. Click in the first cell of the Matching type column and select from the list the method to be used to check the incoming data against the reference data, Exact match in this example. There is no minimum nor maximum distance to set.

  7. Set the matching type for the second column, Levenshtein in this example.

  8. Then set the minimum and maximum distances. In this method, the distance is the number of character changes (insertion, deletion or substitution) that needs to be carried out in order for the entry to fully match the reference. In this example, we want the min. distance to be 0 and the max. distance to be 2. This will output all entries in the firstname column that exactly match or that have maximum two character changes.

Job execution

  1. Double-click the first tLogRow component to display its Basic settings view and define its properties.

  2. In the Mode area, select Table to display the source file and the tBlockedFuzzyJoin results together to be able to compare them.

  3. Do the same for the other two tLogRow components.

  4. Save your Job and press F6 to execute it.

Three output tables are written on the console. The first shows the match entries, the second show the possible match entries and the third shows the non match entries according to the used matching method in the defined columns.

The figure below illustrates extractions of the three output tables.