Scenario: Doing a fuzzy match on two columns and outputting the main and rejected data - 6.1

Talend Components Reference Guide

Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
Talend Studio
Data Governance
Data Quality and Preparation
Design and Development

This scenario describes a five-component Job aiming at: first checking the edit distance between the IdClient column of an input file against the data of the reference input file, and second checking all emails by their pronunciation in the Email column against the data of the reference input file. The outputs of these two matching types are written in two separate files.

In this scenario, we have already stored the input schemas of the input and reference files in the Repository. For more information about storing schema metadata in the Repository tree view, see Talend Studio User Guide.

Setting up the Job

  1. In the Repository tree view, expand Metadata and the file node where you have stored the input schemas and drop the relevant file onto the design workspace.

    The [Components] dialog box displays.

  2. Select tFileInputDelimited from the list and click OK to close the dialog box.

    The tFileInputDelimited component displays in the workspace. The input file used in this scenario is called ClientSample. It holds 12 columns including the two columns IdCient and Email we want to do the fuzzy match on.

  3. Do the same for the second input file you want to use as a reference, ClientSample_Update in this scenario.

  4. Drop the following components from the Palette onto the design workspace: tFuzzyJoin, tFileOutputExcel, and tFileOutputDelimited.

  5. Connect the main and reference input files to tFuzzyJoin using Main links. The link between the reference input file and tFuzzyJoin displays as a Lookup link on the design workspace.

  6. Connect tFuzzyJoin to tFileOutputExcel using the Main link and tFuzzyJoin to tFileOutputDelimited using the Inner join reject link.

  7. If needed, double-click the main and reference input files to display their Basic settings views. All their property fields are automatically filled in. If you do not define your input files in the Repository, fill in the details manually after selecting Built-in in the Property Type field.

Configuring the tFuzzyJoin component

  1. Double-click tFuzzyJoin to display its Basic settings view and define its properties.

  2. Click the Edit schema button to open a dialog box that displays the data structure of the input files and then define the data you want to pass to the output components, two columns in this scenario, IdClient and Email.

  3. Click OK to close the dialog box.

  4. In the Key definition area of the Basic settings view of tFuzzyJoin, click the plus button to add two columns to the list and then select the input columns and the output columns you want to do the fuzzy matching on from the Input key attribute and Lookup key attribute lists respectively, IdClient and Email in this example.

  5. Click in the first cell of the Matching type column and select from the list the method to be used to check the incoming data against the reference data. In this scenario, Levenshtein is the matching type to be used.

  6. Then set the minimum and maximum distances. In this method, the distance is the number of character changes (insertion, deletion or substitution) that needs to be carried out in order for the entry to fully match the reference. In this example, we want the minimum distance to be 0 and the maximum distance to be 2. This will output all entries in the IdClient that exactly match or that have maximum two character changes.


    You can create and store context variables for the minimum and maximum distances in order to start from a low max number to match rows and go up to higher max number to match more possible rows. You can press Ctrl+Space to access the variable list and select these new context variables. For more information about context variables, see Talend Studio User Guide.

  7. Set the matching type for the second column to be checked, Metaphone in this example. There is no minimum nor maximum distance to set because this matching method is based on phonetic discrepancies between the input main and reference data.

  8. Select the Inner join (with reject output) check box to define one of the outputs as inner join reject table.

Configuring the output component

  1. Double-click tFileOutputExcel to display its Basic settings view and define its properties.

  2. Set the destination file name as well as the Sheet name and select the Include header check box.

  3. Double-click tFileOutputDelimited to display its Basic settings view and define its properties.

  4. Set the destination file name as well as row and field separators in the corresponding fields and select the Include header check box.

Job execution

Save your Job and click F6 to execute it.

The output of the Levenshtein matching type on the IdClient column is written in the defined excel file, and the output of the metaphone matching type on the Email column is written in the defined delimited file.