tGenKey in Talend Map/Reduce Jobs - 6.1

Talend Components Reference Guide

EnrichVersion
6.1
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

Warning

The information in this section is only for users that have subscribed to one of the Talend solutions with Big Data and is not applicable to Talend Open Studio for Big Data users.

In a Talend Map/Reduce Job, tGenKey, as well as the other Map/Reduce components preceding it, generates native Map/Reduce code. This section presents the specific properties of tGenKey when it is used in that situation. For further information about a Talend Map/Reduce Job, see Talend Big Data Getting Started Guide.

Component family

Data Quality

 

Basic settings

Schema and Edit schema

A schema is a row description, it defines the number of fields to be processed and passed on to the next component. The schema is either Built-in or stored remotely in the Repository.

Click Sync columns to retrieve the schema from the previous component in the Job.

 

 

Built-in: You create and store the schema locally for this component only. Related topic: see Talend Studio User Guide.

 

 

Repository: You have already created and stored the schema in the Repository. You can reuse it in other projects and job designs. Related topic: see Talend Studio User Guide.

 

Click the import icon to import blocking keys from the match rules that are defined and saved in the Studio repository.

When you click the import icon, a [Match Rule Selector] wizard is opened to help you import blocking keys from the match rules listed in the Studio repository and use them in your Job.

You can import blocking keys only from match rules that are defined with the VSR algorithm and saved in the Studio repository. For further information, see Importing match rules from the studio repository

Algorithm

Column

Select the column(s) from the main flow on which you want to define certain algorithms to set the functional key.

Note

When you select a date column on which to apply an algorithm or a matching algorithm, you can decide what to compare in the date format.

For example, if you want to only compare the year in the date, in the component schema set the type of the date column to Date and then enter "yyyy" in the Date Pattern field. The component then converts the date format to a string according to the pattern defined in the schema before starting a string comparison.

 

Pre-Algorithm

If required, select the relevant matching algorithm from the list:

remove diacritical marks:removes any diacritical mark.

remove diacritical marks and lower case: removes any diacritical mark and converts to lower case before generating the code of the column.

remove diacritical marks and upper case: removes any diacritical mark and converts to upper case before generating the code of the column.

lower case: converts the field to lower case before applying the key algorithm.

upper case: converts the field to upper case before applying the key algorithm.

add left position character: enables you to add a character to the left of the column.

add right position character: enables you to add a character to the right of the column.

 

Value

Set the algorithm value, where applicable.

 

Algorithm

Select the relevant algorithm from the list:

first character of each word: includes in the functional key the first character of each word in the column.

N first characters of each word: includes in the functional key N first characters of each word in the column.

first N characters of the string: includes in the functional key N first characters of the string.

last N characters of the string: includes in the functional key N last characters of the string.

first N consonants of the string: includes in the functional key N first consonants of the string.

first N vowels of the string: includes in the functional key N first vowels of the string.

pick characters: includes in the functional key the characters located at a fixed position (corresponding to the set digital/range).

exact: includes in the functional key the full string.

substring(a,b): includes in the functional key character according to the set index.

soundex code: generates a code according to a standard English phonetic algorithm. This code represents the character string that will be included in the functional key.

metaphone code: generates a code according to the character pronunciation. This code represents the character string that will be included in the functional key.

double-metaphone code: generates a code according to the character pronunciation using a new version of the Metaphone phonetic algorithm, that produces more accurate results than the original algorithm. This code represents the character string that will be included in the functional key.

fingerPrintkey: generates the functional key from a string value through the following sequential process:

  1. remove leading and trailing whitespace,

  2. change all characters to their lowercase representation,

  3. remove all punctuation and control characters,

  4. split the string into whitespace-separated tokens,

  5. sort the tokens and remove duplicates,

  6. join the tokens back together,

    Because the string parts are sorted, the given order of tokens does not matter. So, Cruise, Tom and Tom Cruise both end up with a fingerprint cruise tom and therefore end up in the same cluster.

  7. normalize extended western characters to their ASCII representation, for example gödel to godel.

    This reproduce data entry mistakes performed when entering extended characters with an ASCII-only keyboard. However, this procedure can also lead to false positives, for example gödel and godél would both end up with godel as their fingerprint but they are likely to be different names. So this might work less effectively for datasets where extended characters play substantial differentiation role.

nGramkey: this algorithm is similar to the fingerPrintkey method described above. But instead of using whitespace separated tokens, it uses n-grams, where the n can be specified by the user. This method generates the functional key from a string value through the following sequential process:

  1. change all characters to their lowercase representation,

  2. remove all punctuation and control characters,

  3. obtain all the string n-grams,

  4. sort the n-grams and remove duplicates,

  5. join the sorted n-grams back together,

  6. normalize extended western characters to their ASCII representation, for example gödel to godel.

    For example, the 2-gram fingerprint of Paris is arispari and the 1-gram fingerprint is aiprs.

    The delivered implementation of this algorithm is 2-grams.

Note

If the column on which you want to use the nGramkey algorithm can have data, with only 0 or 1 characters, you must filter this data before generating the functional key. This way you avoid potentially comparing records to each other that should not be match candidates.

colognPhonetic: a soundex phonetic algorithm optimized for the German language. It encodes a string into a Cologne phonetic value. This code represents the character string that will be included in the functional key.

 

Value

Set the algorithm value, where applicable.

 

Post-Algorithm

If required, select the relevant matching algorithm from the list:

use default value (string): enables you to choose a string to replace null or empty data.

add left position character: enables you to add a character to the left of the column.

add right position character: enables you to add a character to the right of the column.

 

Value

Set the option value, where applicable.

 

Show help

Select this check box to display instructions on how to set algorithms/options parameters.

Advanced settings

tStat Catcher Statistics

Select this check box to collect log data at the component level. Note that this check box is not available in the Map/Reduce version of the component.

Global Variables

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio User Guide.

Usage in Map/Reduce Jobs

In a Talend Map/Reduce Job, this component is used as an intermediate step and other components used along with it must be Map/Reduce components, too. They generate native Map/Reduce code that can be executed directly in Hadoop.

For further information about a Talend Map/Reduce Job, see the sections describing how to create, convert and configure a Talend Map/Reduce Job of the Talend Big Data Getting Started Guide.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs, and non Map/Reduce Jobs.

Limitation/prerequisite

n/a