Function behavior in common PII - 6.1

Talend Components Reference Guide

EnrichVersion
6.1
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

What is sensitive data

The definition of sensitive data is broad and may differ from one country to the other or from one organization to the other. Basically, sensitive data can be personal information or business information which includes anything that poses a risk to the person or company in question.

Globally, Credit/Debit card data for example is considered to be sensitive. Also an employee's salary details any information that can be used to identify or locate a person can be considered to be sensitive data. A non-exhaustive list of personal sensitive data may include: first and last names, email addresses, addresses, Security Social Number (SSN), credit card numbers, bank account numbers, race, gender, date of birth, salary and geolocation combined with time.

For further information about personal sensitive data, check Personally Identifiable Information.

Also, business sensitive data may include trade secrets, acquisition plans, financial data and customer information, among other possibilities.

Functions and common PII

There are several functions in the tDataMasking component which vary according to the type of the data column.

It is advisable to use the functions predefined in the component with columns that hold personal information, such as first and last names, email addresses, addresses, SSN, credit card numbers, bank account numbers, race, gender, date of birth and salary.

Functions that are not self-explanatory are explained in the below table:

Function

Description

Date Variance

This function only applies on Date values. It uses a parameter which must be a number, this parameter represents a number of days. It will then modify the input date by adding or retrieving a number of days lower than the parameter.

For example : If the input date is 15-02-1992 and the parameter is 10, then the generated date will be randomly selected between 05-02-1992 (15 - 10) and 25-02-1992 (15 + 10). If the input date is null, then the function returns the current date. If the given parameter is 0 or null or if it is not a number, it will be replaced by 31.

Generate Account Number

This function generates a valid French bank account number. It requires no parameter and only applies on String values.

A French IBAN number is a 27-character code. The numbers are randomly generated but against algorithms. The last digit of the IBAN is known as the "clef RIB" and is generated with an algorithm and the third and fourth digits of the IBAN are also generated through an algorithm.

Generate Account Number and keep original country

This function works like Generate Account Number, it generates a valid bank account number for the original country.

If the input is a correct IBAN number, the function generates an IBAN number from the same country as the input taking into account the IBAN number which is different from one country to the other. If the input is a correct American account number the function keeps the first nine digits and randomly replaces the other.

This function requires a parameter that can be: true or false. If the parameter is true, the function keeps the input format, that is if there are spaces in the input, the output will have the same spaces. If the parameter is not true or false, it is considered to be false. If the input is null or an incorrect number, the function will have the same behavior as Generate Account Number.

Generate Credit Card

This function generates a valid credit card number. It requires no parameter and can be applied on String or Long values.

There are three types of credit card that can be generated: Visa, Master Card or American Express. One of these types is randomly chosen and a credit card number is generated. The number generated is randomly generated and pass algorithms that detect false credit card number.

Generate from Pattern

This function is applied only on Strings and it requires a parameter.

It generates a value that matches the pattern given as parameter. The pattern must follow the below rules:

- the A character is replaced by a random upper case letter.

- the a character is replaced by a random lower case letter.

- the 9 figure is replaced by a random digit.

- all other characters are kept as they are.

You can generate several strings with the same argument (value) by using \\1 in the pattern.

For example, if the given pattern is Aaaaa.Aaaaa99\\1,@gmail.com, the function here will generate something like Dsdf.Ksknt12@gmail.com. The @gmail.com value will be kept unchanged.

Please note that the function does not work correctly if a comma ',' is used in the pattern.

Generate Phone Number

This function is applied only on Strings and requires no parameter.

It generates a random phone number from different countries (France, Germany, Japan, UK and US).

Generate Social Security Number (SSN)

This function is only used on Strings and requires no parameter. It generates a correct random SSN for different countries according to your choice (France, Germany, Japan, UK and US).

Generate Sequence

Note

This function is not supported in the Spark version of the component.

This function can be applied on everything that is not a date (Integer, Long, Strings and so on). It requires a parameter that must be a number. This function returns the parameter, and, for each row, will increase this number by 1. If the parameter is not a number, it is set to 0.

Generate Uuid

This function is only applied on Strings and requires no parameter. It will replace the input value by a randomly generated UUID.

This function uses the UUID.randomUUID() provided by java, meaning that no seed is used here, implying that if the user runs twice the job, the uuids generated will be different.

Generate value between two values

This function generates a value randomly chosen between two values you give as argument. The argument must be a string holding the bounds, separated by comas, that is min and max.

This function can be applied to any types of fields. However, if the field is a date the bounds must also be dates and they must have the same format as in the schema, dd-MM-yyyy for example. Otherwise, the bounds must be integers.

If the input is of Date type, the function returns the current date if the parameter is not in the right format. Otherwise, it returns an empty string for string values and 0 for numeric values.

Keep characters between two positions

This function can be used on Strings and requires two parameters separated by commas.

The two first parameters represent the places of two elements in the input. The function returns a new String that only contains those elements and what is in between.

If the input is null or if the parameter is in a wrong format, the function will return an empty String. If the lower bound is lower than 1, it will be set to 1 and if the higher bound is greater than the length of the string, it will be set to this length. The two parameters can be given in any order. If the argument is 4, 2, it will be replaced by 2, 4. For example, if the input is Steven and the argument is 4, 2, the result will be tev.

Remove Characters between two positions and Replace characters between two positions

These functions have the same behavior as Keep characters between two positions but with a remove or replace statement.

Keep n first digits and replace following ones

This function is used on Strings, Integers and Long values and requires a number as a parameter.

If the parameter is n, the function keeps the first n digits of the input and then replaces all the digits that follow by other digits. Anything that is not a digit will not be changed. A null input will make the function returns an empty string or 0.

If the parameter is bigger than the input length, no modifications are applied.

keep n last digits and replace previous ones

This function is the counterpart of Keep n first digits and replace following ones.

Mask Address

This function can only be used on Strings. It replaces digits by other digits and everything else by X.

Moreover, there is a list of key words that will not be transformed: Rue, rue, r., strasse, Strasse, Street, street, St., St, Strae, Strada, Rua, Calle, Ave., avenue, Av., Allée, allée, alle, Avenue, Avenida, Bvd., Bd., Boulevard, boulevard, Blv., Viale, Avenida, Bulevar, Route, route, road, Road, Rd., Chemin, Way, Cour, Court, Ct., Place, place, Pl., Square, Impasse, Alle, Driveway, Auahrt, Viale, Esplanade, Esplanade, Promenade, Lungomare, Esplanada, Esplanada, Faubourg, faubourg, Suburb, Vorort, Periferia, Subrbio, Suburbio, Via, Via, industrial, area, zone, industrielle, Périphérique, Peripheral, Voie, voie, Track, Gleis, Carreggiata, Caminho, Pista, Forum, STREET, RUE, ST., AVENUE, BOULEVARD, BLV., BD, ROAD, ROUTE, RD., RTE, WAY, CHEMIN, COURT, CT., SQUARE, DRIVEWAY, ALLEE, DR., ESPLANADE, SUBURB, BANLIEUE, VIA, PERIPHERAL, PERIPHERIQUE, TRACK, VOIE, FORUM, INDUSTRIAL, AREA, ZONE, INDUSTRIELLE.

You can give a parameter, it can either be a list of key words to be added to the above list (separated by commas) or it can be a path to a file containing the words.

Mask Email

This function can only be used on Strings. It looks for the @ character and replaces everything before by either one element of the list given as parameter (can also be a path to a file containing those words), or by a series of X if there is no parameter.

Numeric Variance

This function applies only to numerical types (Integer, Long, Float and Double).

It takes a parameter that must be a number, this parameter represents a percentage of modification. The function modifies the input data by multiplying it by a number between the parameter and its opposite. For example, if the input is 100 and the parameter is 10, then the generated value will be a randomly selected value between 90 (100 - 10%) and 110 (100 + 10%). If the input is null, then the function will return 0. If the given parameter is 0, it will be replaced by 10.

Replace by consistent items from input list (or file)

This function modifies the input value by randomly selecting one of the values given as parameter. The values must be stored in a String and separated by commas, for example ("item1, item2, item3, etc."). It uses the hashCode() function provided by Java to choose an element from the list.

It can be applied to Strings or numerical types and it ensures that two similar inputs will have the same output. It will return an empty String or 0 if there is no parameter given or a wrong one.

Replace by item from input list (or file)

This function has the same behavior as Replace by consistent item from input list, but it randomly select the value from the list (or file), so outputs will be different.