Scenario: Filtering rows of data based on a condition and saving the result to a local file - 6.1

Talend Open Studio for Big Data Components Reference Guide

EnrichVersion
6.1
EnrichProdName
Talend Open Studio for Big Data
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

This scenario describes a four-component Job that filters a list of customers to find out customers from a particular country, and saves the result list to a local file. Before the input data is filtered, duplicate entries are first removed from the list.

The input file contains three columns: Name, Country, and Age, and it has some duplicate entries, as shown below:

Mario;PuertoRico;49
Mike;USA;22
Ricky;PuertoRico;37
Silvia;Spain;20
Billy;Canada;21
Ricky;PuertoRico;37
Romeo;UK;19
Natasha;Russia;25
Juan;Cuba;23
Bob;Jamaica;55
Mario;PuertoRico;49

Dropping and linking components

  1. Drop the following components from the Palette to the design workspace: tPigLoad, tPigDistinct, tPigFilterRow, and tPigStoreResult.

  2. Right-click tPigLoad, select Row > Pig Combine from the contextual menu, and click tPigDistinct to link these two components.

  3. Repeat this operation to link tPigDistinct to tPigFilterRow, and tPigFilterRow to tPigStoreResult using Row > Pig Combine connections to form a Pig process.

Configuring the components

Loading the input data and removing duplicates

  1. Double-click tPigLoad to open its Basic settings view.

  2. Click the [...] button next to Edit schema to open the [Schema] dialog box.

  3. Click the [+] button to add three columns according to the data structure of the input file: Name (string), Country (string) and Age (integer), and then click OK to save the setting and close the dialog box.

  4. Click Local in the Mode area.

  5. Fill in the Input file URI field with the full path to the input file.

  6. Select PigStorage from the Load function list, and leave rest of the settings as they are.

  7. Double-click tPigDistinct to open its Basic settings view, and click Sync columns to make sure that the input schema structure is correctly propagated from the preceding component.

    This component will remove any duplicates from the data flow.

Configuring the filter

  1. Double-click tPigFilterRow to open its Basic settings view.

  2. Click Sync columns to make sure that the input schema structure is correctly propagated from the preceding component.

  3. Select Use advanced filter and fill in the Filter field with filter expression:

    "Country matches 'PuertoRico'"

    This filter expression selects rows of data that contains "PuertoRico" in the Country column.

Configuring the file output

  1. Double-click tPigStoreResult to open its Basic settings view.

  2. Click Sync columns to make sure that the input schema structure is correctly propagated from the preceding component.

  3. Fill in the Result file field with the full path to the result file.

  4. If the target file already exists, select the Remove result directory if exists check box.

  5. Select PigStorage from the Store function list, and leave rest of the settings as they are.

Saving and executing the Job

  1. Press Ctrl+S to save your Job.

  2. Press F6 or click the Run button on the Run tab to run the Job.

    The result file contains the information of customers from the specified country.