Scenario: sorting the customer data of large size in parallel - 6.3

Talend Components Reference Guide

EnrichVersion
6.3
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

The Job in this scenario puts in order 20 million customer records by running parallelized executions.

Linking the components

  1. In the Integration perspective of your studio, create an empty Job from the Job Designs node in the Repository tree view.

    For further information about how to create a Job, see Talend Studio User Guide.

  2. Drop the following components onto the workspace: tFileInputDelimited, tPartitioner, tCollector, tSortRow, tDepartitioner, tRecollector, tFileOutputDelimited.

    The tFileInputDelimited component (labeled test file in this example) reads the 20 million customer records from a .txt file generated by tRowGenerator.

    For further information about the tRowGenerator component, see tRowGenerator

    For further information about how to label a component, see Talend Studio User Guide.

  3. Connect tPartitioner to tCollector using the Trigger > Starts link.

  4. Do the same to connect tDepartitioner to tRecollector.

  5. Connect the other components using the Row > Main link.

Splitting the input data flow

Configuring the input flow

  1. Double-click tFileInputDelimited to open its Component view.

  2. In the File name/Stream field, browse to, or enter the path to the file storing the customer records to be read.

  3. Click the button to open the schema editor where you need to create the schema to reflect the structure of the customer data.

  4. Click the button five times to add five rows and rename them as follows: FirstName, LastName, City, Address and ZipCode.

    In this scenario, we leave the data types with their default value String. In the real-world practice, you can change them depending on the data types of your data to be processed.

  5. Click OK to validate these changes and accept the propagation prompted by the pop-up dialog box.

  6. If needs be, complete the other fields of the Component view with values corresponding to your data to be processed. In this scenario, we leave them as is.

Configuring the tPartitioner component

  1. Double-click tPartitioner to open its Component view.

  2. In the Number of Child Threads field, enter the number of the threads you want to partition the data flow into. In this example, enter 3 because we are using 4 processors to run this Job.

  3. If required, change the value in the Buffer Size field to adapt the memory capacity. In this example, we leave the default one.

  4. Click the button next to Edit schema to open the schema editor.

  5. Select all the rows in the tPartitioner component schema table using Ctrl or Shift.

    Then the on the toolbar is activated.

  6. Click to copy the schema selected.

Sorting the input records

Configuring tCollector

  1. Double-click tCollector to open its Component view.

  2. Click the button next to Edit schema to open the schema editor.

  3. Press Ctrl+V to paste the tPartitioner component schema.

  4. Click OK to validate these changes and accept the propagation prompted by the pop-up dialog box.

Configuring tSortRow

  1. Double-click tSortRow to open its Component view.

  2. Under the Criteria table, click the button three times to add three rows to the table.

  3. In the Schema column column, select, for each row, the schema column to be used as the sorting criterion. In this example, select ZipCode, City and Address, sequentially.

  4. In the Sort num or alpha? column, select alpha for all the three rows.

  5. In the Order asc or desc column, select asc for all the three rows.

  6. If the schema does not appear, click the Sync columns button to retrieve the schema from the preceding component.

  7. Click Advanced settings to open its view.

  8. Select Sort on disk. Then the Temp data directory path field and the Create temp data directory if not exist check box appear.

  9. In Temp data directory path, enter the path to, or browse to the folder you want to use to store the temporary data processed by tSortRow. In this approach, tSortRow is enabled to sort considerably more data.

    As the threads will overwrite each other if they are written in the same directory, you need to create the folder for each thread to be processed using its thread ID. To do this, you can drop directly the global variable THREAD_ID of tCollector from the Outline view into this field; then the corresponding code is generated automatically, reading:

    ((Integer)globalMap.get("tCollector_1_THREAD_ID"))

    This makes the path read like:

    "E:/Studio/workspace/temp"+((Integer)globalMap.get("tCollector_1_THREAD_ID")).

    If the Outline view does not appear in the Studio, you can display it by selecting it from the [Show view] dialog box. For further information, see Talend Studio User Guide.

  10. Ensure that the Create temp data directory if not exists check box is selected.

Verifying tDepartitioner

  1. Double-click the tDepartitioner component open its Component view.

  2. If required, change the values in the Buffer Size field to adapt the memory capacity. In this example, we leave the default value.

  3. Click the button next to Edit schema to open the schema editor, then, check that all of the columns you need to output appear in the schema table of tDepartitioner. In this scenario, we output all the columns received from its preceding components.

Outputting the sorted data

  1. Double-click the tRecollector component to open its Component view.

  2. Click the button next to Edit schema to open the schema editor, then, paste the tPartitioner schema we copied earlier when we were configuring tPartitioner.

    This schema should be consistent with that of the tDepartitioner component offering data to the current tRecollector.

  3. Click OK to validate these changes and accept the propagation prompted by the pop-up dialog box.

  4. In the Linked Departitioner field, select the tDepartitioner component you want this component to receive data from. In this example, it is the tDepartitioner component labelled tDepartitioner_1. Therefore, select tDepartitioner_1.

  5. Double click the tFileOutputDelimited component to open its Component view.