tPartitioner - 6.1

Talend Components Reference Guide

EnrichVersion
6.1
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

Warning

This component will be available in the Palette of the Integration perspective of your studio on the condition that you have subscribed to one of the Talend Platform solutions or Big Data solutions.

Note that Talend Studio also enables the automatic implementation of parallelization across a Job without use of the parallelization components and we recommend using that approach. For further information, see the section describing how to enable parallelization of data flows of the Talend Studio User Guide. However, if you need to understand how to use these specific parallelization components, bear in mind that the parallelization components work closely with each other to accomplish parallel execution on given processes: the tPartitioner component dispatches the input records into a specific number of threads; the tCollector component sends these threads to its following components for parallel execution; the tDepartitioner component regroups the outputs of the processed parallel threads; the tRecollector component captures the output of a given tDepartitioner component and sends the captured data to the next component.

tPartitioner Properties

Component family

Orchestration

 

Function

This component splits the input records into a given number of threads which tCollector sends for parallel execution.

Purpose

This component partitions the input data before tCollector can transfer them to the parallel execution processes.

Basic settings

Schema and Edit Schema

A schema is a row description, it defines the number of fields to be processed and passed on to the next component. The schema is either Built-in or stored remotely in the Repository.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this option to view the schema only.

  • Change to built-in property: choose this option to change the schema to Built-in for local changes.

  • Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the [Repository Content] window.

Click Sync columns to retrieve the schema from the previous component connected in the Job.

 

 Number of Child Threads

Enter the number of threads you want to split the input records up into.

We recommend that this number be N-1 where N is the total number of CPUs or cores on the machine processing the data.

 

Buffer Size

Enter the number of rows to be processed before the memory is freed.

This is the number of rows to cache for each of the threads generated.

 

Use a key hash for partitions

Select this check box to use the hash mode to dispatch the input records into threads.

Once selecting it, the Key Columns table appears, in which you set the column(s) you want to apply the hash mode on. In the hash mode, the records meeting the same criteria are dispatched into the same threads.

If you leave this check box clear, the dispatch mode is Round-robin, meaning records are dispatched one-by-one to each thread, in a circular fashion, until the last record is dispatched. Be aware that this mode cannot guarantee that records meeting the same criteria go into the same threads.

Advanced settings

tStatCatcher Statistics

Select this check box to collect the log data at the component level.

Usage

This component should be put after the input component(s) and before tCollector.

This component uses and can only use the Trigger > Start link to connect to tCollector.

Global Variables

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

NB_LINE: the number of rows processed. This is an After variable and it returns an integer.

A Flow variable functions during the execution of a component while an After variable functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio User Guide.

Connections

Outgoing links (from this component to another):

Trigger: Start.

Incoming links (from one component to this one):

Row: Main.

For further information regarding connections, see Talend Studio User Guide.

Log4j

If you are using a subscription-based version of the Studio, the activity of this component can be logged using the log4j feature. For more information on this feature, see Talend Studio User Guide.

For more information on the log4j logging levels, see the Apache documentation at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.

Limitation

n/a

Scenario: sorting the customer data of large size in parallel

The Job in this scenario puts in order 20 million customer records by running parallelized executions.

Linking the components

  1. In the Integration perspective of your studio, create an empty Job from the Job Designs node in the Repository tree view.

    For further information about how to create a Job, see Talend Studio User Guide.

  2. Drop the following components onto the workspace: tFileInputDelimited, tPartitioner, tCollector, tSortRow, tDepartitioner, tRecollector, tFileOutputDelimited.

    The tFileInputDelimited component (labeled test file in this example) reads the 20 million customer records from a .txt file generated by tRowGenerator.

    For further information about the tRowGenerator component, see tRowGenerator

    For further information about how to label a component, see Talend Studio User Guide.

  3. Connect tPartitioner to tCollector using the Trigger > Starts link.

  4. Do the same to connect tDepartitioner to tRecollector.

  5. Connect the other components using the Row > Main link.

Splitting the input data flow

Configuring the input flow

  1. Double-click tFileInputDelimited to open its Component view.

  2. In the File name/Stream field, browse to, or enter the path to the file storing the customer records to be read.

  3. Click the button to open the schema editor where you need to create the schema to reflect the structure of the customer data.

  4. Click the button five times to add five rows and rename them as follows: FirstName, LastName, City, Address and ZipCode.

    In this scenario, we leave the data types with their default value String. In the real-world practice, you can change them depending on the data types of your data to be processed.

  5. Click OK to validate these changes and accept the propagation prompted by the pop-up dialog box.

  6. If needs be, complete the other fields of the Component view with values corresponding to your data to be processed. In this scenario, we leave them as is.

Configuring the tPartitioner component

  1. Double-click tPartitioner to open its Component view.

  2. In the Number of Child Threads field, enter the number of the threads you want to partition the data flow into. In this example, enter 3 because we are using 4 processors to run this Job.

  3. If required, change the value in the Buffer Size field to adapt the memory capacity. In this example, we leave the default one.

  4. Click the button next to Edit schema to open the schema editor.

  5. Select all the rows in the tPartitioner component schema table using Ctrl or Shift.

    Then the on the toolbar is activated.

  6. Click to copy the schema selected.

Sorting the input records

Configuring tCollector

  1. Double-click tCollector to open its Component view.

  2. Click the button next to Edit schema to open the schema editor.

  3. Press Ctrl+V to paste the tPartitioner component schema.

  4. Click OK to validate these changes and accept the propagation prompted by the pop-up dialog box.

Configuring tSortRow

  1. Double-click tSortRow to open its Component view.

  2. Under the Criteria table, click the button three times to add three rows to the table.

  3. In the Schema column column, select, for each row, the schema column to be used as the sorting criterion. In this example, select ZipCode, City and Address, sequentially.

  4. In the Sort num or alpha? column, select alpha for all the three rows.

  5. In the Order asc or desc column, select asc for all the three rows.

  6. If the schema does not appear, click the Sync columns button to retrieve the schema from the preceding component.

  7. Click Advanced settings to open its view.

  8. Select Sort on disk. Then the Temp data directory path field and the Create temp data directory if not exist check box appear.

  9. In Temp data directory path, enter the path to, or browse to the folder you want to use to store the temporary data processed by tSortRow. In this approach, tSortRow is enabled to sort considerably more data.

    As the threads will overwrite each other if they are written in the same directory, you need to create the folder for each thread to be processed using its thread ID. To do this, you can drop directly the global variable THREAD_ID of tCollector from the Outline view into this field; then the corresponding code is generated automatically, reading:

    ((Integer)globalMap.get("tCollector_1_THREAD_ID"))

    This makes the path read like:

    "E:/Studio/workspace/temp"+((Integer)globalMap.get("tCollector_1_THREAD_ID")).

    If the Outline view does not appear in the Studio, you can display it by selecting it from the [Show view] dialog box. For further information, see Talend Studio User Guide.

  10. Ensure that the Create temp data directory if not exists check box is selected.

Verifying tDepartitioner

  1. Double-click the tDepartitioner component open its Component view.

  2. If required, change the values in the Buffer Size field to adapt the memory capacity. In this example, we leave the default value.

  3. Click the button next to Edit schema to open the schema editor, then, check that all of the columns you need to output appear in the schema table of tDepartitioner. In this scenario, we output all the columns received from its preceding components.

Outputting the sorted data

  1. Double-click the tRecollector component to open its Component view.

  2. Click the button next to Edit schema to open the schema editor, then, paste the tPartitioner schema we copied earlier when we were configuring tPartitioner.

    This schema should be consistent with that of the tDepartitioner component offering data to the current tRecollector.

  3. Click OK to validate these changes and accept the propagation prompted by the pop-up dialog box.

  4. In the Linked Departitioner field, select the tDepartitioner component you want this component to receive data from. In this example, it is the tDepartitioner component labelled tDepartitioner_1. Therefore, select tDepartitioner_1.

  5. Double click the tFileOutputDelimited component to open its Component view.

  6. In the File Name field, browse to the file, or enter the directory and the name of the file, that you want to write the sorted data in. At runtime, this file will be created if it does not exist.

Executing the Job

Then you can press F6 to run this Job.

Once done, you can check the file holding the sorted data and the temporary folders created by tSortRow for sorting data on disk. These folders were emptied once the sorting had been done.