tReservoirSampling - 6.1

Talend Components Reference Guide

EnrichVersion
6.1
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

Warning

This component will be available in the Palette of Talend Studio on the condition that you have subscribed to one of the Talend Platform products.

tReservoirSampling properties

Component family

Data Quality

 

Function

tReservoirSampling enables you to extract a random sample data from a big data set.

Purpose

tReservoirSampling extracts a sample from the input data set in such a way that profiling results on the sample data are uniform and homogeneous with the profiling results on the full data set.

Basic settings

Schema and Edit schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

Since version 5.6, both the Built-In mode and the Repository mode are available in any of the Talend solutions.

Click Sync columns to retrieve the schema from the previous component in the Job.

 

 

Built-In: You create and store the schema locally for this component only. Related topic: see Talend Studio User Guide.

 

 

Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs. Related topic: see Talend Studio User Guide.

 

Sample Size

Set how many rows to sample from the input flow.

Advanced settings

Seed for random generator

Set a random number if you want to extract the same sample in different executions of the Job.

Repeating the execution with a different value for the seed will result in a different data samples being extracted.

Keep this field empty if you want to extract a different data sample each time you execute the Job.

 

tStat Catcher Statistics

Select this check box to collect log data at the component level.

Usage

This component helps you to test profiling analyses on a sample data and have results similar to the results on the full data set.

tReservoirSampling can not be used in Map/Reduce Jobs for the time being.

Limitation/prerequisite

n/a

Scenario: Extracting sample data from an input data set

This scenario describes a basic Job that extracts sample data from an input flow.

Below is a capture of the input flow:

Setting up the Job

  1. Drop the following components from the Palette onto the design workspace: tFileInputDelimited, tReservoirSampling and tFileOutputDelimited.

  2. Connect all the components together using the Row > Main link.

Configuring the input data

  1. Double-click tFileInputDelimited to display the Basic settings view and define the component properties.

  2. In the File name/Stream field, browse to the file to be used as the main input.

    This file provides some information about customers.

  3. Define the row and field separators and the header and footer in the corresponding fields, if any.

  4. Click the [...] button next to Edit schema to open a dialog box and define the input schema.

    In this example, according to the input file structure, the schema is made of ten columns.

  5. Click the [+] button and define the input columns in the dialog box as in the above figure. Click OK to close the dialog box.

  6. If needed, right-click tFileInputDelimited and select Data Viewer to display a view of the input data.

Configuring the sample data

  1. Double-click tReservoirSampling to display the Basic settings view and define the component properties.

  2. Click the Edit schema button to view the input and output columns and do any modifications in the output schema, if needed.

  3. In the Sample Size field, enter a number for the rows you want to extract from the input flow, 24 in this example.

  4. Click the Advanced settings tab and enter a random number in the Seed for random generator field.

    By setting a number in this field, you will extract the same sample in each execution of the Job. Change the value if you want to extract a different sample.

Configuring the output component

  1. Double-click tFileOutputDelimited to display its Basic settings view and define the component properties.

  2. In the File Name field, specify the path to the file to which you want to write the sample data.

  3. Define the row and field separators in the corresponding fields, if any.

Executing the Job

  1. Save your Job and press F6 to execute it.

    The sample data is extracted and written to the output file.

  2. Right-click the output component and select Data Viewer to display the extracted data.

    24 records have been extracted from the input file as you defined in the tReservoirSampling component settings. The Code column indicates that data has not been extracted sequentially from the input file. Data has been extracted in a way that any profiling results on the sample data will be close to the profiling results on the full data set.