Skip to main content Skip to complementary content
Announcements

Removing duplicate values

After analyzing the email and postal columns using simple statistics indicators, the analysis results show the number of duplicate records in the columns. You can generate a ready-to-use Job on the analysis results. This Job removes duplicate values in the selected column.

You can follow the same procedure to remove duplicates from the Email or Phone columns.

Procedure

  1. In the Profiling perspective, click Analysis Results at the bottom of the editor.
  2. In the Simple Statistics results of the Email or Phone column, right-click the duplicate count bar in the chart and select Remove duplicates.

    This example uses the outcome of the simple statistics used on the Email column.

    The Integration perspective opens showing the generated Job.

    Job generated automatically from the analysis results.

    The database input component and the tUniqRow component are already configured according to your connection and the columns you are analyzing.

  3. Save the Job and press F6 to execute it.

Results

Duplicate values are written to the specified output database and file.

What to do next

You can follow the same procedure to remove duplicates from the postal column.

For more information on using the Profiling perspective to identify and remove corrupt, incomplete, or inaccurate data, see Data cleansing in the Talend Studio User Guide.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!

Leave your feedback here