Deduplicating rows - Cloud

Talend Cloud Data Preparation User Guide
Version
Cloud
Language
English
Product
Talend Cloud
Module
Talend Data Preparation
Content
Administration and Monitoring > Managing connections
Data Quality and Preparation > Cleansing data
Data Quality and Preparation > Managing datasets
Last publication date
2023-10-03

You can use the Remove duplicate rows function to easily delete all the rows that are exact duplicates and keep only one in your dataset.

Note: This function is not compatible with Spark Jobs, and HDFS or S3 exports.

Duplicated information can be introduced in spreadsheets because of human error, like a bad copy and paste for example, as well as automated operations. In this example, you received a dataset containing customer information, where all the rows are systematically duplicated.

You will use the Remove duplicate rows function to easily clean your dataset.

Procedure

  1. Click the header of any column from your dataset.
  2. Click the Table tab of the functions panel to display the list of functions that can be applied on the whole table.
  3. Point your mouse over the Remove duplicate rows function and click the eye icon to preview its effects.
  4. Click Submit to apply the function.

Results

All the duplicated information has been removed in one simple action, leaving you with only one correct occurrence of each row in your dataset.