Shuffles the data from in an input table to protect the actual data while having a functional data set. Data will remain usable for purposes such as testing and training.
tDataShuffling replaces original values with other values for the same column from a different row.
If you define one or several columns as the partition group, the whole table is split into a given number of partitions. These partitions share the same values in the columns partitioned. Then, the shuffling process is applied independently to each partition. All partitions are merged into one output table.
If you do not set columns as the partition group, the shuffling process is applied to the whole input table.
In local mode, Apache Spark 1.4.0 and later versions are supported.
For more technologies supported by Talend, see Talend components.
Depending on the Talend product you are using, this component can be used in one, some or all of the following Job frameworks:
-
Standard: see tDataShuffling Standard properties.
The component in this framework is available in Talend Data Management Platform, Talend Big Data Platform, Talend Real Time Big Data Platform, Talend Data Services Platform, and in Talend Data Fabric.
-
Spark Batch: see tDataShuffling properties for Apache Spark Batch.
The component in this framework is available in all Talend Platform products with Big Data and in Talend Data Fabric.