Converting and filtering records with Talend Data Mapper
This article explains how to convert data types and filter invalid records using a map.
Talend Data Mapper is a powerful mechanism for processing records. A common operation during records processing is to validate data and reject or adjust records with invalid values.
<readings>
<reading>
<Timestamp>1/5/2019 2:14 AM</Timestamp>
<AmbientTempF>44.8</AmbientTempF>
<ProbeTempF>33</ProbeTempF>
<Humidity>74</Humidity>
</reading>
<reading>
<Timestamp>1/5/2019 2:04 AM</Timestamp>
<AmbientTempF>45</AmbientTempF>
<ProbeTempF>N/C</ProbeTempF>
<Humidity>73</Humidity>
</reading>
</readings>
In some of the records, the value in the ProbeTempF
element is
N/C
, which is invalid. This article explains how configure a map to
split valid and invalid values into different files and create a third file in which the
invalid values are replaced by a valid one to avoid errors.
The map created in this example also converts String data types into more appropriate types, such as Date/Time or Double.
The map is then used in a simple data integration Job.
Creating structures
Creating the input structure
Create the XML input structure from an existing file.
About this task
Procedure
Results
Creating the output structure
Create the XML output structure using the existing input structure.
About this task
Procedure
Results
Creating the map
Before you begin
Procedure
Results
Converting data types
Convert strings into more meaningful data types.
About this task
Procedure
Converting a string to a date
Convert a String data type to a Date/Time type with a specific format.
Procedure
Results
Filtering records
Use filters to separate your data into different output files.
About this task
In the input file, some ProbeTempF elements have
N/C
as a value. This value is invalid and the records
containing it need to be separated from the valid records.