Removing unnecessary fields from flight records

A pipeline with a local source, a Data cleansing processor, a Field remover processor, and a test destination.

Before you begin

You have previously added the dataset holding your source data.

Download the file: field_remover-fields.csv. It contains a local dataset with a list of flights and their information (aircraft call sign, flight number, ICAO 24 aircraft address, flight registration, origin, destination, etc.).
You also have created the connection and the related dataset that will hold the processed data.

Here, a Test destination dataset.

Procedure

Click Add pipeline on the Pipelines page. Your new pipeline opens.
Give the pipeline a meaningful name.
Example
Remove fields from flight records
Click ADD SOURCE to open the panel allowing you to select your source data, here a local file containing flight information.

You can see in the sample that a lot of flight numbers are missing.
Select your dataset and click Select in order to add it to the pipeline.
Rename it if needed.
Click and add a Data cleansing processor to the pipeline. The configuration panel opens.
Give a meaningful name to the processor.
Example
mark empty flight numbers as N/A
Configure the processor:
1. Select Fill cells with value in the Function name list.
2. Select or enter .number in the Fields to process list.
3. Select Value in the Use with list and enter N/A in the Value field, as you want to add N/A in places where flight numbers are not listed.
Click Save to save your configuration.

Look at the preview of the processor to compare your data before and after the cleansing operation.
Click and add a Field remover processor to the pipeline. The configuration panel opens.
Give a meaningful name to the processor.
Example
remove unnecessary fields
To remove the fields that you do not need, here the ICAO 24 aircraft addresses, the ICAO aircraft type codes and the flight day timestamp, configure the processor like this:
1. Select or enter .icao24 in the Field to remove list.
2. Add a new element and select .typecode in the Input list.
3. Add a new element and select .day in the Input list.
Click Save to save your configuration.

Look at the preview of the processor to compare your data before and after the deletion operation.

The unnecessary fields are removed.
Click ADD DESTINATION and select the dataset that will hold your cleansed data.
Rename it if needed.
On the top toolbar of Talend Cloud Pipeline Designer, click the Run button to open the panel allowing you to select your run profile.
Select your run profile in the list (for more information, see Run profiles), then click Run to run your pipeline.