Error handling in Talend Studio

author
Rekha Sree
EnrichVersion
7.1
EnrichProdName
Talend Big Data
Talend Real-Time Big Data Platform
Talend Open Studio for Data Integration
Talend MDM Platform
Talend Open Studio for Data Quality
Talend Cloud
Talend Data Management Platform
Talend Open Studio for Big Data
Talend Open Studio for ESB
Talend Data Integration
Talend Data Fabric
Talend Data Services Platform
Talend Big Data Platform
Talend ESB
Talend Open Studio for MDM
task
Design and Development
EnrichPlatform
Talend Studio

Error handling in Talend Studio

This article shows the various ways to capture, handle and manage errors at the Job level with use cases and sample Jobs.
In Talend Studio, there are three different ways to design error handling:
  • Using the dedicated components provided by Talend
  • Using links between two components in a Job
  • Using a customized, appropriate Job design

Error handling with components

This section explains how to handle errors using components.
Talend offers the following components to design error handling:
  • tAssert and tAssertCatcher
  • tChronometerStart and tChronometerStop
  • tDie, tWarn and tLogCatcher
  • tFlowMeter and tFlowMeterCatcher
  • tLogRow

Using tAssert and tAssertCatcher for error handling

This section explains how to design error handling with tAssert and tAssertCatcher.

tAssert works alongside tAssertCatcher to evaluate the status of a Job execution. It generates a boolean evaluation, either OK or FAIL, for the job execution status.

These two components, when used together, help catching certain types of errors and handling or routing them to the right direction, as per the project requirement.

Use case

A job expects a file with ten lines of data. Each line has a master record data about its data center. The batch processing should start only if this files arrives with ten lines of data.

Sample Job

Design

In this Job, tFileRowCount reads the record count.

In tAssert, there is a condition to validate if the record count is equal to ten. tAssert performs the condition check and declares either OK or FAIL as output.

tAssertCatcher catches the output given by tAssert and, in this example, displays the output to the console.

This job can be used for numerous tasks, such as:
  • Triggering the next set of Jobs in an execution plan
  • Sending an email to the source team stating that the input records are either good or not good, depending on the result

Using tChronometerStart and tChronometerStop for error handling

This section explains how to measure the time a Job or subJob takes using tChronometerStart and tChronometerStop.

Use case

An exclusive chocolate company's world marketing head wants to see a specific report every morning. The devops team decides to look at the time the job takes to complete, to ensure that they keep an eye on the Job and never miss the service-level agreement (SLA).

Sample Job

Design

In this example, before the file is read, the start time is captured by tChronometerStart. Towards the end, tChronometerStop captures the end time of the Job.

Apart from the use case given, these components can be used to:
  • Perform periodical audits to see the performance of each Job
  • Determine dependency wait in case there are too many source/input file streams

Using tDie, tWarn and tLogCatcher for error handling

This section explains how to design error handling with tDie, tWarn and tLogCatcher.

tDie allows you to have a code and message associated with the raised exception. It also has options to exit the JVM and kill the Job on exception. This component is very useful in case you want to take different actions in your Job based on the kind of error raised.

tWarn allows you to have a code and message associated with the raised exception. It is generally used to signal the completion of a Job or exceptions which do not block the execution of the Job. You can choose the message and the code you want to send to the catcher component. For example, you can use tWarn to signal the end of a specific pipeline flow in your Job.

tLogCatcher catches all exceptions and warnings raised by tWarn and tDie. You can also use tWarn or tDie at the end of the Job, to make tLogCatcher update the Job log when a Job finished successfully or to carry out other actions.

Sample Job

Design

In this example, the Job reads from a file, makes some changes and writes to an output file. tWarn is used to signal when the Job is completed successfully and is written to a file. tDie is used to capture any errors when writing the file, and the message is written to a file.

When the Job runs, there are three possible results.
  • The Job works as expected and the file is written successfully. In this case, tWarn displays the message and the Job exits green.
  • The Job works but there is an issue while creating the file. In this case, the Job does not end but displays an error and exits gracefully.
  • The Job meets a fatal error.

tLogCatcher catches all the errors and redirects them as per the conditions given in tMap. Here, all the warnings raised by tWarn are sent to a warning file, all the error messages raised by tDie are sent to an error file and all other unexpected errors are sent to another error file.

Depending on the requirement and design, these components can be used to control the Job and also effectively manage the data flow. For example, you can use tWarn to call on a subJob if the file is created successfully.

Using tFlowMeter and tFlowMeterCatcher for error handling

This section explains how to design error handling using tFlowMeter and tFlowMeterCatcher.

Based on a defined schema, tFlowMeterCatcher catches the processing volumetric from tFlowMeter and passes them on to the output component.

Sample Job

Design

In this example, the Job reads from a file, makes some changes and stores the output data in a user defined variable using the tJava component.

tFlowMeter is used to capture the inner join and reject records.

tFlowMeterCatcher is used to catch the records processed at both links. Then, it is used to check if all the records read are being processed and that no data is lost.

Using tWarn, it is captured as a message.

This Job ensures that there is no data leakage and that all the records that are read from the source file are either processed successfully or rejected.

Using tLogRow for error handling

This section explains how to design error handling using tLogRow.

tLogRow displays data or results in the Run console, it is used to monitor processed data.

Sample Job

Design

The sample Job shows a design in which tLogRow is used to display the processed records to the console.

Error handling with links