Error handling in Talend Studio
- Using the dedicated components provided by Talend
- Using links between two components in a Job
- Using a customized, appropriate Job design
Error handling with components
- tAssert and tAssertCatcher
- tChronometerStart and tChronometerStop
- tDie, tWarn and tLogCatcher
- tFlowMeter and tFlowMeterCatcher
Using tAssert and tAssertCatcher for error handling
tAssert works alongside tAssertCatcher to evaluate the status of a Job execution. It generates a boolean evaluation, either OK or FAIL, for the job execution status.
These two components, when used together, help catching certain types of errors and handling or routing them to the right direction, as per the project requirement.
A job expects a file with ten lines of data. Each line has a master record data about its data center. The batch processing should start only if this files arrives with ten lines of data.
In this Job, tFileRowCount reads the record count.
In tAssert, there is a condition to validate if the record count is equal to ten. tAssert performs the condition check and declares either OK or FAIL as output.
tAssertCatcher catches the output given by tAssert and, in this example, displays the output to the console.
- Triggering the next set of Jobs in an execution plan
- Sending an email to the source team stating that the input records are either good or not good, depending on the result
Using tChronometerStart and tChronometerStop for error handling
An exclusive chocolate company's world marketing head wants to see a specific report every morning. The devops team decides to look at the time the job takes to complete, to ensure that they keep an eye on the Job and never miss the service-level agreement (SLA).
In this example, before the file is read, the start time is captured by tChronometerStart. Towards the end, tChronometerStop captures the end time of the Job.
- Perform periodical audits to see the performance of each Job
- Determine dependency wait in case there are too many source/input file streams
Using tDie, tWarn and tLogCatcher for error handling
tDie allows you to have a code and message associated with the raised exception. It also has options to exit the JVM and kill the Job on exception. This component is very useful in case you want to take different actions in your Job based on the kind of error raised.
tWarn allows you to have a code and message associated with the raised exception. It is generally used to signal the completion of a Job or exceptions which do not block the execution of the Job. You can choose the message and the code you want to send to the catcher component. For example, you can use tWarn to signal the end of a specific pipeline flow in your Job.
tLogCatcher catches all exceptions and warnings raised by tWarn and tDie. You can also use tWarn or tDie at the end of the Job, to make tLogCatcher update the Job log when a Job finished successfully or to carry out other actions.
In this example, the Job reads from a file, makes some changes and writes to an output file. tWarn is used to signal when the Job is completed successfully and is written to a file. tDie is used to capture any errors when writing the file, and the message is written to a file.
- The Job works as expected and the file is written successfully. In this case, tWarn displays the message and the Job exits green.
- The Job works but there is an issue while creating the file. In this case, the Job does not end but displays an error and exits gracefully.
- The Job meets a fatal error.
tLogCatcher catches all the errors and redirects them as per the conditions given in tMap. Here, all the warnings raised by tWarn are sent to a warning file, all the error messages raised by tDie are sent to an error file and all other unexpected errors are sent to another error file.
Depending on the requirement and design, these components can be used to control the Job and also effectively manage the data flow. For example, you can use tWarn to call on a subJob if the file is created successfully.
Using tFlowMeter and tFlowMeterCatcher for error handling
Based on a defined schema, tFlowMeterCatcher catches the processing volumetric from tFlowMeter and passes them on to the output component.
In this example, the Job reads from a file, makes some changes and stores the output data in a user defined variable using the tJava component.
tFlowMeter is used to capture the inner join and reject records.
tFlowMeterCatcher is used to catch the records processed at both links. Then, it is used to check if all the records read are being processed and that no data is lost.
Using tWarn, it is captured as a message.
This Job ensures that there is no data leakage and that all the records that are read from the source file are either processed successfully or rejected.
Using tLogRow for error handling
tLogRow displays data or results in the Run console, it is used to monitor processed data.
The sample Job shows a design in which tLogRow is used to display the processed records to the console.