Using notifications and logs to debug a pipeline - Cloud

Talend Cloud Pipeline Designer User Guide

EnrichVersion
Cloud
EnrichProdName
Talend Cloud
EnrichPlatform
Talend Pipeline Designer
task
Administration and Monitoring > Monitoring executions
Administration and Monitoring > Monitoring logs
Data Governance > Filtering data
Data Quality and Preparation > Filtering data
Data Quality and Preparation > Managing datasets
Deployment > Deploying > Executing Pipelines
Design and Development > Designing Pipelines
How to use notifications and logs to spot and fix errors in a pipeline which execution has failed.

Before you begin

  • You have previously created a complete pipeline.

  • You have executed your pipeline and the execution failed.

Procedure

  1. Open the pipeline that failed on the pipeline Designer.

    Here, this pipeline contains a customers database table, a Python processor and a HDFS topic to store the processed data.

  2. To display the error message that informs you about the execution status, click the icon on the top toolbar.

    From that window, you can expand the error message by clicking on it.

  3. To get more detailed and categorized information about your error, either click the VIEW LOGS link from the Notification window, or go to the Metrics tab of the Pipeline Details panel and click the VIEW LOGS button.

    By default, all INFO logs are displayed.

  4. To sort the logs and only read the error messages, select the Error check box.

    You can see that the error is linked to an incorrect operator used in the Python processor.

  5. Now that you have spotted the error, select the Python processor to edit its configuration.

    Here the year 2017 is surrounded by unnecessary operators that cause the pipeline to fail. Edit the line as follows:output['number_year_registrated'] = 2017 - int(year) and save your changes.

  6. Click the run icon on the top toolbar to execute your pipeline again.

Results

The pipeline is fixed and is executed with success. If you missed the notification informing about the execution success, click the icon on the top toolbar to display it.