Monitoring a specific task without downloading logs - Cloud

Talend Cloud APIs User Guide

Version
Cloud
Language
English (United States)
Product
Talend Cloud
Module
Talend API Designer
Talend API Tester
Talend Data Preparation
Talend Data Stewardship
Talend Management Console
Content
Design and Development > Designing APIs
Design and Development > Testing APIs
Obtain run metrics of a specific task without having to generate or download its logs.

Before you begin

  • Generate a personal access token by following Generating a Personal Access Token.

  • The user or the service account to be used must have the Author, the Viewer and the Executor permissions for the workspace to which the task to be monitored belongs.
  • You must know the ID of this workspace. Get this ID either via an API request or through Talend Cloud Management Console:
    • API: use GET at the https://api.<your_environment>.cloud.talend.com/orchestration/workspaces.
    • Talend Cloud Management Console: the ID of the workspace appears in the URL of the Edit workspace page.
  • You must know the ID of the task you need to monitor.
    In Talend Cloud Management Console, this ID is displayed on the dedicated page of the task to be monitored. For example:
    Or you can obtain task IDs via the following API request:
    method: GET
    endpoint: https://api.<env>.cloud.talend.com/processing/executables/tasks
    headers: {
      "Content-Type": "application/json",
      "Authorization": "Bearer <your_personal_access_token>"
              }
    query parameters: {
       "workspaceId": "id_of_the_workspace_of_the_task_to_be_monitored"
                      }

About this task

In this section, the following API call is firstly issued:
method: GET
endpoint: https://api.<env>.cloud.talend.com/processing/executables/tasks/{taskId}/executions
headers: {
 "Content-Type": "application/json",
 "Authorization": "Bearer <your_personal_access_token>"
}
payload: {
  "lastDays": "1",
  "status": "execution_failed",
  "workspaceId": "id_of_the_workspace"
}
It is implemented in Talend API Tester for demonstration purposes.

Procedure

  1. Open Talend API Tester in your browser and select GET from the Method list.
  2. In the field next to the Method drop-down list, enter the endpoint to be used:

    Example

    In this example, the status, lastDays and workspaceId parameters are used to retrieve all the failed runs on a previous day of a specific task in the given workspace.
  3. Issue this API request to get a list of the failed runs.
  4. Use each of these run IDs to issue the following API request to their observability metrics:
    method: GET
    endpoint: 
      https://api.<env>.cloud.talend.com/monitoring/observability/executions/{runId}/component
    headers: {
       "Authorization": "Bearer <personanl_access_token>"
                      }
    payload: N/A
  5. Analyze the response to identify the cause of each failure. For example, if you see a component does not have the component_execution_duration_milliseconds field, this component could be hanging.
    The following section presents more examples about how to use this metrics.

Results

Below is an example of the response of the GET request at /monitoring/observability/executions/{runId}/component.
{
            "account_id": "2be59707-2230-45dc-a43d-db7e6d798425",
            "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
            "engine_type": "CLOUD",
            "engine_version": "2.10.8",
            "workspace_id": "61273932d0366133d05729b7",
            "task_id": "612739e79a0ac71b8f3ed4dd",
            "task_execution_id": "947e3e2f-d199-4988-a5ab-14ceb36c80f3",
            "artifact_id": "612739e79a0ac71b8f3ed4db",
            "artifact_name": "job_with_rejected_rows",
            "artifact_version": "0.1.0.20212608065119",
            "start_time": "2021-08-26T06:53:30.127Z",
            "finish_time": "2021-08-26T06:53:35.361Z",
            "rows_rejected": 1,
            "operator": "admin",
            "operator_type": "HUMAN",
            "processes": [
            {
            "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
            "job_name": "MainJob",
            "pid": "20210806181617_2Y68h",
            "father_pid": "20210806181617_2Y68h",
            "root_pid": "20210806181617_2Y68h"
            }
            ],
            "metrics": {
            "items": [
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tMongoDBConnection",
            "connector_label": "tMongoDBConnection_1",
            "connector_id": "tMongoDBConnection_1",
            "component_start_time_seconds": 1628266578
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tRowGenerator",
            "connector_label": "tRowGenerator_1",
            "connector_id": "tRowGenerator_1",
            "target_connector_type": "tFlowMeter",
            "target_label": "vFlowMeter_row1",
            "target_id": "vFlowMeter_row1",
            "component_start_time_seconds": 1628266578,
            "component_connection_rows_total": 5000000,
            "component_execution_duration_milliseconds": 491585
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tFlowMeter",
            "connector_label": "vFlowMeter_row1",
            "connector_id": "vFlowMeter_row1",
            "component_start_time_seconds": 1628266578
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tFlowMeter",
            "connector_label": "vFlowMeter_row1",
            "connector_id": "vFlowMeter_row1",
            "target_connector_type": "tMongoDBOutput",
            "target_label": "Insert from SQL",
            "target_id": "tMongoDBOutput_1",
            "component_start_time_seconds": 1628266578,
            "component_connection_rows_total": 5000000,
            "component_execution_duration_milliseconds": 491605
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tMongoDBOutput",
            "connector_label": "Insert from SQL",
            "connector_id": "tMongoDBOutput_1",
            "component_start_time_seconds": 1628266578
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tMongoDBInput",
            "connector_label": "tMongoDBInput_2",
            "connector_id": "tMongoDBInput_2",
            "target_connector_type": "tLogRow",
            "target_label": "tLogRow_2",
            "target_id": "tLogRow_2",
            "component_start_time_seconds": 1628267070,
            "component_connection_rows_total": 2158754
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tLogRow",
            "connector_label": "tLogRow_2",
            "connector_id": "tLogRow_2",
            "component_start_time_seconds": 1628267070
            }
            ],
            "limit": 50,
            "offset": 0,
            "total": 7
            }
            }