Monitoring a specific task without downloading logs

Obtain run metrics of a specific task without having to generate or download its logs.

Before you begin

Generate a personal access token by following Generating a Personal Access Token.
The user or the service account to be used must have the Author, the Viewer and the Executor permissions for the workspace to which the task to be monitored belongs.
You must know the ID of this workspace. Get this ID either via an API request or through Talend Management Console:
- API: use GET at the https://api.<your_environment>.cloud.talend.com/orchestration/workspaces.
- Talend Management Console: the ID of the workspace appears in the URL of the Edit workspace page.

You must know the ID of the task you need to monitor.

In Talend Management Console, this ID is displayed on the dedicated page of the task to be monitored. For example:

The task ID on the page of the task is highlighted.

Or you can obtain task IDs via the following API request:

method: GET
endpoint: https://api.<env>.cloud.talend.com/processing/executables/tasks
headers: {
  "Content-Type": "application/json",
  "Authorization": "Bearer <your_personal_access_token>"
          }
query parameters: {
   "workspaceId": "id_of_the_workspace_of_the_task_to_be_monitored"
                  }

About this task

In this section, the following API call is firstly issued:

method: GET
endpoint: https://api.<env>.cloud.talend.com/processing/executables/tasks/{taskId}/executions
headers: {
 "Content-Type": "application/json",
 "Authorization": "Bearer <your_personal_access_token>"
}
payload: {
  "lastDays": "1",
  "status": "execution_failed",
  "workspaceId": "id_of_the_workspace"
}

It is implemented in Talend API Tester for demonstration purposes.

Procedure

Open Talend API Tester in your browser and select GET from the Method list.
In the field next to the Method drop-down list, enter the endpoint to be used:
Example

In this example, the status, lastDays and workspaceId parameters are used to retrieve all the failed runs on a previous day of a specific task in the given workspace.
Issue this API request to get a list of the failed runs.

Use each of these run IDs to issue the following API request to their observability metrics:

method: GET
endpoint:
  https://api.<env>.cloud.talend.com/monitoring/observability/executions/{runId}/component
headers: {
   "Authorization": "Bearer <personanl_access_token>"
                  }
payload: N/A

Analyze the response to identify the cause of each failure. For example, if you see a component does not have the component_execution_duration_milliseconds field, this component could have stopped responding.
The following section presents more examples about how to use this metrics.

Results

Below is an example of the response of the GET request at /monitoring/observability/executions/{runId}/component.

{
            "account_id": "2be59707-2230-45dc-a43d-db7e6d798425",
            "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
            "engine_type": "CLOUD",
            "engine_version": "2.10.8",
            "workspace_id": "61273932d0366133d05729b7",
            "task_id": "612739e79a0ac71b8f3ed4dd",
            "task_execution_id": "947e3e2f-d199-4988-a5ab-14ceb36c80f3",
            "artifact_id": "612739e79a0ac71b8f3ed4db",
            "artifact_name": "job_with_rejected_rows",
            "artifact_version": "0.1.0.20212608065119",
            "start_time": "2021-08-26T06:53:30.127Z",
            "finish_time": "2021-08-26T06:53:35.361Z",
            "rows_rejected": 1,
            "operator": "admin",
            "operator_type": "HUMAN",
            "processes": [
            {
            "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
            "job_name": "MainJob",
            "pid": "20210806181617_2Y68h",
            "father_pid": "20210806181617_2Y68h",
            "root_pid": "20210806181617_2Y68h"
            }
            ],
            "metrics": {
            "items": [
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tMongoDBConnection",
            "connector_label": "tMongoDBConnection_1",
            "connector_id": "tMongoDBConnection_1",
            "component_start_time_seconds": 1628266578
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tRowGenerator",
            "connector_label": "tRowGenerator_1",
            "connector_id": "tRowGenerator_1",
            "target_connector_type": "tFlowMeter",
            "target_label": "vFlowMeter_row1",
            "target_id": "vFlowMeter_row1",
            "component_start_time_seconds": 1628266578,
            "component_connection_rows_total": 5000000,
            "component_execution_duration_milliseconds": 491585
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tFlowMeter",
            "connector_label": "vFlowMeter_row1",
            "connector_id": "vFlowMeter_row1",
            "component_start_time_seconds": 1628266578
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tFlowMeter",
            "connector_label": "vFlowMeter_row1",
            "connector_id": "vFlowMeter_row1",
            "target_connector_type": "tMongoDBOutput",
            "target_label": "Insert from SQL",
            "target_id": "tMongoDBOutput_1",
            "component_start_time_seconds": 1628266578,
            "component_connection_rows_total": 5000000,
            "component_execution_duration_milliseconds": 491605
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tMongoDBOutput",
            "connector_label": "Insert from SQL",
            "connector_id": "tMongoDBOutput_1",
            "component_start_time_seconds": 1628266578
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tMongoDBInput",
            "connector_label": "tMongoDBInput_2",
            "connector_id": "tMongoDBInput_2",
            "target_connector_type": "tLogRow",
            "target_label": "tLogRow_2",
            "target_id": "tLogRow_2",
            "component_start_time_seconds": 1628267070,
            "component_connection_rows_total": 2158754
            },
            {
            "pid": "20210806181617_2Y68h",
            "connector_type": "tLogRow",
            "connector_label": "tLogRow_2",
            "connector_id": "tLogRow_2",
            "component_start_time_seconds": 1628267070
            }
            ],
            "limit": 50,
            "offset": 0,
            "total": 7
            }
            }

Monitoring a specific task without downloading logs - Cloud

Talend Cloud APIs User Guide

Before you begin

About this task

Procedure

Example

Results