Using component observability metrics to build reports for task runs - Cloud

Talend Cloud APIs User Guide

Version
Cloud
Language
English (United States)
Product
Talend Cloud
Module
Talend API Designer
Talend API Tester
Talend Data Preparation
Talend Data Stewardship
Talend Management Console
Content
Design and Development > Designing APIs
Design and Development > Testing APIs
Use POST at /monitoring/observability/metrics/component to build reports about task runs.

Procedure

  1. Get the ID of the task you need to analyze.
    The task ID is available on the Task details page in Talend Cloud Management Console. If you need to get this ID via API, follow this example.
  2. Issue the following API request to find the historical runs of this task during a given period:
    method: GET
    endpoint: 
          https://api.<env>.cloud.talend.com/processing/executables/tasks/{taskId}/executions?from={start_timestamp_ms}&to={end_timestamp_ms}
    headers: {
     "Authorization": "Bearer <personanl_access_token>"
    }
    payload: N/A
    From the response, count the number of runs and find those that failed, if any.
  3. Issue the following API request to find the execution duration of a specific type of component in each of the task runs:
    method: POST
    endpoint: 
          https://api.<env>.cloud.talend.com/monitoring/observability/metrics/component
    headers: {
     "Authorization": "Bearer <personanl_access_token>"
    }
    payload: {
        "startTime": "job start time", 
        "endTime": "job end time", 
        "environmentId": "environment id value",
        "sortBy": "component_execution_duration_milliseconds",
        "sortOrder": "desc";
        "filters": [
            {
                "field": "task_id ",
                "operator": "in", 
                "value": ["ID_of_the_task_to_analyze"]
            },
            {
                "field": "connector_type ",
                "operator": "in", 
                "value": ["a_component_type_such_as_tMongoDBConnection"]
            }
        ]
    }
  4. Analyze the response to identify how the execution duration of this type of components varies from one task run to another.

Results

This report helps you understand the circumstances in which a component changes its behavior among task runs.

Below is an example of the response of the POST request at /monitoring/observability/metrics/component.
{
  "items": [
    {
      "account_id": "2be59707-2230-45dc-a43d-db7e6d798425",
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "engine_type": "CLOUD",
      "engine_version": "2.10.7",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "xxx",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tMongoDBConnection",
      "connector_label": "tMongoDBConnection_1",
      "connector_id": "tMongoDBConnection_1",
      "component_start_time_seconds": 1628266578
    },
    {
      "account_id": "2be59707-2230-45dc-a43d-db7e6d798425",
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "engine_type": "CLOUD",
      "engine_version": "2.10.7",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "xxx",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tRowGenerator",
      "connector_label": "tRowGenerator_1",
      "connector_id": "tRowGenerator_1",
      "target_connector_type": "tFlowMeter",
      "target_label": "tFlowMeter_row1",
      "target_id": "tFlowMeter_row1",
      "connection_type": "row1",
      "connection_name": "output",
      "component_start_time_seconds": 1628266578,
      "component_connection_rows_total": 5000000,
      "component_execution_duration_milliseconds": 491585
    },
    ...
  ],
  "limit": 50,
  "offset": 0,
  "total": 7
}