Setting up a monitoring dashboard for the API

Setting up a monitoring dashboard for the API - Cloud

Talend Cloud Data Inventory User Guide

Version

Cloud

Language

English

Product

Talend Cloud

Module

Talend Data Inventory

Content

Administration and Monitoring > Managing connections

Data Governance

Data Quality and Preparation > Enriching data

Data Quality and Preparation > Identifying data

Data Quality and Preparation > Managing datasets

Last publication date

2024-02-28

The activity tile of the API window is limited to the last 10 request statuses. However, when an API is enabled for a dataset, metrics are gathered, and can be consumed by the monitoring dashboard of your choice.

For more information about the metrics available, see Metrics available for the API.

Procedure

In your Remote Engine Gen2 directory, add the following configuration to the default/nginx/nginx.conf file to make the metrics endpoint visible:

location /metrics {
      proxy_pass http://dataset-api-server/api/v1/metrics;
    }

This configuration can be added under the location /apis element in the file. For example:

events { }
http {
  server {
    listen 9102;

    location /apis {
      proxy_pass http://dataset-api-server/apis;
    }

    location /metrics {
      proxy_pass http://dataset-api-server/api/v1/metrics;
    }

  }
}

Restart your Remote Engine Gen2 instance to apply the new configuration.

In the monitoring system of your choice, use the following URL with the correct parameters to get the metrics for the APIs running on your Remote Engine Gen2:

protocol://remote_engine_host:DATASET_API_SHARING_EXTERNAL_PORT/metrics

Different parameters that make the URL
Parameter	Value
`protocol`	`http` by default, or `https` if it is configured.
`remote_engine_host`	Host where your Remote Engine Gen2 is installed. This is specific to your infrastructure, if you do not know the host, contact your administrator.
`DATASET_API_SHARING_EXTERNAL_PORT`	Value of the `DATASET_API_SHARING_EXTERNAL_PORT` parameter in the default/.env file in the Remote Engine Gen2 directory.

Configure your monitoring system as needed.
Example
With Prometheus, you can use the following configuration. For more information, see the Prometheus documentation.
```
 - job_name: 'dataset_api_sharing'
   scrape_interval: 10s
   metrics_path: /metrics
   static_configs:
    - targets: [ 'localhost:8080' ]
```

Results

Examples of Grafana dashboards which query Prometheus data for memory usage, number of requests per API, or average latency for example.