Setting up a monitoring dashboard for the API - Cloud

Talend Cloud Data Inventory User Guide

Version
Cloud
Language
English
Product
Talend Cloud
Module
Talend Data Inventory
Content
Administration and Monitoring > Managing connections
Data Governance
Data Quality and Preparation > Enriching data
Data Quality and Preparation > Identifying data
Data Quality and Preparation > Managing datasets
Last publication date
2023-11-08

The activity tile of the API window is limited to the last 10 request statuses. However, when an API is enabled for a dataset, metrics are gathered, and can be consumed by the monitoring dashboard of your choice.

For more information about the metrics available, see Metrics available for the API.

Procedure

  1. In your Remote Engine Gen2 directory, add the following configuration to the default/nginx/nginx.conf file to make the metrics endpoint visible:
    location /metrics {
          proxy_pass http://dataset-api-server/api/v1/metrics;
        }
    This configuration can be added under the location /apis element in the file. For example:
    events { }
    http {
      server {
        listen 9102;
    
        location /apis {
          proxy_pass http://dataset-api-server/apis;
        }
    
        location /metrics {
          proxy_pass http://dataset-api-server/api/v1/metrics;
        }
    
      }
    }
  2. Restart your Remote Engine Gen2 instance to apply the new configuration.
  3. In the monitoring system of your choice, use the following URL with the correct parameters to get the metrics for the APIs running on your Remote Engine Gen2:
    protocol://remote_engine_host:DATASET_API_SHARING_EXTERNAL_PORT/metrics
    Different parameters that make the URL
    Parameter Value
    protocol http by default, or https if it is configured.
    remote_engine_host Host where your Remote Engine Gen2 is installed. This is specific to your infrastructure, if you do not know the host, contact your administrator.
    DATASET_API_SHARING_EXTERNAL_PORT Value of the DATASET_API_SHARING_EXTERNAL_PORT parameter in the default/.env file in the Remote Engine Gen2 directory.
  4. Configure your monitoring system as needed.

    Example

    With Prometheus, you can use the following configuration. For more information, see the Prometheus documentation.

     - job_name: 'dataset_api_sharing'
       scrape_interval: 10s
       metrics_path: /metrics
       static_configs:
        - targets: [ 'localhost:8080' ]

Results

Examples of Grafana dashboards which query Prometheus data for memory usage, number of requests per API, or average latency for example.