Monitoring Job runs on Remote Engines - Cloud

Talend Remote Engine User Guide for Linux

EnrichVersion
Cloud
EnrichProdName
Talend Cloud
EnrichPlatform
Talend Remote Engine
task
Design and Development
Installation and Upgrade

You can observe your Data Integration Jobs running on Talend Remote Engines.

Note: This feature requires Talend Remote Engine version 2.9.2 and Talend Studio version 7.3 R2020-07 or later versions.

If your Jobs are scheduled to run on Talend Remote Engine version 2.9.2, Talend Cloud Management Console can provide the data needed to build your own customized dashboards in your company monitoring system or an Application Performance Monitoring (APM) tool such as ELK, Datadog, Azure Monitor, or Amazon CloudWatch.

Once you have set up the metric and log collection system in Talend Remote Engine and your APM, you can design and organize your dashboards thanks to the information sent from Talend Cloud Management Console to the APM through the engine.

For example, with this data you can monitor in your dashboards:
  • Job run statuses
  • Talend Remote Engine availability
  • Resource consumption (CPU, memory, disk usage)
  • Workload (number of tasks running)
Thanks to the monitoring capabilities, you can also set up alerts in case errors occurs, and optimize Job run performance by comparing its duration and the records processed.

Available metrics

The following metrics and additional logs are provided by Talend Remote Engine that you can integrate in your APM tool.

Already existing logs from previous Remote Engine versions:
  • Job run logs
  • Remote Engine logs
New logs from Talend Remote Engine v2.9.2 or later:
  • Job management logs
  • Performance metrics and context of Jobs, Remote Engines, and operating system
    • OS resource consumption
    • Remote Engine resource consumption and workload
    • Job run resource consumption (CPU, memory)
    • Job run history (including component information, status, and duration)
    • Job run performance (for example, records processed or rejected)
The metrics are available in two different formats, accessible from an HTTP endpoint exposed by the Remote Engine:
  • generic JSON file
  • Prometheus-compatible format

Using the feature

To start using the monitoring feature, you must:
  1. Download and install the new Remote Engine version.
  2. Set up the Remote Engine on your development environment in Talend Cloud Management Console.
  3. Install/update Talend Studio to version 7.3 R2020-07 or later.
  4. Republish your Jobs from the new Studio version to Talend Cloud Management Console.
  5. Update you tasks to use the latest Jobs versions (if not done automatically).
  6. Run your Jobs.

Setting up a connection between the APM and the Talend apps depends on which APM you use. You can access some examples on the Talend Community website.