You can observe your Data Integration Jobs running on Talend Remote Engines.
If your Jobs are scheduled to run on Talend Remote Engine version 2.9.2, Talend Cloud Management Console can provide the data needed to build your own customized dashboards in your company monitoring system or an Application Performance Monitoring (APM) tool such as ELK, Datadog, Azure Monitor, or Amazon CloudWatch.
Once you have set up the metric and log collection system in Talend Remote Engine and your APM, you can design and organize your dashboards thanks to the information sent from Talend Cloud Management Console to the APM through the engine.
- Job run statuses
- Talend Remote Engine availability
- Resource consumption (CPU, memory, disk usage)
- Workload (number of tasks running)
The following metrics and additional logs are provided by Talend Remote Engine that you can integrate in your APM tool.
- Job run logs
- Remote Engine logs
- Job management logs
- Performance metrics and context of Jobs, Remote Engines, and
- OS resource consumption
- Remote Engine resource consumption and workload
- Job run resource consumption (CPU, memory)
- Job run history (including component information, status, and duration)
- Job run performance (for example, records processed or rejected)
- generic JSON file
- Prometheus-compatible format
Using the feature
- Download and install the new Remote Engine version.
- Set up the Remote Engine on your development environment in Talend Cloud Management Console.
- Install/update Talend Studio to version 7.3 R2020-07 or later.
- Republish your Jobs from the new Studio version to Talend Cloud Management Console.
- Update you tasks to use the latest Jobs versions (if not done automatically).
- Run your Jobs.
Setting up a connection between the APM and the Talend apps depends on which APM you use. You can access some examples on the Talend Community website.