Available metrics for monitoring - Cloud

Talend Remote Engine User Guide for Linux

EnrichVersion
Cloud
EnrichProdName
Talend Cloud
EnrichPlatform
Talend Remote Engine
task
Design and Development
Installation and Upgrade

The following metrics are provided by Talend Cloud Management Console through Talend Remote Engine and can be integrated in your Application Performance Monitoring (APM) tool to observe your Jobs.

The metrics below are captured only when a Job run is started in Talend Cloud Management Console.

Process metrics are sent by Prometheus by default.

Operating system metrics

The following metrics are present and enabled by default. They cannot be disabled.

Parameter Description
os_cpu_cores_total Number of logical CPUs available
os_cpu_usage Operating system's real-time CPU usage
os_system_load_average Average number of processes that are executed by the CPU calculated over a given period of time
os_memory_bytes_total Amount of actual physical memory (in bytes)
os_memory_bytes_available Amount of physical memory currently available (in bytes)
os_filestore_bytes_total Amount of actual physical filestore (in bytes)
os_filestore_bytes_available Amount of actual physical filestore currently available (in bytes)

Business metrics

The following metrics are present and enabled by default. They cannot be disabled.

Parameter Category Description
component_connection_rows_total Components Total number of rows treated by a Job's component (generated for components that are managing rows)
component_start_time_seconds Components Time stamp indicating the component start time (any component in the Job)
Note: Minimum requirement is Talend Studio 7.3 R2020-11 patch and Talend Remote Engine v2.9.4 (or later versions).
component_execution_duration_milliseconds Components Amount of time it takes to execute a Job component (generated for components that are managing rows)
job_execution_duration_seconds Job Amount of time it takes to execute a Job
job_start_time_seconds Job Time stamp indicating the execution start time
re_execution_count_max Remote Engine Maximum number of tasks a Remote Engine can execute in parallel
re_execution_count_current Remote Engine Number of tasks currently deployed and executed by the Remote Engine
re_logs_directory_bytes_used Remote Engine Size of the log directory of the Remote Engine
re_jobserver_directory_bytes_used Remote Engine Size of the JobServer directory in the Remote Engine

To determine when the Job execution ends, add the values of the job_execution_duration_seconds and job_start_time_seconds parameters. Similarly, to determine when the component execution ends, add the values of the component_start_time_seconds and component_execution_duration_milleseconds.

JMX metrics

Default JMX metrics
Parameter Description
jvm_memory_bytes_used Used bytes of a given JVM memory area
jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area
jvm_memory_bytes_max Max (bytes) of a given JVM memory area
jvm_memory_bytes_init Initial bytes of a given JVM memory area
jvm_start_time_milliseconds Time stamp indicating the process start time
jvm_process_cpu_time Total user and system CPU time spent (in seconds)
jvm_process_cpu_load Recent CPU usage

JVM arguments not set by default can be added manually to the <RemoteEngineInstallationDirectory>/etc/org.talend.ipaas.rt.observability.cfg with the omc.jvm.allowlist= parameter, separated by commas.

Optional JMX metrics
Parameter Description
jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool (only updated after GC, not continuously)
jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool
jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool
jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool
jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool
jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool
jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool
jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool
jvm_threads_current Current thread count of a JVM
jvm_threads_daemon Daemon thread count of a JVM
jvm_threads_peak Peak thread count of a JVM
jvm_threads_started_total Started thread count of a JVM
jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
jvm_threads_state Current count of threads by state
jvm_info JVM version info
jvm_classes_loaded The number of classes that are currently loaded in the JVM
jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
jvm_gc_collection_seconds Time spent in a given JVM garbage collector (in seconds)