Creating files or database tables - 7.3

Talend Activity Monitoring Console User Guide

Version
7.3
Language
English
Product
Talend Big Data Platform
Talend Data Fabric
Talend Data Management Platform
Talend Data Services Platform
Talend MDM Platform
Talend Real-Time Big Data Platform
Module
Talend Activity Monitoring Console
Content
Administration and Monitoring
Last publication date
2023-01-05

Talend Activity Monitoring Console is an application that allows you to monitor Job executions. The Job executions are monitored using three files or database tables that relate to the following data:

  • collection of logs,

  • component statistics,

  • data flow volumes.

To store this data, you need to create three files or database tables respectively using the schema of the tLogCatcher, tStatCatcher, tFlowMeterCatcher components (present in the Palette of your Talend Studio).

The files can be empty, and their structure will be generated automatically if you use files as datasources to store the activity monitoring information.

Note: The installation of the <AMC> Database and the use of the Talend Activity Monitoring Console in the Studio is optional for Talend ESB and is only required if the tFlowMeterCatcher, tLogCatcher and tStatCatcher components are used. These components can be used in Jobs (for example, ESB Consumer Jobs) for REST and Soap ESB Service Providers. Only the tLogCather is supported for now as the other components are not fully supporting the "Keep listening" environments, and Routes currently can not use these components.

To create the database tables to be used as data sources for the monitoring console, do the following:

  1. Create a database that you name amc, for example.

  2. Launch Talend Studio.

  3. Create a Job that contains three tCreateTable components.

  4. Define the connection parameters to the amc in the three tCreateTable components.

  5. Define the data structure by taking the schema provided in the components: tLogCatcher, tStatCatcher and tFlowMeterCatcher.

    Note that after creating the tStatCatcher, tLogCatcher and tFlowMeterCatcher tables, you have to make sure the moment data type is set to datetime instead of date.

To create these tables in a MySQL database, you can also use the following script:
-- MySQL dump 10.13  Distrib 5.5.24, for Win32 (x86)
--
-- Host: 127.0.0.1    Database: amc
-- ------------------------------------------------------
-- Server version	5.5.24-log

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;

--
-- Current Database: `amc`
--

CREATE DATABASE /*!32312 IF NOT EXISTS*/ `amc` /*!40100 DEFAULT CHARACTER SET latin1 */;

USE `amc`;

--
-- Table structure for table `flowmeter`
--

DROP TABLE IF EXISTS `flowmeter`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `flowmeter` (
  `moment` datetime DEFAULT NULL,
  `pid` varchar(20) DEFAULT NULL,
  `father_pid` varchar(20) DEFAULT NULL,
  `root_pid` varchar(20) DEFAULT NULL,
  `system_pid` bigint(8) DEFAULT NULL,
  `project` varchar(50) DEFAULT NULL,
  `job` varchar(255) DEFAULT NULL,
  `job_repository_id` varchar(255) DEFAULT NULL,
  `job_version` varchar(255) DEFAULT NULL,
  `context` varchar(50) DEFAULT NULL,
  `origin` varchar(255) DEFAULT NULL,
  `label` varchar(255) DEFAULT NULL,
  `count` int(3) DEFAULT NULL,
  `reference` int(3) DEFAULT NULL,
  `thresholds` varchar(255) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `flowmeter`
--

LOCK TABLES `flowmeter` WRITE;
/*!40000 ALTER TABLE `flowmeter` DISABLE KEYS */;
/*!40000 ALTER TABLE `flowmeter` ENABLE KEYS */;
UNLOCK TABLES;

--
-- Table structure for table `logcatcher`
--

DROP TABLE IF EXISTS `logcatcher`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `logcatcher` (
  `moment` datetime DEFAULT NULL,
  `pid` varchar(20) DEFAULT NULL,
  `root_pid` varchar(20) DEFAULT NULL,
  `father_pid` varchar(20) DEFAULT NULL,
  `project` varchar(50) DEFAULT NULL,
  `job` varchar(255) DEFAULT NULL,
  `context` varchar(50) DEFAULT NULL,
  `priority` int(3) DEFAULT NULL,
  `type` varchar(255) DEFAULT NULL,
  `origin` varchar(255) DEFAULT NULL,
  `message` varchar(255) DEFAULT NULL,
  `code` int(3) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `logcatcher`
--

LOCK TABLES `logcatcher` WRITE;
/*!40000 ALTER TABLE `logcatcher` DISABLE KEYS */;
/*!40000 ALTER TABLE `logcatcher` ENABLE KEYS */;
UNLOCK TABLES;

--
-- Table structure for table `statcatcher`
--

DROP TABLE IF EXISTS `statcatcher`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `statcatcher` (
  `moment` datetime DEFAULT NULL,
  `pid` varchar(20) DEFAULT NULL,
  `father_pid` varchar(20) DEFAULT NULL,
  `root_pid` varchar(20) DEFAULT NULL,
  `system_pid` bigint(8) DEFAULT NULL,
  `project` varchar(50) DEFAULT NULL,
  `job` varchar(255) DEFAULT NULL,
  `job_repository_id` varchar(255) DEFAULT NULL,
  `job_version` varchar(255) DEFAULT NULL,
  `context` varchar(50) DEFAULT NULL,
  `origin` varchar(255) DEFAULT NULL,
  `message_type` varchar(255) DEFAULT NULL,
  `message` varchar(255) DEFAULT NULL,
  `duration` bigint(8) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `statcatcher`
--

LOCK TABLES `statcatcher` WRITE;
/*!40000 ALTER TABLE `statcatcher` DISABLE KEYS */;
/*!40000 ALTER TABLE `statcatcher` ENABLE KEYS */;
UNLOCK TABLES;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;

/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;

Now that the Talend Activity Monitoring Console database has been created, you can monitor Job executions either from Talend Studio or from the Monitoring page of Talend Administration Center.

To do so, you simply need to tell Talend Studio and Talend Administration Center about the connection parameters to these tables. For more information, refer to Configuring datasources for the monitoring console.