How to configure Event Logging in Talend ESB to use the JMS interface instead of the default REST interface

author
Irshad Burtally
EnrichVersion
6.4
6.3
6.2
6.1
6.0
EnrichProdName
Talend MDM Platform
Talend Data Services Platform
Talend ESB
Talend Data Fabric
Talend Real-Time Big Data Platform
task
Administration and Monitoring
EnrichPlatform
Talend Runtime
Talend Log Server
Talend ESB
Talend Studio
Talend Administration Center

How to configure Event Logging in Talend ESB to use the JMS interface instead of the default REST interface

The Event Logging feature within the Talend ESB product allows users to collect events across distributed containers, and provides the functionalities to index and search through these events via a Web User Interface. In addition to the pure collection of events, the Event Logging feature supports custom processing (for example: custom filtering, customer data enrichment and reduction), aggregation, signing and also server-side custom pre- and post-processing of events. These events can be used in various ways, for example:
  • To send events as a post processing step to an Intrusion Detection System
  • To send events to any other kind of potential higher level log processing and management system.
  • To send events to alerting and monitoring systems

Source: Documentation: Event Logging

Architecture

There are two ways to configure and implement the Event Logging feature in Talend ESB. Even if most of the components are used in both configuration, the biggest difference is the communication protocol used between the agent and the server components of the Event Logging architecture.

The two architecture styles you can implement are:

  • REST based architecture: Communication between agent and server is provided by a REST service.
  • JMS based architecture: all events are stored in a JMS Queue before consumption by the server

Components

These two architectures use the following components:

Main components:

  • Event Logging - Server : Retrieves events from the collector and performs processing and persistence of events, with support for Event Database (RDBMS), for example into Service Activity Monitoring Database (Service Activity Monitoring Events only) and Elastic Search with some custom pre- and post-processing extension points.
  • Event Logging - Agent : Receives events from listeners, buffers, processes and sends them to a final destination using one of the configured Event Logging Appenders.

Communication components:

  • Event Logging - Sender : Sends events to the Event Logging backend. A JMS Appender and a REST Appender for the Event Logging RESTService are provided by default to write into a local log file or to a local Elastic Search instance, for example. Developers can add and configure their own appenders, if required.
  • Event Logging - Service : It is a RESTful Service to collect events sent from the Event Logging REST Appender and to retrieve events from the backend. Events are stored in the Event Database only (since Talend ESB 5.4).

  • Event Logging - Collector : Consists of routes which listen on endpoints to receive events for further server-side processing by the Event Logging Server. By default, a Standard Collector which exposes a fixed Direct-VM endpoint, and a JMS Collector which takes Event out of a JMS Queue are provided.

Listeners:

  • Event Logging - Listener : It gets log information and passes it to the Agent for further local processing. Talend ESB supports four default listeners:
    • Log Listener to get all Pax Logging log events.
    • OSGi Event Listener to get all OSGi Events on Talend Runtime.
    • SAM Listener to get Service Activity Monitoring Event (via SOAP) from the Service Activity Monitoring Agent.
    • Locator Listener to get the Locator events.

REST based architecture

In this architecture, we have two parts :

  • Agent (Talend Runtime) : which retrieves all events coming from listeners and calling a REST service on the server side.
  • Server (Talend Runtime) : it hosts a REST service called by the agent and it forwards events to a database and/or Elasticsearch.

Applications used in this architecture are :

Server Side

  • tesb:start-el-server - starts the Event Logging Server.
  • tesb:start-el-restservice - starts the REST Service and the Direct Receiver.
  • Database Datasource - connectivity to a database to store events

Agent Side

  • tesb:start-el-agent - starts the Event Logging Agent.
  • tesb:start-el-restsender - starts the REST Sender.
  • tesb:start-el-loglistener - starts the Log Listener.
  • tesb:start-el-osgilistener - starts the OSGi Listener.
  • tesb:start-el-samlistener - starts the SAM Listener.
  • tesb:start-el-locatorlistener - for the Locator Listener

JMS based architecture

In this architecture, we have three parts :

  • Agent (Talend Runtime) : which retrieves all events coming from listeners and pushes them to a JMS queue.
  • ActiveMQ : stores events before being consumed by the server
  • Server (Talend Runtime) : Reads events from the JMS queue and forwards them to a database and/or Elasticsearch.

Applications used in this architecture are :

Server Side

  • tesb:start-el-server - starts the Event Logging Server.
  • tesb:start-el-jmsreceiver - starts the JMS Receiver.
  • Database Datasource - connectivity to a database to store events

Agent Side

  • tesb:start-el-agent - starts the Event Logging Agent.
  • tesb:start-el-jmssender - starts the JMS Sender.
  • tesb:start-el-loglistener - starts the Log Listener.
  • tesb:start-el-osgilistener - starts the OSGi Listener.
  • tesb:start-el-samlistener - starts the SAM Listener.
  • tesb:start-el-locatorlistener - for the Locator Listener

Environment

For this tutorial, the installation will require at least the following 3 Talend components (readers are expected to know how to install these components already) :

  • Talend Administration Center: This components will install the Talend Administration together with the Nexus and Kibana Application. The Kibana UI is used to browse events sent to the Elastic Search cluster.
  • Talend Log Server: This log server is based on logstash and is a key component of the ELK stack.
  • Talend Runtime: This is the Talend ESB Runtime which can host webservices. The Talend Runtime also host the Event Logging applications.

In this article, we are using only one runtime as a agent and server. To use two different servers, we need to configure the host and port of the server runtime and ActiveMQ server that will receive the events through JMS.

Installation & Configuration

Server side configuration

Connect to the Talend Runtime using a ssh client:

ssh -p 8101 tadmin@localhost

Or start it with trun command:

./trun

Derby Database

Derby Database

First, we need to install the Derby database which will be used to store the events. The Derby database is appropriate for a test environment only. For production environments, it is recommended to use a proper relational database like MySQL. Refer to the Talend Infrastructure Services for thelist of supported databases for the Event Database

Installation

To install a Derby database embedded in the runtime, we need to run the following command:

karaf@trun()> feature:install tesb-datasource-derby tesb-derby-starter
  • tesb-datasource-derby : install the derby datasource
  • tesb-derby-starter : starts the Derby server

We can check the installation with :

karaf@trun()> feature:list | grep -i derby

We should see the following:

pax-jdbc-derby 			| 0.9.0 | 	| Uninstalled 	| org.ops4j.pax.jdbc-0.9.0 | Provides JDBC Derbi Embedded DataSourceFactory 
pax-jdbc-derbyclient 	| 0.9.0 | 	| Uninstalled 	| org.ops4j.pax.jdbc-0.9.0 | Provides JDBC Derbi Client DataSourceFactory 
tesb-datasource-derby 	| 6.3.1 | x | Started 		| tesb-6.3.1 | 
tesb-derby-starter 		| 6.3.1 | x | Started 		| tesb-6.3.1 | 
karaf@trun()> list | grep -i derby 
275 | Active | 80 | 6.3.1 		| Talend ESB :: SAM :: Derby Starter 276 | Active | 80 | 6.3.1 | Talend ESB :: SAM :: Datasource :: Derby 
278 | Active | 80 | 10.8.1.2_1 	| Apache ServiceMix :: Bundles :: derby

Configuration

If needed, we can edit the configuration file etc/org.talend.esb.datasource.derby.cfg to further tweak the Derby Datasource settings.

Event Logging Server

Event Logging Server

Installation

To install the Event Logging server, we need to run the following command:

karaf@trun()> tesb:start-el-server

We can check the installation status with the following command:

karaf@trun()> list |grep -i event

We should see a list similar to the following:

281 | Active | 80 | 6.3.1 | Talend :: ESB :: Event Logging :: Common 
282 | Active | 91 | 6.3.1 | Talend :: ESB :: Event Logging :: Server 
294 | Active | 91 | 6.3.1 | Talend :: ESB :: Event Logging :: Elastic Search :: Client

Configuration

We can further tweak the configuration for the Event Logging Server in the configuration file etc/ org.talend.eventlogging.server.cfg , if needed.

For instance, we can define if we want to store events in a database, and which events to store as per the sample configuration file below.

etc/org.talend.eventlogging.server.cfg

# Default behaviour for saving events in Event database 
persistence.event.db.active.default=true 
# Behaviour for saving events with audit flag in Event database 
persistence.event.db.active.audit=true 
# Behaviour for saving events with sam category in Event database 
persistence.event.db.active.sam=false 
# Default behaviour for saving events in SAM database 
persistence.sam.db.active.default=false 
# Behaviour for saving events with SAM category in Sam database 
persistence.sam.db.active.sam=true

Within the same configuration file, we can configure the datasource to use:

etc/org.talend.eventlogging.server.cfg

# Datasource for Event Logging database 
event.logging.db.datasource=ds-derby 
# Dialect for Event Logging database 
event.logging.db.dialect=derbyDialect 
# Datasource for SAM database 
sam.db.datasource=ds-derby 
# Dialect for SAM database 
sam.db.dialect=derbyDialect

We can further adapt the ElasticSearch configuration:

etc/org.talend.eventlogging.server.cfg

# Set to "true" to enable sending events to elasticsearch (if tesb-el-elasticsearch feature is installed)
elasticsearch.available=true
# Host address where elasticsearch instance is started
elasticsearch.host=localhost
# Port of elasticsearch server instance
elasticsearch.port=9200
# Elasticsearch is secured or not
elasticsearch.secured=false
# Optional, Trust store with SSL certificate for secured elasticsearch (any -> no trust check)
# By default the base dir of the truststore is $tesb.home
elasticsearch.ssl.truststore=any
# Optional, Trust store password (none -> no password)
elasticsearch.ssl.truststore.password=none
# Optional, Certificate hostname accepted independent from the server URL (any -> no hostname check)
elasticsearch.ssl.pass.hostname=any
# Username and password for secured elasticsearch
elasticsearch.username=es_admin
elasticsearch.password=es_admin
# Name of index that will be used in elasticsearch
elasticsearch.indexname=talendesb
# Type of the index
elasticsearch.indextype=ESB 

If you configure another database/datasource, by default, the Event Logging Server will start with a Derby configuration. You will need to change it to match your target database in order for it to work.

Troubleshooting

You can encounter various errors while configuring the Event Logging Server. The followings are the 2 most common issues:

  • Derby installation is not correct. You need to check the Derby datasource configuration to make sure it is correct. Stop and restart the services for the tesb-datasource-derby and tesb-derby-starter

    2017-03-06 16:09:44,546 | WARN  | pool-2-thread-1  | EventDBInitializer               | 282 - event-logging-server - 6.3.1 | Exception during retrieving DB schema. Tables and indexes will not be created. java.sql.SQLException: Datasource 'ds-derby' is not available.
  • This error is the result of a problem of the installation of tesb-derby-starter

    2017-03-06 16:16:27,420 | WARN  | pool-2-thread-1  | EventDBInitializer               | 306 - event-logging-server - 6.3.1 | Exception during retrieving DB schema. Tables and indexes will not be created. java.sql.SQLNonTransientConnectionException: java.net.ConnectException : Error connecting to server localhost on port 1527 with message Connection refused (Connection refused).

JMS Receiver

JMS Receiver

Installation

To install the JMS receiver, we need to run the following command:

karaf@trun()> tesb:start-el-jmsreceiver

We can check the installation status with the following command:

karaf@trun()> list |grep -i jms

We should see an output similar to:

164 | Active | 50 | 2.17.3 | camel-jms 298 | Active | 92 | 6.3.1 | Talend :: ESB :: Event Logging :: Collector :: JMS

Configuration

Edit the configuration file etc/org.talend.eventlogging.collector.jms.cfg and change it based on the architecture requirements. You can adjust port, username, password, etc. in the file.

etc/org.talend.eventlogging.collector.jms.cfg

# In the part of the Server we can receive events via jms 
# JMS Collector able to read/get events from a JMS Broker 
# JMS connection url 
collector.jms.url=tcp://localhost:61616 
# Name of JMS queue 
collector.jms.queue=event.logging.server 
# JMS Username 
collector.jms.username=tadmin 
# JMS Password 
collector.jms.password=tadmin 
# Name of JMS collector queue 
collectorjms.dlq.buffer.jms.queue = collectorjms.dlq

Agent Side Configuration

Agent Side Configuration

We need to configure the following 3 agent side applications :

  • JMS Sender
  • Event Logging Agent
  • Listeners

JMS Sender

Installation

To install the JMS sender, we need to run the following command:

karaf@trun()> tesb:start-el-jmssender

We can check the installation status with the following command:

karaf@trun()> list |grep -i jms

We will see an output similar to the following:

164 | Active | 50 | 2.17.3 | camel-jms 
298 | Active | 92 | 6.3.1 | Talend :: ESB :: Event Logging :: Collector :: JMS 
300 | Active | 95 | 6.3.1 | Talend :: ESB :: Event Logging :: Sender :: JMS    

Configuration

Edit the configuration file etc/org.talend.eventlogging.sender.jms.cfg and change it based on the architecture.

etc/org.talend.eventlogging.sender.jms.cfg

 # JMS connection url 
sender.destination.jms.url=tcp://localhost:61616 
# Name of JMS queue 
sender.destination.jms.queue=event.logging.server 
# JMS Username 
sender.destination.jms.username=tadmin 
# JMS Password 
sender.destination.jms.password=tadmin

Event Logging Agent

Installation

To install the Event Logging agent, we need to run the following command:

karaf@trun()> tesb:start-el-agent

We can check the installation status with the following command:

karaf@trun()> list |grep -i event

We will see an output similar to the following:

286 | Active | 80 | 6.3.1 | Talend :: ESB :: Event Logging :: Common 
287 | Active | 91 | 6.3.1 | Talend :: ESB :: Event Logging :: Server 
297 | Active | 91 | 6.3.1 | Talend :: ESB :: Event Logging :: Elastic Search :: Client 
298 | Active | 92 | 6.3.1 | Talend :: ESB :: Event Logging :: Collector :: JMS 
300 | Active | 95 | 6.3.1 | Talend :: ESB :: Event Logging :: Sender :: JMS 
301 | Active | 97 | 6.3.1 | Talend :: ESB :: Event Logging :: Agent

Configuration

We can change the agent name in etc/org.talend.eventlogging.agent.cfg

etc/org.talend.eventlogging.agent.cfg

# Unique agentID (string) 
agent.id=agent1

By default, the agent is configured to use the REST communication. We need to change the configuration to enable it to use JMS:

etc/org.talend.eventlogging.agent.cfg

# The Event Logging Sender is responsible to get the messages from the processing part of the Agent 
# and send them to the final destination which can be either a JMS Broker Queue or the Event Logging Collector Service 
agent.sender.destination.default=eventlogsenderjms 
#agent.sender.destination.audit=eventlogsenderjms 
#agent.sender.destination.security=eventlogsenderjms

Listeners

Installation

To install the listeners, we need to run the following commands:

karaf@trun()> tesb:start-el-loglistener 
karaf@trun()> tesb:start-el-osgilistener 
karaf@trun()> tesb:start-el-samlistener

We can check the installation status with the following command:

karaf@trun()> list |grep -i listener

We will see an output similar to the following:

305 | Active | 98 | 6.3.1 | Talend :: ESB :: Event Logging :: Listener :: Log 
307 | Active | 99 | 6.3.1 | Talend :: ESB :: Event Logging :: Listener :: OSGi Event 
309 | Active | 98 | 6.3.1 | Talend :: ESB :: Event Logging :: Listener :: SAM Event

Demo Sample Web Service

To test our Event Logging configuration with JMS interface, we are going to create a simple REST service (route). You can download this zip file DemoWS.zip in the Downloads tab on the left panel of this page and import it in your studio. The screenshot below shows the DemoWS REST service.

This service is configured to use Service Activity Monitor and Service Locator as shown in the options below.

In Talend Administration Center, we can create an ESB Conductor task to deploy this service as shown below:

Once deployed and started, the service will appear as follows in the ESB Conductor:

We can also check that the service is properly started in the Talend Runtime container by issuing the following command:

karaf@trun()> list | grep -i restdemo 

We will see an output as follows:

320 | Active | 80 | 0.1.0.SNAPSHOT | RestDemo

We can also confirm that the service is available at http://localhost:8040/services with the wsdemo endpoint:

Dashboards

The Talend Logging UI, based on Kibana, provides several dashboards. The 3 dashboards are:

  • ESB Events
  • ESB Locator Endpoints
  • ESB SAM (Service Activity Monitoring)

ESB Events

This dashboard displays all events coming from the OSGI Listener and the Log Listener, as shown in the example below.

ESB Locator Endpoints

This dashboard displays all the events for the ESB Service Locator, as shown below:

ESB SAM

This dashboard displays all events from the Service Activity Monitoring, including request and response messages from the webservices, as shown in the screenshots below:

This screenshot shows a list of all SAM events: