Robust event processing - 7.1

Talend ESB Infrastructure Services Configuration Guide

EnrichVersion
7.1
EnrichProdName
Talend Data Fabric
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for ESB
Talend Real-Time Big Data Platform
EnrichPlatform
Talend ESB
Talend Runtime
task
Design and Development
Installation and Upgrade

The primary focus of the Robust event processing is to make sure none of the events are lost or skipped in case the REST or the JMS brokers are down, or the JMS collector is not able to deliver events to the Event Logging server. As shown in the image below, these are the four critical paths where robust event processing is currently implemented.

To start the robust event processing feature, use the following command in the container:

tesb:start-el-dlq

This command will start a camel Route which handles the robust re-delivery of messages to the Event Logging server. In case the REST receiver or the JMS broker are down, the REST and JMS senders will send the events to a DLQ (Dead letter Queue) configured in the org.talend.eventlogging.dlq.cfg file. The Robust processing camel Route will monitor this DLQ.

Once the events arrive in DLQ, it will ping the JMS broker to check for its availability.

Once the broker is available, it will send the events from the DLQ to the JMS broker.

The configuration for this is as follows:

# JMS connection url
dlq.buffer.jms.url=vm://eventloggingbroker?create=true&broker.use
Jmx=false&broker.persistent=true
# Name of JMS queue
dlq.buffer.jms.queue=event.logging.dlq
# JMS Username
dlq.buffer.jms.username=tadmin
# JMS Password
dlq.buffer.jms.password=tadmin

dlq.timeout=10000

dlq.destination.jms.url=tcp://localhost:61616
dlq.destination.jms.queue=event.logging.server
dlq.destination.jms.username=tadmin
dlq.destination.jms.password=tadmin

As shown in the above image at Step 3, if the events arrive in the JMS broker but the JMS Collector is down, or for some reason not able to pick the events, the events will be stored in the JMS broker until the JMS collector picks them up.

In Step 4, if the JMS Collector is not able to send events to the Event Logging server, it will send the events to a DLQ configured in the org.talend.eventlogging.dlq.cfg file.

The events will be stored in this DLQ permenantly and will not be processed by the JMS Collector or any other component of the Event Logging, or will not be send to the Event Logging server later. It will be the responsibility of the Administrator to manually handle them.

The detailed configuration for the Robust event processing feature is as follows:

# JMS connection url
dlq.buffer.jms.url=vm://eventloggingbroker?create=tr
ue&broker.useJmx=false&broker.persistent=true

The address of the JMS broker which is used to store the events in the DLQ. The default configuration uses the embedded broker which is also used for internal event jms buffering at the agent side.

# Name of JMS queue
dlq.buffer.jms.queue=event.logging.dlq

DLQ queue name

# JMS Username
dlq.buffer.jms.username=tadmin
# JMS Password
dlq.buffer.jms.password=tadmin

DLQ queue Active MQ broker credentials

dlq.timeout=60000

Timeout in milliseconds in which the robust event processing feature will re-deliver events from DLQ queue to destination.

dlq.destination.jms.url=tcp://localhost:61616

Destination (Event Logging server) queue broker address

dlq.destination.jms.queue=event.logging.server

Destination (Event Logging server) queue name

dlq.destination.jms.username=tadmin
dlq.destination.jms.password=tadmin

Destination (Event Logging server) broker credentials