Robust event processing - 7.3

The primary focus of the Robust event processing is to make sure none of the events are lost or skipped in case the REST or the JMS brokers are down, or the JMS collector is not able to deliver events to the Event Logging server. As shown in the image below, these are the four critical paths where robust event processing is currently implemented.

To start the robust event processing feature, use the following command in the container:

tesb:start-el-dlq

This command will start a camel route which handles the robust re-delivery of messages to the Event Logging server. In case the REST receiver or the JMS broker are down, the REST and JMS senders will send the events to a DLQ (Dead letter Queue) configured in the org.talend.eventlogging.dlq.cfg file. The Robust processing camel route will monitor this DLQ.

Once the events arrive in DLQ, it will ping the JMS broker to check for its availability.

Once the broker is available, it will send the events from the DLQ to the JMS broker.

The configuration for this is as follows:

# JMS connection url
dlq.buffer.jms.url=vm://eventloggingbroker?create=true&broker.use
Jmx=false&broker.persistent=true
# Name of JMS queue
dlq.buffer.jms.queue=event.logging.dlq
# JMS Username
dlq.buffer.jms.username=tadmin
# JMS Password
dlq.buffer.jms.password=tadmin

dlq.timeout=10000

dlq.destination.jms.url=tcp://localhost:61616
dlq.destination.jms.queue=event.logging.server
dlq.destination.jms.username=tadmin
dlq.destination.jms.password=tadmin

As shown in the above image at Step 3, if the events arrive in the JMS broker but the JMS Collector is down, or for some reason not able to pick the events, the events will be stored in the JMS broker until the JMS collector picks them up.

In Step 4, if the JMS Collector is not able to send events to the Event Logging server, it will send the events to a DLQ configured in the org.talend.eventlogging.dlq.cfg file.

The events will be stored in this DLQ permenantly and will not be processed by the JMS Collector or any other component of the Event Logging, or will not be send to the Event Logging server later. It will be the responsibility of the Administrator to manually handle them.

The detailed configuration for the Robust event processing feature is as follows:

# JMS connection url
dlq.buffer.jms.url=vm://eventloggingbroker?create=tr
ue&broker.useJmx=false&broker.persistent=true

# Name of JMS queue
dlq.buffer.jms.queue=event.logging.dlq

# JMS Username
dlq.buffer.jms.username=tadmin
# JMS Password
dlq.buffer.jms.password=tadmin

dlq.timeout=60000

dlq.destination.jms.url=tcp://localhost:61616

dlq.destination.jms.queue=event.logging.server

dlq.destination.jms.username=tadmin
dlq.destination.jms.password=tadmin