Agent - Receiver part - 7.1

Talend ESB Infrastructure Services Configuration Guide

Version
7.1
Language
English (United States)
Product
Talend Data Fabric
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for ESB
Talend Real-Time Big Data Platform
Module
Talend ESB
Talend Runtime
Content
Design and Development
Installation and Upgrade

About this task

The Agent Receiver part is responsible for getting the messages from the local listeners and doing a minimal processing before the event gets stored in the local buffer.

Procedure

  1. The Event Logging listener sends each event via in-memory synchronous communication (direct-vm: component) to the Agent using a fixed Route ID eventloggingagent.
  2. The user can optionally configure a custom Route at this early stage to filter, shorten, enrich metadata of the log event, as soon as the listener receives it.
    The custom Route must be deployed on the same JVM or on the same Container and must expose a unique Route ID via the direct-vm: component. The custom Route has full access to the Event data. The configuration part of the org.talend.eventlogging.agent.cfg file in the agent configuration is as follows:
    agent.receiver.custom.routeid.default=myCustomReceiverRoute
    agent.receiver.custom.routeid.audit=myCustomReceiverRoute
    agent.receiver.custom.routeid.security=myCustomReceiverSecurityRoute
    The agent.receiver.custom.routeid. is the fixed part followed by the event category. default and audit are reserved categories and can not be used as normal category names. Default will be used if no specific mapping for the current event category is found and the default custom Route will be called. If the Event is already marked as an audit event and if an audit custom Route is defined, this one will be called. Even though we do not limit what the user can do at this stage with the custom Route, it is strongly advised that these early processing Routes are designed for maximum performance as we are still in a synchronous processing with the listener at this stage. By default, no custom Route for none of the categories is defined. The above example will be commented out, for reference, with an # character in our default agent configuration file, and no custom Route will be called by default.
  3. The data enrichment component in the agent will add the minimal required system data to the event before it is stored for further processing in the local buffer.
    At this stage, if one of the following attributes of the Event Data Structure is not already filled (for example by the listener, or the customer Route), the Data Enrichment will fill the values as follows:
    Attribute Value Remark
    eventuuid string (UUID) If empty, it will be generated within the agent.
    agentid string (Unique agentID ) It is retrieved from the agent configuration file. Example: property agentID=agent1.
    agent_timestamp timestamp

    Local machine date and time (converted to UTC time). See the event structure.

    Example: 2013-07-23T08:45:30.453Z

    hostname string (hostname)

    Hostname of the current machine.

    If the hostname cannot be resolved via the Java API, the IP address will be used as fallback.

    processid string (processID) Current JVM process ID.
    audit boolean (default=false, if not set at this stage)  
    auditsequenceno long - unique sequence number for audit event A unique sequence ID will be assigned only if the event is marked as audit. The sequence ID is a long number that starts with 1 and is incremented for each new audit event from this agent. To avoid any repetition, agentid and auditseqenceno should be shown on the tables in the Backend Event Database with no gaps. The auditsequenceno will be persisted in a local file and saved each time a new number is used.
    correlationid string If empty, an attribute will be found via the event.correlationid.map.attribute configuration parameter in the eventlogging_custominfo attribute collection.
    subject string If empty, the event.subject.map.attribute configuration will look in the in the eventlogging_custominfo attribute collection to find an attribute.
    The configuration parts for this step are defined in the org.talend.eventlogging.agent.cfg as follows:
    # Unique agentID (string)
    agentid=agent1
    
    # The location of file system used to persist audit sequence
    agentsequencedir=./data/audit-sequence
    
    # Take correlation from customInfo using this key
    event.correlationid.map.attribute=CorrelationID
    
    # Used to map an attribute to the Subject property in the Event. User can 
    # provide this value in his logs as attributes (e.g. for log events as 
    # MDC attribute)
    event.subject.map.attribute=Subject
  4. The Router will send the Event to the appropriate configured local buffer. Two buffers are supported in Talend ESB:
    • jms, a JMS-based buffer (for the time being, only ActiveMQ is supported) with or without persistent queue, intended to be a local ActiveMQ Broker for the local buffer.

    • memory, use of the in-memory queue, as provided by the VM (component of Apache Camel).

    The router configuration is defined in the agent as follows:
    # Receiver buffer config
    # For 'agent.buffer.jms.queue' (and only for it) its possible to
    # reuse other property values defined in this file by putting
    # property name between ${ and }. For example:
    #     agentid=some-agent-id
    #     agent.buffer.jms.queue=event.logging.${agentid}.cache
    # will be interpreted by EL as 'event.logging.some-agent-id.cache'
    agent.buffer.jms.url = vm://eventloggingbroker?create=true&broker.useJ
    mx=false&broker.persistent=true
    agent.buffer.jms.queue =event.logging.${agentid}.cache
    agent.buffer.jms.username=tadmin
    agent.buffer.jms.password=tadmin
    
    # By default, all events will be sent via a memory buffer but the user
    # can easily change this and use jms instead for audit event and for
    # other event categories.
    agent.receiver.buffer.default=memory
    #agent.receiver.buffer.audit=jms
    #agent.receiver.buffer.security=jms
    By default, all events will be sent via a memory buffer but the user can easily change this and use jms instead for audit events and for other event categories. For example, in the example above, if the user uncomments agent.receiver.buffer.security=jms, all events of the security category, that correspond to the value after agent.receiver.buffer. (static part), will also be stored via the jms buffer.