Capturing Salesforce object changes in real time with Talend ESB - 7.1

EnrichVersion
7.1
EnrichProdName
Talend Data Fabric
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
EnrichPlatform
Talend Studio
task
Data Governance > Third-party systems > Business applications > Salesforce components
Data Quality and Preparation > Third-party systems > Business applications > Salesforce components
Design and Development > Designing Routes
Design and Development > Third-party systems > Business applications > Salesforce components

Capturing Salesforce object changes in real time with Talend ESB

This article shows how to capture and propagate real time changes happening in Salesforce using Talend ESB.

Salesforce provides on-demand customer relationship management (CRM) software services and many Talend users have chosen Salesforce as their cloud CRM provider.

Some users want to capture real time changes occurring in Salesforce on their CRM data in order to propagate or synchronize those changes on third-party applications. It is somewhat similar to Change Data Capture with Relational Databases, but for Salesforce.

Overview

In order to capture and propagate real time changes happening in Salesforce, you need the following technologies:

Outbound Messaging is part of the workflow rule functionality in Salesforce. Workflow rules watch for specific kinds of field changes and trigger automatic Salesforce actions, such as sending an outbound message to external applications.

To understand this article, you need some knowledge of Salesforce and Outbound Messaging, Talend ESB and Mediation routes, web services technologies, and Apache ActiveMQ.

Environment

This article was written using the commercial version of Talend ESB 6.2 which provides more features than the open source version.

This article is also valid for previous commercial versions of Talend ESB.

Prerequisites

  • You have a valid Salesforce account: a free Developer Edition account can be created for testing purposes on the Salesforce web site at https://developer.salesforce.com/signup.
  • You have installedTalend Studio 6.2 installed.
  • You have a installed Apache ActiveMQ: ActiveMQ is shipped with the Talend ESB installation. Alternatively, it can be downloaded from the Apache ActiveMQ web site.

For more information on the procedure to install and configure both Talend ESB and ActiveMQ, see Talend Help Center.

Architecture of the flow to capture changes

The diagram below shows the architecture for this example:

  1. Salesforce User updates a Salesforce object.
  2. Salesforce sends an outbound message to Talend ESB to notify object update.
  3. Talend ESB persists the notification message in an ActiveMQ queue.
  4. Talend ESB receives acknowledgement of successful persistence in ActiveMQ.
  5. Talend ESB answers with a successful acknowledgement back to Salesforce.

Configuring Salesforce to send changes notifications to Talend

This section explains how to setup Salesforce to send outbound notifications on objects changes.

It focuses on the standard Salesforce Account object, but the procedure remains the same for all other Salesforce standard or custom objects.

Procedure

  1. Connect to your existing Salesforce account, or create a free Developer Edition account if you do not have one.
  2. Navigate to Build > Create > Workflow & Approvals > Workflow Rules in the left panel.

    Alternatively, you can search for workflow rules in the search area.

    If you do not see or do not have access to this menu, check your account permissions with your Salesforce Administrator.
  3. Click the Workflow Rules menu to open a page where you can manage your organization’s workflow rules.
  4. Click New rule to create a workflow rule which will be triggered every time an account is created or updated in your Salesforce organization.
  5. Select the object on which the workflow rule will apply. In this case, select Account, then click Next.

    This should also work with any other Salesforce standard or custom object.

  6. Configure the workflow rule. Since the workflow must be triggered each time an account is created or updated, configure the rule as follows:
    Field Value
    Rule name UpdateOrCreateAccountRule
    Rule criteria Account name not equal to ""

    Use double quotes in the rule criteria to indicate that Account name should not be empty.

    Since the account name is mandatory for Account objects, the clause Account name not equal to "" should be always true.

  7. Click Save & next to open a page with the rule criteria you defined.
  8. Click Add Workflow Action, then choose New Outbound Message to send a new Outbound Message to Talend ESB.

    This option is used to specify one or more actions to be executed by the workflow when triggered. You have the choice between Immediate Workflow Actions and Time-dependant Workflow Actions. In this case, choose to create an Immediate Worflow Action to send a new outbound message to Talend ESB.

  9. Configure Outbound Message as follows.
    Field Value
    Name Account Outbound Message
    Unique Name This field should be auto-filled. You can also update it by entering a unique name.
    Endpoint URL http://52.28.xxx.xxx:8765/services/account
    This is the endpoint on which Talend ESB is listening for Salesforce Outbound Messages.
    Note: At this stage, you can put any dummy IP and come back later to replace it with the correct endpoint after the Talend web service has been developed and deployed. If your are using an EC2 instance, you need to update your instance security group to allow inbound traffic from Salesforce.
    User to send as Keep the default user.
    Account fields to send Choose any available fields that you want to send along with the outbound message, according to your use case. Good practice would be to send only the ID and the minimum relevant information to keep the notification message as light as possible.
  10. Click Save.
    You should see the following screen.
  11. Click Done, then click Activate on the Workflow Rule Detail page.

    The flag Active is checked, indicating that the workflow is activated.

  12. Navigate to Administer > Security Controls > Remote Site Settings and click Remote Site Settings.

    By default, Salesforce does not authorize outbound traffic towards remote hosts. In order to send outbound messages to Talend ESB, you need to add to the allowlist the remote URL of Talend ESB. To do this, you have to add the remote URL information on the allowlist sites.

  13. Click New Remote Site to add a new remote site and configure it as follows.
    Field Value
    Remote Site Name WsTalendAccountNotificationServer
    Remote Site URL http://52.28.xxx.xxx:8765/
    This is the web service URL with the IP Address of the server hosting the web service and the PORT on which the web service is listening. It is not mandatory to put the relative path, for example services/account. Choose not to mention the relative path in order to be able to handle multiple web services on the same host.
  14. Click Save.

Results

Salesforce is now setup and ready to send notifications to Talend ESB every time an account is created or updated in your Salesforce organization. The next step is to implement the Talend web service which will be listening for these notifications coming from Salesforce.

It is important to handle these upcoming notifications in a reliable manner. You must not lose any of these notifications once you receive and acknowledge them, even if a failure or crash occurs on Talend ESB, because Salesforce is not going to send again any acknowledged notification.

Downloading the Salesforce notification WSDL document

This section explains how to download the Salesforce notification WSDL (Web Services Description Language) you need to use to implement the web service with Talend.

Procedure

  1. Navigate to Build > Create > Workflow & Approvals > Workflow Rules and click Workflow Rules in the left menu.
  2. Click the workflow rule UpdateOrCreateAccountRule on the All Workflow Rules page.
    The details of the workflow rule are displayed.
  3. Click Account Outbound Message.
  4. Click Click for WSDL next to the Endpoint WSDL label. This should open a new tab on your browser displaying the WSDL.
  5. Download and save the WSDL file on your computer as workflowOutboundMessage.wsdl for example.

Configuring Talend ESB to listen for and process Salesforce notifications using the Notification WSDL

This section explains how to use Talend ESB to implement the web service which will be listening for notifications messages coming from Salesforce.

Procedure

  1. Download and install Talend ESB and Apache ActiveMQ.
    ActiveMQ is shipped with the Talend ESB installation if you subscribe to the commercial version.
  2. Start ActiveMQ.
    If you are not familiar with it, see Talend Help Center for more information on how to install and start ActiveMQ.
  3. Launch your Talend Studio, then open the Mediation perspective.
    Starting from Talend 6.2, the Mediation and Integration perspectives are merged in Talend Studio. Therefore, you need to open the Integration perspective.
  4. Create a new mediation Route named salesforce_account_synchro for example.
  5. Drag and drop the following components to the design workspace: cMQConnectionFactory, cCxf, cExchangePattern, cJms, cSetBody, cLog.

    The Route should look like this before you connect the components to each other:

  6. Right-click cCxf, select Row > Route from the contextual menu and click cExchangePattern.
  7. Repeat the previous step to connect cExchangePattern to cJms, cJms to cSetBody and cSetBody to cLog.
  8. Rename the components as follows for more clarity.
    Component Name
    cCxf webservice_salesforce.notifcation.account
    cExchangePattern InOnly
    cJms activemq_salesforce.notification.account
    cSetBody set_webservice_response
    cLog log_response
    cMQConnectionFactory activemq_connection_factory

Results

The Route salesforce_account_synchro should now look like this: