Amazon Web Services (AWS) - How to deploy a highly-available and scalable Architecture with Talend ESB and AWS Auto Scaling - 7.3

EnrichVersion
7.3
EnrichProdName
Talend Data Fabric
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for ESB
Talend Real-Time Big Data Platform
EnrichPlatform
Talend ESB
task
Administration and Monitoring
Deployment

Amazon Web Services (AWS) - How to deploy a highly-available and scalable Architecture with Talend ESB and AWS Auto Scaling

In this article, we will show how to deploy a scalable and dynamic architecture using Talend ESB in AWS platform.

We will be using an interesting feature of AWS named AWS Auto Scaling. This will be a step-by-step guide to build this scalable architectucre based on Auto Scaling Group and Talend ESB.

Introduction

Elasticity is one of the major benefits customers are looking for when moving their architecture into the cloud.In cloud computing, Elasticity can be defined as the ability of an architecture to automatically adapt its resources or computing capability to the workload or traffic variations.

Talend can be deployed in Amazon Web Services to support an elastic and scalable architecture which can grow and shrink automatically as required while making optimal use of the AWS resources and minimizing cost.

Architecture

The diagram below shows the architecture for the design we will put in place with Auto Scaling Group and Talend ESB.

Assumptions

  • Amazon Web Services (AWS):
    • You should be familiar with AWS platform since this article does not take a deep dive into details regarding Administration and Management of AWS services.

      For more information about all the AWS functionalities that Talend provides, see the article Amazon Web Services (AWS) - Getting Started.

    • You should also have full access to the main AWS services described in the Prerequisites section below.

  • Talend:
    • You should be familiar with Installation and Management of Talend ESB since this article does not take a deep dive into details regarding these points.

Environment

This demonstration is based on AWS Cloud Platform and Talend ESB 6.2.

Prerequisites

  1. A valid AWS account with full access to following services:
  2. Talend ESB (Commercial Edition) - https://fr.talend.com/products/application-integration.

Amazon Web Services (AWS) - How to deploy a highly-available and scalable Architecture with Talend ESB and AWS Auto Scaling

Choose an AWS Region

Procedure

  1. Connect into your AWS console.
  2. Once connected, choose an AWS region.
    Generally, AWS customers may choose regions which provide the best network latency for their applications. Let's choose Ireland region for this demonstration.

Create a Virtual Private Cloud (VPC)

Let's create a VPC to host our application. As per AWS documentation, VPC will help us provision a logically isolated section of the Amazon Web Services (AWS) cloud where AWS resources can be launched. It also gives a full control of network configuration.

As described in the Architecture diagram, our bespoke VPC spans across two Availability Zones within the region to add more high-availability.

Create then Setup a VPC with two public subnets.

Create a Virtual Private Cloud (VPC)

Procedure

  1. Open the VPC console.
  2. Click Start VPC wizard.
  3. Select VPC with a Single Public Subnet.
    Note: For the sake of simplicity, we will limit to Public subnets. In real-life uses cases, you would certainly use Private subnets for hosting database and sensitive application data. For more information on Public/Private subnets and VPC, read the AWS documentation.
  4. To configure the VPC, use the following information:
    Field Value
    IP CIDR block 10.10.0.0/16
    VPC Name Talend VPC
    Public subnet 10.10.0.0/24
    Availability Zone eu-west-1a
    Subnet Name Public subnet eu-west-1a Talend ESB
  5. Click Create VPC.
    The VPC is successfully created.

    AWS should also have created along with the VPC the following items:

    • One public subnet named Public subnet eu-west-1a Talend ESB in Availability Zone eu-west-1a.
    • Two route tables associated with the VPC Talend VPC.
    • One Internet Gateway.
    • One default Network ACL associated with the public subnet Public subnet eu-west-1a Talend ESB.

    Once the VPC is created, we can then add a second subnet hosted in Avaibility Zone eu-west-1b.

Add a second public subnet

Procedure

  1. Open the VPC console. Then select the Subnets menu.
    Note:

    Actually, AWS recommends creating as many subnets as available Availability Zones within a region for better high-availability. We will limit to two Availability Zones for this demo even if Ireland Region contains three Availability Zones.

  2. Click Create Subnet. Use the following information:
    Field Value
    Name tag Public subnet eu-west-1b Talend ESB
    VPC Talend VPC in the drop-down list
    Availability Zone Select eu-west-1b.
    CIDR block 10.10.1.0/24

  3. Click Yes, Create.

    The subnet Public subnet eu-west-1b Talend ESB is created. We need to make it public by updating its route table.

Update the route table

Procedure

  1. On the Navigation pane, choose Subnets.
  2. Then select your first created subnet Public subnet eu-west-1a Talend ESB for eu-west-1a.
  3. Select the Route Table panel under the main panel.
  4. Click the Route Table rtb-7e89981a to open the route table.
    Note: Note that the name will certainly be different in your environment.
    This should open a new window with the selected route table.
  5. Select the route table then click the sub-panel Subnet Associations.
  6. Click Edit.
  7. Check the subnet Public subnet eu-west-1b Talend ESB to associate it with the route table. Then, click Save.

    The subnet Public subnet eu-west-1b Talend ESB is now public and uses the same route table than subnet Public subnet eu-west-1a Talend ESB.

Create a Load Balancer

AWS Elastic Load Balancing will be used to distribute traffic evenly between all Talend ESB instances across the Availability Zones. You will create a front-end HTTP loadbalancer to load-balance incoming requests between all Talend ESB instances.

Procedure

  1. Open EC2 console.
  2. On the Navigation pane, under Load Balancing, choose Load Balancers.
  3. Click Create Load Balancer.
  4. Select Classic Load-balancer then click Continue.
  5. Define the load-balancer:
    Field Value
    Load Balancer name ELB-Talend-ESB
    Create LB inside Choose Talend VPC.
  6. In Listener Configuration, add the following listener:
    Field Value
    Load Balancer Protocol HTTP
    Load Balancer Port 8040
    Instance Protocol HTTP
    Instance Port 8040
  7. In Select subnets, select both subnets Public subnet eu-west-1a Talend ESB and Public subnet eu-west-1b Talend ESB.

    The configuration should look like this: