Talend ESB auto-scaling on AWS platform overview
Online shopping companies always need to design a stable and scalable environment for their web services applications – specially need to be 100% available during sales event (Christmas sales, Black-Friday sales, etc). It’s becoming very challenging for them if they need to have heavy capacity preparation activities before each event, such as bringing extra server infrastructures, web service code deployment activities, manual configurations...
In this article, you will see how one of online shopping companies using a modern auto-scaling design solution with Talend and AWS platform to fit with their stable demand patterns or usage variabilities.
AWS Auto Scaling helps to maintain application availability and allows to scale your Amazon EC2 capacity up or down automatically according to conditions defined. You can use Auto Scaling to help ensure that you are running your desired number of Amazon EC2 instances. Auto Scaling can also automatically increase the number of Amazon EC2 instances during demand spikes to maintain performance and decrease capacity during lulls to reduce costs.
Talend ESB Runtime provides an Apache Karaf-based ESB container pre-configured to support Talend Mediation Route (Apache Camel routing) and Talend Web Services Jobs (Apache CXF-based services - both REST and SOAP-based).
In this architecture, you will build a Talend Data Services platform hosted on Amazon EC2 instances, and setup AWS Auto Scaling capacities to increase/decrease Talend ESB Runtime servers based on demand.
When you will have completed this the entire Talend Runtime Auto-Scaling demo, you will be able to build a 100% scalable environment for your web service applications by using Talend and AWS technologies.
You may implement the similar solution based on your need, and bring more AWS services to your designs such as: OpsWorks, CloudFormation, Elastic Beanstalk, Lambda function, etc. Also, the AWS Auto-Scaling feature is not limited to server Talend Runtime applications, other Talend server components could equally benefit from this (such as Talend Jobserver for ETL Job processing), so you will share more use cases with technologies mentioned here. Please find more Talend + AWS use cases on https://www.talend.com.
Talend ESB Runtime auto-scaling architecture on AWS platform
Talend ESB Runtime auto-scaling architecture on AWS platform
Main Talend Runtime Auto-Scaling process:
Procedure
-
Talend TAC has one or multiple pre-configured Runtime servers (from same Auto Scaling group) and web services applications (Talend Jobs or Routes) have been deployed from Nexus to each Runtime container. Existing Runtime(s) have AWS Elastic Load Balancer in front to distribute incoming traffic.
-
When one or more Runtime(s) EC2 instances have passed resource capacity limitation pre-defined in AWS CloudWatch Events (E.g one of Runtime EC2 instance’s CPU is running above 80% during the last 30 minutes - with increased web service requests during sales event).
-
AWS CloudWatch notifies Talend Auto Scaling Group (AWS Launch Configuration) to spin up one or more Runtime server EC2 instances and use the same Load Balancer in front, so that increased incoming traffic could be redirected to new Runtime instances.
-
Once new Runtime Instance(s) are initialised by Auto Scaling group, it will use Talend MetaServlet as Linux service scripts to:
-
Once incoming traffic is back to “normal” – E.g. All Runtime server EC2 instance’s CPU are running under 20% for last hour - AWS CloudWatch Events will notify Auto Scaling group to stop/determine Runtime server instances and some Talend MetaServlet scripts can be used here:
Talend ESB Runtime auto-scaling assumptions
Amazon Web Services (AWS):
- You should be familiar with the AWS platform so this article does not take a deep dive into details regarding Administration and Management of AWS services. You can refer to the Amazon Web Services (AWS) - Getting Started to read on all the AWS functionalities that Talend provides.
- You should also have full access to the main AWS services described in the Prerequisites section below.
Talend:
- You should be familiar with the Environment and Prerequisites of the Installation and Management of Talend Data Services Platform.
- You should have basic knowledge for Talend MetaServlet APIs. For further information, please refer to: Talend Administration Center MetaServlet API commands.
Talend ESB Runtime auto-scaling platform prerequisites
A valid AWS account with full access to following services:
Read the documentation at Valid AWS Access Keys to programmatically access AWS services to know how to create/manage/use AWS access keys.
Enabling TAC and Nexus on EC2 and configuration
Launching an EC2 instance
Procedure
Installing Talend Administrator Centre
Procedure
Preparing a Talend REST service Job
You would also need to prepare a Talend Web Service Job/Route (REST/SOAP) that can be deployed into Talend Runtime instances later on in this demo, you can download the Demo Job directly from here.
Procedure
Enabling Talend Runtime on EC2 configuration
Launching an EC2 instance
Procedure
Installing Talend Runtime
Procedure
Talend Metaservlet API to be used at Talend Runtime host setup
In this article, you will need to use Talend metaservlet API call as shown below, so that AWS Auto-scaling could allow Talend Runtime host to interact automatically with Talend TAC host without any manual configuration:
- "addServer": Declare Talend Runtime server to TAC.
- "saveEsbTask": Create a new ESB task (service) in TAC/ESB conductor.
- "requestDeployEsbTask": Request Talend Runtime to deploy the ESB task previously created in "saveEsbTask".
- "requestUndeployEsbTask": Request Talend Runtime to undeploy ESB task specified in script.
- "deleteEsbTask": Delete undeployed ESB task
- - "removeServer": Remove Talend Runtime server declaration from TAC.
For complete Talend metaservlet API details, please refer to:Talend Administration Center MetaServlet API commands.
Building a Linux shell script to convert clear text to Base64 encoding string
Talend MetaServlet string need to be encoded to Base64 when requesting TAC MetaServlet REST API.
To post a JSON script, take metaservlet - “addServer” as an example, as shown below. This JSON sample adds a new Jobserver/Runtime server into TAC server page.
{"actionName": "addServer","adminConsolePort": 8040,"authPass": "admin","authUser":"admin@company.com",
"commandPort": 8000,"description": "RemoteRT", "filePort": 8001,"host": "34.251.88.225",
"instance": "trun", "label": "Remote RT server auto-scaling","mgmtRegPort": 1099,
"mgmtServerPort": 44444,"monitoringPort": 8888,"runtimePassword": "tadmin",
"runtimeUsername": "tadmin","shutdownBehavior": "Stop",
"timeoutUnknownState": "120","useSSL": false}
Procedure
Error messages when testing the base64url.sh
xxd: command not found
Depending on your Linux version, you might see the below error messages during testing the base64url.sh:
If you encounter the following error message:
Install the "vim-common" package as shown below:
See more details about "vim-common" on https://stackoverflow.com/questions/36179338/official-fedora-package-for-xxd-command.
bc: command not found
If you encounter the following error message:
Install “bc” as shown below:
See more details for “bc” on: http://thelinuxfaq.com/159-bc-command-not-found-in-centos-rhel-fedora-ubuntu
Building a Linux service script to call TAC Metaservlet APIs
Buidling a start() script
In this section, you will need to write the scripts below to build a Linux service script TALEND-INIT -including start() and stop() scripts.
Main steps in start() section:
Procedure
Building a stop() script
Procedure
Executing a Linux service script
Retrieve the complete service script TALEND-INIT file from the Downloads tab in the left panel of this page.
Once downloaded, please copy this file into your Runtime host server under /etc/init.d repository.
Procedure
Starting the TALEND-INIT script
-
A new Talend Runtime Server has been added to TAC/servers declaration page
(completed by the “addServer” metaservlet):
-
A new ESB task has been created in the TAC/ESB Conductor tab and also has
been deployed to new Talend Runtime server (completed by the “saveEsbTask”
and “requestDeployEsbTask” metaservlets):