Amazon Web Services (AWS) - How to configure ActiveMQ in Master/Slave mode with KahaDB on AWS Elastic File System
Overview
AWS Elastic File System is a simple and scalable file storage which supports the Network File System version 4.1 protocol. Thus, it provides a highly-available and elastic file storage system for use with Amazon EC2 instances and it supports file locking while providing strong data consistency.
Apache ActiveMQ is a very popular open source message broker, fully compliant with JMS 1.1, which is shipped with your Talend ESB product.
To use ActiveMQ in a highly-available architecture, you will need to setup a Master/Slave configuration. The followings are the three widely known approaches to the Master/Slave configurations for ActiveMQ:
- Shared File System Master Slave
- JDBC Master Slave
- Replicated LevelDB Store
For customers deploying on AWS Cloud, leveraging the AWS EFS as a Shared File System to configure ActiveMQ in a Master/Slave configuration can be the most appropriate and straight-forward configuration.
Architecture
The main components of this architecture are:
- Two ActiveMQ brokers (1 Master + 1 Slave) running on separate EC2 instances each.
- A shared file system provided by AWS EFS that will host the KahaDB directory.
Assumptions
- Amazon Web Services (AWS):
- You should be familiar with the AWS platform.
For more information about all the AWS functionalities that Talend provides, see Amazon Web Services (AWS) - Getting Started.
- You should have full administrator access to the main AWS services described in the Prerequisites section below.
- You should be familiar with the AWS platform.
- Apache
ActiveMQ:
- You should be familiar with the installation and configuration of Apache ActiveMQ.
Environment
This tutorial is based on AWS Cloud Platform and Apache ActiveMQ 5.14.1.
Prerequisites
- Amazon Elastic Compute Cloud - https://aws.amazon.com/ec2.
- Amazon Elastic File System - https://aws.amazon.com/efs.
Configuring ActiveMQ in Master/Slave mode for AWS deployment
Choose an AWS Region
At the time of writing this tutorial, Elastic File System (EFS) is currently not available in all the Amazon Web Services regions. Therefore, it is important to select a region that provides this service.
For this tutorial, you will use the Ireland region.
Create a VPC
Procedure
Create a Security Group For ActiveMQ
In this section, you will create a security group for the EC2 instances hosting your ActiveMQ brokers.
Procedure
Create a Security Group For Elastic File System
Procedure
Create the Elastic File System
In this section, you will create the EFS file system to be used by ActiveMQ.
Procedure
Launch the first EC2 Instance
Procedure
Configure the EC2 Instance with EFS
As soon the first ActiveMQ EC2 instance has finished launching, you can proceed to the EC2 instance setup to access the EFS File System.
Setting up your EC2 instance
Procedure
Mounting your file system
Procedure
Option 1: If you have copied the Amazon EC2 Mount instructions in your favorite editor
Procedure
Option 2: If you did not copy/paste the Amazon EC2 Mount instructions
Procedure
Install and configure ActiveMQ
Talend ESB 6.3.1 ships with ActiveMQ 5.14. Since this article is focused solely on ActiveMQ, you can also install ActiveMQ 5.14 using packages from Apache ActiveMQ official web site at http://activemq.apache.org/activemq-5140-release.html.
Please read first this AWS guide at http://docs.aws.amazon.com/en_en/AWSEC2/latest/UserGuide/putty.html to know how to use putty to ssh into your instance.
Install and configure ActiveMQ
Procedure
Configure ActiveMQ to run as daemon
Before you begin
Procedure
Configure a second EC2 Instance For ActiveMQ
Up to this point, you have successfully setup EFS, installed and configured an ActiveMQ broker on an EC2 instance, and also configured the ActiveMQ broker to use the shared EFS file system.
You are now ready to launch a second ActiveMQ broker which will share the same configuration and thus the same shared EFS file system as the first ActiveMQ instance. This will help us validate that the second ActiveMQ broker will automatically become a slave of the first ActiveMQ broker, since the latter is already running and locking the shared file system (thanks to EFS/NFS4 locking feature).
- Option 1: repeat the steps of the previous sections to launch an EC2 instance and configure a similar ActiveMQ broker with exactly the same configuration (just use a different name for the EC2 instance in order not to be confused with the first instance).
- Option 2: create an AMI from the running ActiveMQ broker EC2 instance, and then use the AMI to launch a similar instance with ActiveMQ embedded.
You are going to perform Option 2 for this tutorial.