Amazon Web Services (AWS) - How to configure ActiveMQ in Master/Slave mode with KahaDB on AWS Elastic File System - 7.0

EnrichVersion
7.0
EnrichProdName
Talend Data Fabric
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
EnrichPlatform
Talend ESB
task
Administration and Monitoring
Data Governance
Data Quality and Preparation
Deployment
Design and Development
Installation and Upgrade

Amazon Web Services (AWS) - How to configure ActiveMQ in Master/Slave mode with KahaDB on AWS Elastic File System

Overview

AWS Elastic File System is a simple and scalable file storage which supports the Network File System version 4.1 protocol. Thus, it provides a highly-available and elastic file storage system for use with Amazon EC2 instances and it supports file locking while providing strong data consistency.

Apache ActiveMQ is a very popular open source message broker, fully compliant with JMS 1.1, which is shipped with your Talend ESB product.

To use ActiveMQ in a highly-available architecture, you will need to setup a Master/Slave configuration. The followings are the three widely known approaches to the Master/Slave configurations for ActiveMQ:

For customers deploying on AWS Cloud, leveraging the AWS EFS as a Shared File System to configure ActiveMQ in a Master/Slave configuration can be the most appropriate and straight-forward configuration.

Architecture

The diagram below describes the architecture which you will build throughout this tutorial.

The main components of this architecture are:

  • Two ActiveMQ brokers (1 Master + 1 Slave) running on separate EC2 instances each.
  • A shared file system provided by AWS EFS that will host the KahaDB directory.

Assumptions

  • Amazon Web Services (AWS):
    • You should be familiar with the AWS platform.

      For more information about all the AWS functionalities that Talend provides, see Amazon Web Services (AWS) - Getting Started.

    • You should have full administrator access to the main AWS services described in the Prerequisites section below.

  • Apache ActiveMQ:
    • You should be familiar with the installation and configuration of Apache ActiveMQ.

Environment

This tutorial is based on AWS Cloud Platform and Apache ActiveMQ 5.14.1.

Prerequisites

A valid AWS account with full administrative access to the following services: 

Configuring ActiveMQ in Master/Slave mode for AWS deployment

This simple tutorial will walk you through the steps you need to follow to configure ActiveMQ in Master/Slave for AWS deployment.

Choose an AWS Region

At the time of writing this tutorial, Elastic File System (EFS) is currently not available in all the Amazon Web Services regions. Therefore, it is important to select a region that provides this service.

For this tutorial, you will use the Ireland region.

You should check within your AWS Console to confirm the availability of this service in your region, if selecting a different one.
Note: Other considerations when choosing an AWS region: latency, compliance requirements.

Create a VPC

Procedure

  1. Go to VPC console in Ireland region.
  2. Click Start VPC Wizard.
  3. Select VPC with a Single Public subnet.
    Note: For the sake of simplicity, you will limit to only 1 subnet in this tutorial.

    In Production environments, we recommend using multiple subnets deployed in at least 2 Availability Zones as per the AWS best practices for high-availability requirements. Deploying in multiple Availability Zones caters for the scenario when 1 Availability Zone is not available completely due to network segmentation or data center issues.

  4. Use the following values to configure the VPC:
    Field Value
    IPv4 CIDR block 10.0.0.0/16

    (Use a different CIDR block if this one is unavailable in your environment)
    VPC name Talend VPC
    Public subnet's IPv4 CIDR 10.0.0.0/24
    Availability Zone Select eu-west-1a.
    Subnet Name Public subnet – eu-west-1a – ActiveMQ

  5. Click Create VPC to proceed with creation.

Create a Security Group For ActiveMQ

In this section, you will create a security group for the EC2 instances hosting your ActiveMQ brokers.

Procedure

  1. In the EC2 console, click Security Groups.
  2. Click Create Security Group.
  3. Use the settings below:
    Field Value
    Security group name ActiveMQ Security Group
    Description Security Group of ActiveMQ brokers
    VPC Select Talend VPC.
  4. In Security group rules, add an SSH rule:
    Field Value
    Type ssh
    Protocol TCP
    Port Range 22
    Source Custom 0.0.0.0/0
  5. In Security group rules, add a TCP Rule:
    Field Value
    Type Custom TCP Rule
    Protocol TCP
    Port Range 8161
    Source Custom 0.0.0.0/0
  6. In Security group rules, add a TCP Rule:
    Field Value
    Type Custom TCP Rule
    Protocol TCP
    Port Range 61616
    Source Custom 0.0.0.0/0

    Review your settings:

  7. Click Create.
    Note: Note the group-id of the security group e.g. sg-bc2021da in this tutorial. You will need this information when creating the EFS security group.

Create a Security Group For Elastic File System

Procedure

  1. Go into VPC console.
  2. Click Security Groups > Create security group.
  3. Use the following values:
    Field Value
    Name tag EFS for ActiveMQ Security Group
    Group name EFS for ActiveMQ Security Group
    Description EFS for ActiveMQ Security Group
    VPC Select Talend VPC.
  4. Add a new Inbound rule as below:
    Field Value
    Type NFS
    Protocol TCP
    Source sg-bc2021da

    Put the id of the ActiveMQ Security group created before.

  5. Click Create.

Create the Elastic File System

In this section, you will create the EFS file system to be used by ActiveMQ.

Procedure

  1. Connect to EFS Console then choose Ireland region.
  2. Click Create file system.
  3. In Configure file system access:
    • VPC: Select Talend VPC.
    • Select the only subnet for Talend VPC.
      Note: In this example, you only have 1 subnet called Public subnet – eu-west-1a - ActiveMQ.
    • Remove all default security group added by default then Select the security group EFS for ActiveMQ Security Group.

    • Click Next Step.
  4. In Configure optional settings:
    • Add any relevant tags you want. Add at least a name tag with value Talend ActiveMQ EFS.
    • Performance mode: leave General Purpose (default).

    Click Next step.

  5. In Review and create, review the configuration.
  6. Click Create File System.

    Congratulations! You just have created your first Elastic File System on AWS in Ireland region. It was pretty easy and straightforward. This EFS File System will be used to share data between the ActiveMQ brokers.

    Our AWS console now displays the new file system with a link to a page describing the mount instructions for EC2.
  7. Click on the link Amazon EC2 mount instructions to display the following information:
    Tip: We suggest you to copy/paste the instructions in your favorite text editor as you will need it when setting up the EC2 instance
    These instructions and the command inside will help to configure your EC2 instances with ActiveMQ to access this shared file system.
    Note: But before you do this configuration, you need to setup your EC2 instances and install ActiveMQ.

Launch the first EC2 Instance

In this section, you will cover the instructions to launch the first EC2 instance which will host the first ActiveMQ broker.

Procedure

  1. Open the EC2 console and make sure that Ireland region is selected.
  2. Click Launch Instance.
  3. In Choose AMI, select Amazon Linux AMI 2016.09.1 64 bit which is a good fit for this demo.
  4. In Choose an Instance type, t2.medium is a good choice for this tutorial. Then, click Next: Configure Instance details.
  5. In Configure Instance Details, use the following values then click Next: Add Storage:
    Field Value
    Network Select Talend VPC.
    Subnet Choose Public subnet – eu-west-1a - ActiveMQ.
    Auto-assign Public IP Select Enable.
    For all other fields, leave everything as default.
  6. In Add Storage, use the following value then click Next: Add Tags:
    • Size = 8 GiB
  7. In Add Tags, use the folowing value then click Next: Configure Security Group:
    • Name: ActiveMQ Broker
  8. In Configure Security Group:
    • Choose the option Select an existing security group.
    • Select the security group ActiveMQ Security Group.

      Then click Review and Launch.

  9. Review your settings and click Launch.
  10. Choose your existing key pair or create a new key pair if you do not have any.
  11. Once you have set up your key pair, click Launch Instances.

    For more information on AWS Key pair, please refer to AWS documentation at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html.

Configure the EC2 Instance with EFS

As soon the first ActiveMQ EC2 instance has finished launching, you can proceed to the EC2 instance setup to access the EFS File System.

Setting up your EC2 instance

Procedure

  1. Use putty to ssh into your EC2 instance. Please read AWS guide at http://docs.aws.amazon.com/en_en/AWSEC2/latest/UserGuide/putty.html to learn how to use putty to ssh into your instance.
  2. As recommended by AWS, install all updates for the Linux System with the command: 
    sudo yum update
  3. Then, install the nfs client on your EC2 instance. 
    sudo yum install -y nfs-utils

Mounting your file system

Procedure

  1. Create a new directory efs under /opt: 
    sudo mkdir /opt/efs
  2. Change the owner to ec2-user: 
    sudo chown ec2-user:ec2-user /opt/efs
  3. Enter the /opt directory: 
    cd /opt
  4. Mount your file system using the DNS name of your EFS File System. To do that, you have two options:
    • Option 1: If you have copied the Amazon EC2 Mount instructions in your favorite editor.
    • Option 2: If you did not copy/paste the Amazon EC2 Mount instructions.
    Note: Note down this mounting command as it will be used again when launching a second ActiveMQ instance.
    Note: If you are unable to connect, please see the troubleshooting documentation at http://docs.aws.amazon.com/en_en/efs/latest/ug/troubleshooting.html.

Option 1: If you have copied the Amazon EC2 Mount instructions in your favorite editor

Procedure

  1. Check the instructions and locate the mount command which looks like the command below: 
    sudo mount -t nfs4 -o 
nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2<put your DNS name here>:/ efs
  2. Copy the mount command then paste it in your ssh terminal.
  3. Press Enter to execute.

Option 2: If you did not copy/paste the Amazon EC2 Mount instructions

Procedure

  1. First, locate the DNS name of the File System to use:
    • Go back in EFS console.
    • Click on Talend ActiveMQ EFS file system to show its details: the DNS Name to use is displayed under File System Access as per below:
  2. Copy the DNS Name and paste it in the mount command below: 
    sudo mount -t nfs4 -o 
nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2<put your DNS name here>:/ efs
  3. Copy the mount command then paste it in your ssh terminal.
  4. Press Enter to execute.

Install and configure ActiveMQ

Talend ESB 6.3.1 ships with ActiveMQ 5.14. Since this article is focused solely on ActiveMQ, you can also install ActiveMQ 5.14 using packages from Apache ActiveMQ official web site at http://activemq.apache.org/activemq-5140-release.html.

Now that you have configured the first EC2 instance, you can proceed with the ActiveMQ installation.
Note:

Please read first this AWS guide at http://docs.aws.amazon.com/en_en/AWSEC2/latest/UserGuide/putty.html to know how to use putty to ssh into your instance.

Install and configure ActiveMQ

Procedure

  1. Use ssh into EC2 instance with putty, then create the folders /opt/talend and /opt/talend/activemq. 
    sudo mkdir -p /opt/talend/activemq
sudo chown ec2-user:ec2-user /opt/talend/activemq
  2. Create the folder /opt/efs/kahadb, then change the owner to ec2-user. 
    sudo mkdir /opt/efs/kahadb
sudo chown ec2-user:ec2-user /opt/efs/kahadb
  3. Go into the directory /opt/talend/activemq, then execute the following command to download ActiveMQ 5.14 distribution from internet. 
    cd /opt/talend/activemq
wget https://archive.apache.org/dist/activemq/5.14.0/apache-activemq-5.14.0-bin.tar.gz
  4. Extract the downloaded archive. 
    tar xvfz apache-activemq-5.14.0-bin.tar.gz
  5. A directory apache-activemq-5.14.0 is created by the extraction. Enter that directory and verify that it contains several files and directories as per below:
  6. Go into the conf directory then edit activemq.xml file to setup the KahaDB directory:
    • Locate the KahaDB section under PersistenceAdapter element and update the KahaDB configuration as per below: 
      • old: <persistenceAdapter> <kahaDB directory="${activemq.data}/kahadb"/> </persistenceAdapter>
      • new: <persistenceAdapter><kahaDB directory="/opt/efs/kahadb"/></persistenceAdapter>
    • Save and close the file.

Configure ActiveMQ to run as daemon

Before you begin

For more instructions about the shell script, refer to the documentation available at http://activemq.apache.org/unix-shell-script.html.

Procedure

  1. Go into the directory /opt/talend/activemq. 
    cd /opt/talend/activemq
  2. Create a link current to the Apache installation. 
    ln -snf apache-activemq-5.14.0 current
  3. Create a global default configuration and edit the configuration. 
    sudo cp apache-activemq-5.14.0/bin/env /etc/default/activemq
sudo sed -i '~s/^ACTIVEMQ_USER=""/ACTIVEMQ_USER="ec2-user"/' /etc/default/activemq
sudo chmod 644 /etc/default/activemq
  4. Install the init script. 
    sudo ln -snf  /opt/talend/activemq/current/bin/activemq /etc/init.d/activemq
  5. Activate the init script at the system startup. 
    sudo chkconfig --add activemq
sudo chkconfig activemq on
sudo chkconfig –list
  6. Check that your activemq is fully setup and configured to start at boot instance:
    • Reboot your EC2 instance
      • You can do that from AWS console or directly within your ssh session with the command sudo reboot.
      • Note that this will terminate your ssh session.
    • Wait a few minutes for your instance to reboot.
    • Use putty to ssh again into your instance.

Configure a second EC2 Instance For ActiveMQ

Up to this point, you have successfully setup EFS, installed and configured an ActiveMQ broker on an EC2 instance, and also configured the ActiveMQ broker to use the shared EFS file system.

You are now ready to launch a second ActiveMQ broker which will share the same configuration and thus the same shared EFS file system as the first ActiveMQ instance. This will help us validate that the second ActiveMQ broker will automatically become a slave of the first ActiveMQ broker, since the latter is already running and locking the shared file system (thanks to EFS/NFS4 locking feature).

There are 2 possible options to launch a second broker:
  • Option 1: repeat the steps of the previous sections to launch an EC2 instance and configure a similar ActiveMQ broker with exactly the same configuration (just use a different name for the EC2 instance in order not to be confused with the first instance).
  • Option 2: create an AMI from the running ActiveMQ broker EC2 instance, and then use the AMI to launch a similar instance with ActiveMQ embedded.

You are going to perform Option 2 for this tutorial.

Create an AMI from the running ActiveMQ broker EC2 instance

Procedure

  1. Connect to EC2 console.
  2. Click Instances then locate the ActiveMQ broker EC2 instance.
  3. Click Actions > Image > Create Image.
  4. In Create Image, use the following parameters:
    Field Value
    Image Name AMI ActiveMQ Broker
    Image Description AMI ActiveMQ Broker

    Leave all other parameters with default values.

  5. Click Create Image.
    You can follow the status of the image creation by clicking on the link View Pending Image....
    You can also click on the menu Images > AMIs then search for the image AMI ActiveMQ Broker.