Skip to content

Overview

Development

Overview

IDEs

API Explorer

Deploying TORO Integrate with CloudFormation

Introduction

With a Server license, you can provision TORO Integrate to sit on top of your organization's own on-premise servers or run on top of a public cloud provider such as AWS, Azure, or the Google Cloud Platform. In this document, we will discuss how to deploy TORO Integrate on AWS using CloudFormation.

Flexible deployment models

There are multiple ways to deploy TORO Integrate, and your deployment options will depend on the type of license that you have.

A CloudFormation template defines a stack of AWS servers and services, and provides a mechanism to reference recipes that can automate network and application configuration. TORO has made a flexible and customizable AWS CloudFormation template available for public use which allows enterprise customers to configure a three-tier, scalable network architecture for TORO Integrate without much effort, and that which benefits from AWS’s extensive range of services. This template was designed, built, and tested by the same experts managing the TORO Cloud infrastructure on AWS.

CloudFormation template?

An AWS CloudFormation template defines a stack of AWS servers and services and provides a mechanism to reference recipes that can automate network and application configuration.

Why AWS? Why CloudFormation?

Amazon Web Services (AWS) is the world’s leading cloud infrastructure solutions provider, providing a flexible and economical alternative to organisations deploying bare metal servers in a private data center whilst offering a breadth of services and quality of service demanded by some of the world’s largest brands. AWS CloudFormation dramatically simplifies the deployment of applications, servers, and services on AWS.

The CloudFormation template provided by TORO is designed to work with organizations running on Server, Enterprise Edition. It will make use of as many AWS-managed resources and services as possible to minimize the amount of server and infrastructure management required. This deployment method is useful for organizations looking to migrate to the cloud and gain all the benefits of deploying an infrastructure in AWS – customizable, flexible, manageable, and easy to deploy.

Services Used

The AWS CloudFormation template provided by TORO uses the following AWS services:

AWS Service Function Benefit Implementation
AWS CloudFormation Deployment service For simplifying the provisioning and managing of related AWS resources, as well as for updating them in an orderly manner. Deploys resources to AWS, all of which have been inspected and passed TORO’s standards and security policies.
AWS Elastic Container Service (ECS) Container management service Docker container orchestration and management on EC2 instances. Automatic scaling of production containers managed by ECS. Automatic health checks also verifies redundancy.
AWS Elastic Cloud Computing (EC2) Virtual servers in the cloud Elastic and on demand compute capacity. By default, the template will provision micro server instances that are available under AWS’s free billing tier. The server specifications can be changed during or after deployment to any EC2 server size to suit the sizing of the target application.
AWS Virtual Private Cloud (VPC) Virtual network Complete control over your own virtual network. Resources are managed and isolated according to their corresponding role or group which makes it easier to determine their permissions and access. This setup has been one of the best practices in the industry.
AWS Relational Database Service (RDS) SQL database A managed and scalable SQL database. By default, the template will provision a MySQL instance on RDS. The database is automatically replicated to another instance in a separate Availability Zone. RDS will automatically backup the database and switch to the fail-over replica in case of an outage in the primary database.
AWS CloudWatch Monitoring and logging Collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in AWS resources. Every resource has a corresponding monitoring agent and an alerting scheme.
AWS OpsWorks Auto scaling and EC2 manager Auto-scale instances on demand. Easily manage the resources in your infrastructure. All instances in the VPC are managed by OpsWorks. With this, system administrators don’t need to manually provision instances and worry about its OS, upgrades, and the applications installed in all VMs. With OpsWorks, you can manage one or 1000 servers without affecting the workflow.
AWS Elastic File System (EFS) Central file storage Elastic file storage that can be accessed from multiple EC2 instances at the same time. AWS EFS enables the Docker containers to have their data retained even when a container is transferred to another EC2 instance. We don’t need to worry about performance as it is in Max I/O performance mode.
AWS Simple Notification Service (SNS) Notification service Managed messaging service. With AWS SNS, separating notifications into different priority tiers can easily be managed. We can configure whether an EC2 instance failure for a system check will trigger a notification to the proper team or just send the notification to an Enterprise Service Bus (ESB), like TORO Integrate, to digest the content and execute the proper remedy.
AWS Route 53 DNS Managed Domain Name System web service which connects user requests to the infrastructure running on AWS. Route 53 organizes all your domain names. It also provides a private domain functionality for your internal resources and tools.
AWS Certificate Manager SSL Certificate Manager Provision, manage, and deploy SSL certificates within AWS resources. Through the AWS Certificate Manager, it can be determined if an SSL certificate needs to be renewed. It can also generate a custom SSL certificate for an Elastic Load Balancer (ELB) or CloudFront distribution.

Design

The AWS CloudFormation template for TORO Integrate addresses the key challenges of deploying a scalable enterprise application in the cloud. Users will benefit from:

  • AWS’s managed infrastructure by minimizing downtime with automatic failover and recovery
  • Monitoring services that minimize time and costs spent on troubleshooting issues
  • Leveraging the expertise of TORO’s in-house team of system administrators to ensure that an organization’s deployment of TORO Integrate is deployed on tested and proven technologies whilst applying the industry’s best practices.

The diagram below illustrates the topology configured in the AWS CloudFormation template, which implements a highly-available three-tier network architecture:

CloudFormation template topology

FAQ

  • Is there an additional cost for the template?

    No, the template is free.

  • Can we use the template with any edition?

    By default, no. The template deploys an instance of SolrCloud. Only the Enterprise Edition supports SolrCloud.

  • What is the estimated cost of this AWS deployment?

    It depends on which type of instances you’ve chosen for your EC2 instances and RDS instances. Data transfers and storage costs may also vary depending on the size of the data. Use the AWS simple monthly calculator to compute for an estimate.

  • If I remove or add services or EC2 instances, how will it affect the infrastructure?

    Once the CloudFormation template has been fully deployed, you can then use OpsWorks to add EC2 instances. As for removing a service, you may edit the template and it will warn you if a resource has a dependency and cannot be deleted.

  • Do I need to configure anything before or after deploying this infrastructure?

    No. Everything is production-ready.

Deployment Procedures

Using the Template

To use the AWS CloudFormation template to deploy TORO Integrate, you must:

Permissions and resources

The user performing the steps below will need to have permission to provision instances, spanning multiple types of AWS resources.

AWS free tier

By default, all instances are micro-instances which means you can deploy it for free provided that your account is eligible for the free tier on AWS. The default instance size can be changed during deployment.

  1. Download the template.
  2. Proceed to CloudFormation by choosing CloudFormation from the list of Amazon Web Services or by clicking the drop-down menu located at the upper left corner for services and choosing CloudFormation.

    Navigating to AWS CloudFormation

  3. Click the Create a New Stack button.

    Creating a new stack

    Choose the correct AWS Region

    For this deployment, we will select the North Virginia (us-east-1) region. Ensure you select the right region. You can check your region by looking at the upper right menu.

    List of AWS Regions

  4. Choose to upload the template to Amazon S3, then browse and select the template downloaded earlier.

    Choosing the template

  5. Input your custom data.

    Entering custom data for template

    The parameters you can enter for this step are described below:

    Parameter Purpose
    Stack name Choose a name for your infrastructure.
    AllowCidrBlock Enter a CIDR block to allow access to your network e.g. 0.0.0.0/0 (optional).
    DBAllocatedStorage Determine the size of the database (in GB).
    DBInstanceClass Specify the database instance type.
    DbMasterPassword Configure a database master password.
    DbMasterUser Configure a database master username. It is not recommended to use root.
    Email Specify the email address to notify when a CloudWatch metric has triggered an alarm.
    MultiAZ Choose whether you would like a Multi-AZ master database which automatically creates a primary DB instance and synchronously replicates the data to a standby instance in a different Availability Zone (AZ).
    OrganizationName Specify the name of your organisation/company.
    ParentDomain Enter the DNS name of an existing Amazon Route 53 hosted zone.
    SslCertificateName Identify the name of an SSL certificate uploaded in AWS Certificate Manager that you want to use for TORO Integrate.
    T1BCidrBlock Enter CIDR block for Tier 1 - Zone B subnet.
    T1CCidrBlock Enter CIDR block for Tier 1 - Zone C subnet.
    T1WebEc2Type Specify WebServer EC2 instance type.
    T2ActivemqEc2Type Specify ActiveMQ EC2 instance type.
    T2BCidrBlock Enter CIDR block for Tier 2 - Zone B subnet.
    T2CCidrBlock Enter CIDR block for Tier 2 - Zone C subnet.
    T2EcsEc2Type Specify ECS cluster EC2 types. They will run the Docker containers of TORO Integrate.
    T2SolrEc2Type Specify SolrCloud EC2 instance type.
    T2ZookeeperEc2Type Specify ZooKeeper EC2 instance type.
    T3BCidrBlock Enter CIDR block for Tier 3 - Zone B subnet.
    T3CCidrBlock Enter CIDR block for Tier 3 - Zone C subnet.
    VPCSubnetCidrBlock Enter VPC CIDR block. (i.e. 10.0.0.0/16).
  6. Add tags and permissions (optional).

    Adding tags

  7. Review your details and if satisfied, create your stack by clicking the Create button at the bottom of the page.

    Review and create Cloudformation stack information

And that's it! You're done. You’ll see your stack in the dashboard that it is being created. You can also monitor the deployment process of the CloudFormation template in the Events tab.

Creating a DNS Record

Instead of using the IP address generated by AWS to access your TORO Integrate instance, you can use your own domain name. In this section, we'll be showing you how to do this via creating an A record in AWS Route 53. Route 53 is a service available on AWS that is used to manage the Domain Name Service (DNS) of web services to direct users to web applications by translating domain names into numeric IP addresses.

Creating a Hosted Zone

  1. Proceed to Route 53 by clicking Route 53 under the category Networking & Content Delivery. You'll be able to view the list of AWS services in the AWS Management Console home or via the Services drop-down at the static, navigation bar at the top.

    Navigating to AWS Route 53

  2. Select Hosted Zones from the left menu bar.

  3. Choose Create Hosted Zone at the top portion of the screen.
  4. A form will appear on the left side of the screen. Fill in the details.

    Field hints

    If you don't know what a field is for, you can easily hover over the field to see a short description about it.

    Hovering over fields to show hints

  5. Once done, click the Create button.

    Creating a hosted zone

Hosted zone type

The accessibility of your domain will depend on your network and the type of hosted zone you have selected upon creation (either Public Hosted Zone or Private Hosted Zone for Amazon VPC).

Creating a Record Set

  1. Select your domain and click on the Go to Record Sets button.

    Viewing record sets of a hosted zone

  2. Next, a button labeled Create Record Set should be visible; click it.

    Creating a record set

  3. Fields to fill in the information for your record set will pop up on the right side of the screen. Modify the Value field so that it contains the NGINX instance’s IP address.

    Record set form

  4. Once done, click Save Record Set.

And that's it. Open your web browser and enter the domain name on the address bar to verify your changes.

Migrating from Bare Metal Servers to AWS

If you are looking to migrate data from your local Linux servers to AWS EC2 instances, then read on because this guide is for you. The steps required for this type of migration are split into sections and are discussed in order.

Prerequisites

  • Successfully deploy the infrastructure using the CloudFormation template. The Linux instances deployed on AWS will be where we will be migrating data to.
  • Ensure that you have the credentials needed to access the AWS EC2 instance(s).
  • You need to make sure that you have ROOT access for both the source and destination server instances.
  • Your SSH ports should be set to default (port 22).
  • You must have screen installed (yum install -y screen).

Password-less SSH

You need to configure a temporary password-less configuration to achieve a connection from your source server to the destination server. You may achieve this using the following instructions:

  1. Login to your local server and generate a public/private key pair, if you don’t have one already.

    1
    ssh-keygen -t rsa
    
  2. Use SSH to create an .ssh directory for the source server. This directory may already exist, so if you are sure it does, you may skip this step.

    1
    ssh user@EC2IP mkdir -p .ssh
    
  3. Finally, you may append the public key of your local server to the EC2 server.

    1
    cat .ssh/id_rsa.pub | ssh user@EC2IP 'cat >> .ssh/authorized_keys'
    
  4. Verify the password-less SSH by trying to connect to it.

    1
    ssh user@EC2IP
    

Script-Assisted Migration

To help us with migrating resources, we will be running a script.

It is recommended to install screen on the source server so that we can maintain a persistent session during this potentially long process. You may use your package manager to perform the installation.

  1. We start by launching a screen session with the command:

    1
    screen -S OCSMigration
    
  2. In case you get disconnected, you can run this command to resume the session after logging back in:

    1
    screen -r
    
  3. Now it is time to retrieve the script. You may retrieve it via:

    1
    wget https://raw.githubusercontent.com/TORO-IO/toro-integrate-enterprise-cloudformation/master/migration-scripts/rsyncmigration.sh
    
  4. Make sure it is executable by running the following command:

    1
    chmod +x rsyncmigration.sh
    
  5. Once you are able to retrieve the script, you may now execute it by running:

    1
    bash ./rsyncmigration.sh
    

    This script will:

    1. Prompt you for the IP address of the target server.
    2. Ask you which directories you would like to migrate. It will ask you the path of the following directories:

      • /data
      • /jdbc-pool
      • /logs
      • /packages
      • /system-tmp
      • /code
      • /tmp
    3. Afterwards, the script will attempt to a connection with the destination server as ROOT.

    4. After that it will ask you to specify your organization name which will be relative to the path which was created for you during the CloudFormation setup. The syntax goes as follows: /datastore/clients/${ORGNAME}/apps/integrate/${ORGNAME}1/assets.
    5. It will also ask you to verify the path and to make sure it exists in the destination server as there is no way to check if the directory exists or not.
    6. After authentication, the script will attempt to setup a key-based authentication between the source and destination server. Make sure that your SSH port is 22, at least for the migration.
    7. The script will attempt to verify the installation of rsync on both servers.
    8. It will run rsync twice. First to attempt to migrate the data, and second to perform a final sweep in an attempt to follow up on any missed files or files that may have had changes during the first sync.

    After the process has ended, verify if the files have been successfully migrated. Once verified, you may now use this data for your TORO Integrate instance.

Manually Removing Instances

It's also possible to tweak the default CloudFormation template deployed over AWS to suit your organization's design needs.

AWS OpsWorks Instances

You will have a list of instances organized in stacks and layers in OpsWorks after the deployment of your CloudFormation template. To remove any of these AWS instances:

  1. Proceed to OpsWorks by clicking the dropdown menu at the upper left corner labeled Services and then choose OpsWorks.

    Accessing AWS OpsWorks

  2. Next, click on the stack you’d like to modify.

    List of OpsWorks stacks

  3. Click Instances from the appearing left navigation bar.

    Viewing stack instances

  4. Stop the instance that you want to delete if it is running by clicking on the stop button under the stack's Actions entry. Confirm your action once a confirmation alert pops-up.

    Stopping OpsWorks instance

  5. Finally, click on the delete link under the Actions entry once available. Confirm your action.

    Deleting an OpsWorks instance

Amazon RDS Instances

Amazon Relational Database Service (Amazon RDS) is a service that is used to set up, operate, and scale a relational database in the cloud.

The basic building block of Amazon RDS are RDS instances. An RDS instance is an isolated database environment in the cloud. An RDS instance can contain multiple, user-created databases, and you can access them by using the same tools and applications that you use with a stand-alone database instance.

In this portion, we will discuss how to delete RDS instances.

  1. Proceed to the Relational Database Service page by clicking the drop down menu at the upper left corner labeled Services, and then choose RDS.

    Accessing the AWS RDS service

  2. Select Databases at the menu bar on the left to see all existing RDS instances.

    Select RDS Instances

  3. Select the RDS instance you want to remove from the list that appears; in the instance's page, proceed to click the Instance actions dropdown, and then click on Delete.

    Deleting an RDS instance

  4. A modal will appear, asking you whether to create a final snapshot of the RDS instance, as well as if automated back-ups should be kept. Make your choice, and confirm your action.

Confirming deletion of RDS instance

The deletion process may take some time to finish. A successfully deleted RDS instance will no longer appear in the Databases tab.