Skip to content

Overview

Concepts and Principles

Development

Overview

IDEs

API Explorer

Releases

Release Notes

TORO Integrate

Coder Studio

Coder Cloud

Bug Reports

Search

Deploying TORO Integrate with CloudFormation

Overview

TORO Integrate's flexible deployment models enable it to be deployed on a server behind a corporate firewall or on a public cloud provider such as AWS, Azure, or the Google Cloud Platform.

For those looking for a fully managed service that includes software, hosting, patch management and monitoring, TORO Integrate is also available with a pay per use model on TORO Cloud. Hybrid deployments involving a combination of two or more of these deployment models are also possible and sometimes advisable depending on the use case.

To simplify enterprise class deployments in the cloud, TORO has made available a flexible and customizable AWS CloudFormation Template. The AWS CloudFormation Template for TORO Integrate Enterprise Edition offers enterprise customers a simple way to configure a three tier scalable architecture that leverages all of the benefits of AWS’s extensive range of services. The template was designed, built, and tested by the same experts that manage the TORO Cloud infrastructure on AWS.

Introduction

Amazon Web Services (AWS) is the world’s leading cloud infrastructure solutions provider, providing a flexible and economical alternative to organisations deploying bare metal servers in a private data center whilst offering a breadth of services and quality of service demanded by some of the world’s largest brands. AWS CloudFormation dramatically simplifies the deployment of applications, servers and services on AWS. An AWS CloudFormation template defines a stack of AWS servers and services and provides a mechanism to reference recipes that can automate network and application configuration.

CloudFormation template wherein configurations of your network are defined in either JSON or YAML format. You may deploy an infrastructure or environment that best serves the requirements of your application and anticipated transaction volume. The table below illustrates the AWS services used in the AWS CloudFormation template, their purpose, and TORO's implementation.

Services

AWS Service Function and Benefit TORO’s Implementation
AWS CloudFormation Deployment: Simplify provisioning and managing a collection of related AWS resources and updating them in an orderly manner. Deploy resources in AWS that have been inspected and passed TORO’s standards and security policies.
AWS Elastic Container Service (ECS) Container management service: Docker container orchestration and management on EC2 instances. Automatic scaling of production containers managed by ECS. Automatic health checks also verifies redundancy.
AWS Elastic Cloud Computing (EC2) Virtual servers in the cloud: Elastic and on demand compute capacity. By default the template will provision micro server instances that are available under AWS’s free billing tier. The server specifications can be changed during or after deployment to any EC2 server size to suit the sizing of the target application.
AWS Virtual Private Cloud (VPC) Virtual network: Complete control over your own virtual network. Resources are managed and isolated according to their corresponding role or group which makes it easier to determine their permissions and access. This setup has been one of the best practices in the industry.
AWS Relational Database Service (RDS) SQL database: A managed and scalable SQL database. By default the template will provision a MySQL instance on RDS. The database is automatically replicated to another instance in a separate availability zone. RDS will automatically backup the database and switch to the failover replica in case of an outage in the primary database.
AWS CloudWatch Monitoring and logging: Collect and track metrics, collect and monitor log files, set alarms, and automatically react to changes in AWS resources. Every resource has a corresponding monitoring agent and an alerting scheme.
AWS OpsWorks Auto scaling and EC2 manager: Autoscale instances on demand. Easily manage the resources in your infrastructure. All instances in the VPC are managed by OpsWorks. With this, System Administrators don’t need to manually provision instances and worry about its OS, upgrades, and the applications installed in all VMs. With OpsWorks, we can manage one or 1000 servers without affecting the workflow
AWS Elastic File System (EFS) Central file storage: Elastic file storage that can be accessed from multiple EC2 instances at the same time. AWS EFS enables the docker containers to have its data retained even when a container is transferred to another EC2 instance. We don’t need to worry about performance as it has Max I/O performance mode.
AWS Simple Notification Service (SNS) Notification service: Managed messaging service. With AWS SNS, separating notifications into different priority tiers can easily be managed. We can configure whether an EC2 instance failure for a system check will trigger a notification to the proper team or just send the notification to an Enterprise Service Bus (ESB), like TORO Integrate, to digest the content and execute the proper remedy.
AWS Route 53 DNS: Managed Domain Name System web service connects user requests to infrastructure running in AWS. Route 53 organizes all your domain names. It also provides a private domain functionality for your internal resources and tools.
AWS Certificate Manager SSL Certificate Manager: Provision, manage and deploy SSL certificates within AWS Resources. Through AWS Certificate manager, it can be determined if an SSL certificate needs to be renewed. It can also generate a custom SSL certificate for an Elastic Load balancer (ELB) or CloudFront distribution.

Benefits

The AWS CloudFormation template for TORO Integrate Enterprise Edition addresses the key challenges of deploying a scalable enterprise application in the cloud. Users will benefit from AWS’s managed infrastructure, by minimizing downtime with automatic fail-over and recovery, monitoring services to minimize time and costs spent on troubleshooting issues, whilst leveraging the expertise of TORO’s in-house team of system administrators to ensure that an organisation’s deployment of TORO Integrate is deployed on tested and proven technologies whilst applying the industry’s best practices.

Below is a diagram illustrating the topology configured in the AWS CloudFormation.

cloudfomation-design

The architecture is deployed in an AWS Region, which contains one Amazon Virtual Private Cloud (Amazon VPC) to manage the virtual networking environment where in this VPC contains Availability Zones.

There are 2 Availability Zones in this design, Zone B and Zone C. It also consists of 4 divisions namely, the external network (essentially the Internet), Tier 1, Tier 2, and Tier 3.

The Amazon EC2 resources specified for this design can be automatically configured and managed using a service called AWS OpsWorks.

The EC2 instances in this design are also configured with Amazon Elastic File System (Amazon EFS), which is a service offered by Amazon that lets you configure scalable elastic file storages that you can mount on EC2 instances.

You can check out more information about this design here

If you have made the choice or even the consideration to subscribe to Amazon Web Services, you can checkout our brochure for TORO Integrate Enterprise for AWS, which includes more information on this deployment option.

How to Deploy TORO Integrate Enterprise Edition with CloudFormation

This section discusses how to deploy the AWS Cloudformation Template for TORO Integrate.

Permissions and Resources

  • The user performing the following steps will need to have permission to provision instances.
  • It should be noted that these steps will provision a number of AWS resources.

AWS Free Tier

By default all instances are micro instances which means you can deploy it for free provided that your account is eligible for a Free Tier on AWS. You can check this page out for more information. The default instance size can be changed during deployment.

  1. Download the template from the TORO website.

  2. Proceed to CloudFormation by choosing CloudFormation on the list of services or by clicking the drop down menu located at the upper left corner for Services and choosing CloudFormation.

    AWS Cloudformation

  3. Select the button that says Create a New Stack.

    Create Stack

    AWS Region

    The recommended region for this deployment is in North Virginia (us-east-1) make sure you select the right region. You can check your region by looking at the upper right menu.

    AWS Regions

  4. Choose to upload a template to Amazon S3 and then browse to select the template downloaded earlier.

    Choose Template

  5. Input your custom data.

    Input Data

    Below is a table specifying the parameters used for this step and their purpose.

    Parameter Purpose
    Stack name Choose a name for your infrastructure.
    AllowCidrBlock Enter a CIDR Block to Allow Access to your Network e.g. 0.0.0.0/0 (Optional).
    DBAllocatedStorage Determine the size of the database (in GB).
    DBInstanceClass Specify the database instance type.
    DbMasterPassword Configure a database master password.
    DbMasterUser Configure a database master username. It is not recommended to use root.
    Email Specify email address to notify when a CloudWatch metric has triggered an alarm.
    MultiAZ Choose whether you would like a Multi-AZ master database which automatically creates a primary DB Instance and synchronously replicates the data to a standby instance in a different Availability Zone (AZ).
    OrganizationName Specify the name of your Organization/Company.
    ParentDomain Enter the DNS name of an existing Amazon Route 53 hosted zone.
    SslCertificateName Identify the name of an SSL certificate uploaded in AWS Certificate Manager that you want to use for TORO Integrate.
    T1BCidrBlock Enter CIDR Block for Tier 1 - Zone B Subnet.
    T1CCidrBlock Enter CIDR Block for Tier 1 - Zone C Subnet.
    T1WebEc2Type Specify WebServer EC2 instance type.
    T2ActivemqEc2Type Specify ActiveMQ EC2 instance type.
    T2BCidrBlock Enter CIDR Block for Tier 2 - Zone B Subnet.
    T2CCidrBlock Enter CIDR Block for Tier 2 - Zone C Subnet.
    T2EcsEc2Type Specify ECS Cluster EC2 Types. They will run the docker containers of TORO Integrate.
    T2SolrEc2Type Specify Solr Cloud EC2 instance type.
    T2ZookeeperEc2Type Specify Zookeeper EC2 instance type.
    T3BCidrBlock Enter CIDR Block for Tier 3 - Zone B Subnet.
    T3CCidrBlock Enter CIDR Block for Tier 3 - Zone C Subnet.
    VPCSubnetCidrBlock Enter VPC CIDR Block. (i.e. 10.0.0.0/16).
  6. Add Tags and Permissions (Optional).

    Tags and Permissions

  7. Review and Create.

    Review and Create Cloudformation Information

  8. Done! You’ll see in the Dashboard that it is being created. You can also monitor the deployment process of the CloudFormation template in the Events tab.

Viewing the Homepage of TORO Integrate

This portion of the documentation covers how to view the homepage of TORO Integrate by making A records on Amazon Web Services’ Route 53. Route 53 is a service available on AWS that is used to manage the Domain Name Service (DNS) of web services to direct users to web applications by translating domain names into numeric IP addresses. You may find more information about it here.

Below is a guide on how to create an A record which is a type of DNS record that will point your chosen domain to a specified IP address.

Field Concerns

If you have any concerns with the fields and what they are for, you can easily hover over the field and a short description will pop up.

Route53 Field Hover

Creating a Hosted Zone

  1. Proceed to the home page of the AWS Console and look for Route 53 under the category, Networking & Content Delivery.

    AWS Home Route 53

  2. Select Hosted Zones from the left menu bar.

  3. Choose Create Hosted Zone at the top portion of the screen.

  4. A form will appear on the left side of the screen. Fill in the details as you'd like.

  5. Click on Create.

    Create Hosted Zone

Hosted Zone Type

Please note that the accessibility of your domain will depend on your network and the type of Hosted Zone you selected upon creating it (Public Hosted Zone or Private Hosted Zone for Amazon VPC).

Creating a Record Set

  1. Select your domain and click on Go to Record Sets.

    Go Record Set

  2. Next, an option called Create Record Set should be available, click it.

  3. Fields to fill in the information for your record set will pop up on the right side of the screen.

  4. Modify the Value and enter NGINX’s IP address.

  5. Press Save Record Set.

  6. Open your web browser and enter the domain name on the address bar, to verify your changes.

    Create Record Set

Frequently Asked Questions (FAQ)

  1. Is there an additional cost for the template?
    • No, the template is free.
  2. Can we use any TORO Integrate edition?
    • The template deploys an instance of SolrCloud. Only the TORO Integrate Enterprise Edition supports SolrCloud.
  3. What is the estimated cost if this is deployed in AWS permanently?
    • It depends with what type of instances you’ve chosen for your EC2 instances and RDS instances. Data transfers and storage costs may also vary depending on the size of the data. You can use AWS’ Price Calculator for to compute for an estimate.
  4. If I removed or added any services or EC2 instances how will it affect the infrastructure?
    • Once the CloudFormation has been fully deployed, you can then use OpsWorks to add EC2 instances. As for removing a service, you can actually edit the template and it will warn you if a resource has a dependency and cannot be deleted.
  5. Do I need to do or configure anything before or after deploying this infrastructure?
    • No. Everything is production ready.

AWS CloudFormation Infrastructure Modifications

Migrating from Bare Metal Servers to AWS

If you are looking to migrate the data from your local Linux servers to AWS EC2 instances, you may refer to this portion of the document. Before proceeding, you must successfully deploy the infrastructure using the Cloudformation template. Make sure that the AWS EC2 instances exist. These are Linux instances that we will be migrating to.

Prerequisites

  1. First, we need to make sure that you have the credentials to access your AWS EC2 instance.
  2. You need to make sure that you have ROOT access for both the source and destination server instances.
  3. Your SSH ports should be set to default (Port 22).
  4. You must have screen installed (yum install -y screen)

Passwordless SSH

You need to configure a temporary passwordless configuration to achieve a connection from your source server to the destination server.You may achieve this with the following instructions.

  1. Login to your local server and generate a public/private key pair, if you don’t have one already.
1
ssh-keygen -t rsa
  1. Use ssh to create an ssh directory for the source server. This directory may already exist, so if you are sure it does you can skip this step.
1
ssh user@EC2IP mkdir -p .ssh
  1. Finally, you may append the public key of your local server to the EC2 server.

1
cat .ssh/id_rsa.pub | ssh user@EC2IP 'cat >> .ssh/authorized_keys'
4. Verify the passwordless SSH by trying to connect to it.

1
ssh user@EC2IP

Script Assisted Migration

In this next portion of the document, you will need to run a script to assist you in migrating your resources.

It is recommended that you install screen on the source server so that you can maintain a persistent session during this potentially long process. If you would like to install screen, the package name should be “screen” in most distributions. You may use your package manager to perform the installation.

  1. We start by launching a screen session with the command:
1
screen -S OCSMigration
  1. If you get disconnected, you can run this command to resume the session after logging back in:
1
screen -r
  1. Now it is time to retrieve the script. You may retrieve it here.
1
wget http://link.com/rsyncmigration.sh
  1. Make sure it is executable by running the following command:
1
chmod +x rsyncmigration.sh
  1. Once you are able to retrieve the script, you may now execute it by running:
1
bash ./rsyncmigration.sh
  • It will prompt you for an IP address of the target server.
  • Then it will ask you what directories you would like to migrate over. It will ask you the path for the /data, /jdbc-pool, /logs, /packages, /system-tmp, /code and /tmp files. Afterwards, the script will attempt to a connection with the destination server as ROOT.
  • After that it will ask you to specify your organization name which will be relative to the path which was created for you during the CloudFormation setup. The syntax goes as : /datastore/clients/${ORGNAME}/apps/integrate/${ORGNAME}1/assets.
  • It will ask you to verify that path and to make sure it exists in the destination server as there is no way to check if the directory exists or not.
  • After you have authenticated, the script will attempt to setup a key based authentication between the source and destination server. Make sure that your ssh port is 22, at least for the migration.
  • The script will attempt to verify the installation of rsync on both servers.
  • It will run rsync twice. First to attempt to migrate the data and second to perform a final sweep in an attempt to follow up on any missed files or any files that may have changes during the first sync.
  • The script should be finished. Verify if the files have been successfully migrated. Once verified, you may now use this data for your TORO Integrate instance.

Manually Removing Instances

Manually removing instances after the deployment of the Cloudformation template is also another way to customize your network architecture on AWS.

AWS OpsWorks Instances

You will have a list of instances organized in stacks and layers in OpsWorks after the deployment of your Cloudformation template.

  1. Proceed to OpsWorks by clicking the drop down menu at the upper left corner for Services and choosing OpsWorks.

    Select OpsWorks Service

  2. Next, click the stack you’d like to modify.

    Select OpsWorks Stack

  3. Click Instances on the right side.

    Select Instances

  4. Stop the instance that you want to delete if it is running.

    Stop Instance

  5. Press Delete.

    Delete Instance

  6. Confirm the deletion, and you’re all set.

    Confirm Deletion

Amazon RDS Instances

Amazon Relational Database Service (Amazon RDS) is a service that is used to set up, operate, and scale a relational database in the cloud.

The basic building block of Amazon RDS is the RDS instance. A RDS instance is an isolated database environment in the cloud. An RDS instance can contain multiple user-created databases, and you can access it by using the same tools and applications that you use with a stand-alone database instance. This portion of the documentation covers how to delete an RDS instance.

  1. Proceed to Relational Database Service by clicking the drop down menu at the upper left corner for Services and choosing Relational Database Service.

    Select RDS Service

  2. Select Instances at the menu bar on the left.

    Select RDS Instances

  3. Select the RDS instance you want to remove then click Instance actions at the far right, then Delete.

    RDS Instance Options

  4. Click Delete.

    Delete RDS Instance

  5. Make a choice whether to take snapshots of your database or not and proceed.

    Confirm Snapshots Deletion

  6. Press Delete and you should be all set.

Summary

TORO Integrate Enterprise Edition comes with a deployment option configured using an AWS CloudFormation template. This template will bootstrap or automatically deploy your infrastructure, which includes TORO Integrate, at Amazon Web Services (AWS). It will also make use of as many AWS managed resources and services as possible to minimise the amount of server and infrustucture management required. This deployment method is useful for organisations who'd like to migrate to the cloud and gain all the benefits of deploying an infrastructure in AWS that's customizable, flexible, manageable, and easy to deploy.