TIBCO GridServer Manager

AWS Deployment Guide

Reference Architecture
Prerequisites and Requirements
Technical Requirements and Specialized Knowledge
GridServer Architecture on AWS
Architecture Diagram

Planning your Deployment

Security
Networking
Costs
Sizing

Deployment Guidance
Deployment Assets
Launching GridServer for AWS using CloudFormation
Prerequisites

Operational Guidance
Health Check
Backup and Recovery
Maintenance

Product Support

Further reading material

Introduction
TIBCO GridServer is a highly scalable software infrastructure that allows application services to operate
in a virtualized fashion, unattached to specific hardware resources. Client applications submit requests to
the Grid environment and GridServer dynamically provisions services to respond to the request. Multiple
client applications can submit multiple requests in parallel and GridServer dynamically creates multiple
service instances to handle requests in parallel on different Grid nodes. This architecture is therefore
highly scalable in both speed and throughput. For example, a single client will see scalable performance
gains in the processing of multiple requests, and many applications and users will see scalable
throughput of the aggregate load.
TIBCO GridServer enables a truly Service-oriented approach to Grid computing. The Grid client creates a
Service instance and calls operations (methods) on the Service. These calls are virtualized and invoked
on multiple Grid nodes. This virtualization provides limitless scalability for hosting these Services and
offers a simple programming model for invoking these Services.

A Service is a self-contained application or business function that is distributed to remote computing

resources. A client application makes requests, which are routed by GridServer to those resources, which
then return results to the client. One use of a Service is to load balance an application, whereby the client
application receives requests and submits them asynchronously to the Service, which then executes
these multiple, unrelated computations in parallel. Another use is to execute a single, larger computation.
The client splits the computation into multiple, independent pieces, submits them asynchronously, and
combines the returned results.

TIBCO GridServer stack has four different

components in its architecture:

● Grid Clients​ — The components that

submit service requests to the Grid. (Also
called Drivers).
● Engines​ — The processes that host and
run services on the Grid nodes.
● Brokers​ — The component that provides
request queuing, scheduling and
 load-balancing. Brokers are also responsible for managing Engines.
● Directors​ — The component that assigns Grid Clients and Engines to Brokers based on policy.
They manage and load balance Engines across available Brokers

Reference Architecture

Prerequisites and Requirements

Technical Requirements and Specialized Knowledge

In this article, we assume that you are already familiar with traditional GridServer components and
architecture. This guide supports TIBCO GridServer version 7.0 or greater. You need to obtain a TIBCO
GridServer ​software binaries for the Manager, Broker, and Engines​ and an appropriate license.
To request a license, contact TIBCO at ​https://www.tibco.com/contact-us​.
For a full list of the requirements please check the ​TIBCO GridServer Installation Guide​, or if you are new
to the product make sure you read the ​TIBCO GridServer Introduction Guide​.
Before you deploy a full stack of TIBCO GridServer on AWS, we recommend that you become familiar
with the following AWS services. Make sure to check the ​AWS Getting Started Resource Center​ if you are
not familiar with AWS.

● Amazon EC2
● Amazon EBS
● Amazon S3
● Amazon VPC
● Auto Scaling
● Identity and Access Management

Other AWS Services that are not required but recommended that you are familiar with:

● Amazon EC2 Spot Instances

● AWS Virtual Private Network
● Amazon RDS
 ● Amazon Aurora
● Amazon CloudWatch

GridServer Architecture on AWS

Below is a standard reference architecture for TIBCO GridServer running on AWS. This setup leverages
the use of ​Amazon Spot Fleet​ for the GridServer Engines which enables a cost-effective way to deploy an
elastic and highly available computing Grid
Using this deployment scenario is how we worked with AWS to set up and run a proof of concept grid in
the AWS Cloud that was able to scale up to 1.3 Million vCPUs. ​You can learn more details about this
project and its use case in this Blog Post​.

Architecture Diagram
Important considerations:

● All communication between Directors, Brokers, and Managers is done via HTTP over SSL
(HTTPS). For SSL configuration details refer to Chapter 8 of the ​TIBCO GridServer Installation
Guide​.
● For HA and fault tolerance brokers should be setup different AZs and the engine nodes
automatically will follow the AZ of the broker.
● The standard setup is to have an Active/Active configuration so all brokers and engines are
available for the Grid.
● The directors are setup as Primary and Secondary (each in a different AZ) If the Primary Director
fails, the Secondary Director takes over balancing and routing Drivers and Engines to Brokers.
The Directors do not maintain any state, so no work is lost if a Director fails and the secondary
takes over. Also, both Directors follow the same rules for routing to Brokers, so it makes no
difference which Director is used for login.
● Grid Clients (Drivers) also connect to the Grid using HTTPS, in the configuration below the clients
reside on the same VPC, if you need to have your clients on your data center (on-premise) you
will need to set up an ​AWS Site-to-Site VPN
● GridServer Authentication is commonly configured externally which is recommended, but if you
are testing or doing a POC you can use the GridServer internal authentication (this information is
encrypted and stored in GridServer metadata database)
● Amazon S3 and Amazon RDS can be used by the Grid Libraries as external resources that can
be used from Grid Libraries.

For more information on configuration, high availability, fault tolerance, authentication and configuration
please refer to the ​GridServer Administration Guide​.

Configuration Management
The Director is used to configure many settings of a GridServer Manager, including users and passwords,
Driver profiles, routing properties, and Engine configurations. These settings are configured with the
web-based GridServer Administration Tool.
GridServer can use an external database, in this case, Amazon RDS Postgres to store metadata (such as
User, Engine, Driver, and Broker information) and also reporting information (job statistics, log events,
etc.).
This database is accessed through JDBC and the configuration of the database used is changed on the
Director from the GridServer Administration Tool.

Below is a reference architecture diagram for a TIBCO GridServer deployment with 2 active/active AZs.
Engines can be run in any mix of on-demand, reserved and spot instances.

Planning your Deployment

Security

When deploying any workload in AWS, there are a number of different approaches to consider when
addressing identity and access management. Make sure you follow the best practices for protecting AWS
credentials, ensuring least privilege access via IAM users, roles and policies as outlined in the ​AWS Well
Architected Framework​.

On standard TIBCO GridServer where the complete workload is in AWS, no resources will be accessible
outside the VPC. If you are planning on a hybrid deployment where either the GridServer Drivers (Client)
or Directors will be hosted on-premise is recommended that you leverage ​AWS Direct Connect​ or other
VPN Solution to establish a private dedicated network connection from your premises to the AWS VPC
where the grid is deployed.

To enable GridServer to control the size of the Grid based on the current load. The HPCCA adapter in
GridServer Broker needs to have control of the Autoscaling Group that manages the GridServer Engines.
This is enabled via IAM policies as seen in the Cloud Formation Template under ​Launching GridServer
for AWS using CloudFormation​.
These policies allow actions including "aws:policy/AutoScalingFullAccess" ,
"aws:policy/AmazonEC2FullAccess" that will enable TIBCO GridServer HPCCA adapter to manage the
size of the Grid depending on the current load.

Please refer to AWS Documentation for best practices on ​AWS IAM​, ​VPC Security Groups and ACL​, etc.
We recommend you enable resource Tagging in AWS following the best practices outlined in ​AWS
Tagging Strategies​.

For more information on securing your deployment and installation check the ​GridServer Manager
Installation Guide
Networking

TIBCO GridServer Networking 
Since GridServer is a distributed computing application, successful

deployment depends on your network configuration.
Treat GridServer Managers the same way you treat your other mission-critical application servers: 

● Deploy GridServer in a VPC (see Reference Architecture)
● Assign GridServer Managers (Brokers and Directors) static IP addresses (Elastic IPs) and
resolvable DNS hostnames. For information on setting up DNS and EIPs in your AWS VPC see
https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html
● Non-static or DHCP-assigned addresses are acceptable for Engines and Drivers.


You can configure GridServer Engines and Drivers in several different ways. To receive the full benefit of
peer-to-peer communication you must enable communication between Engines and Drivers (default), but
you can also configure GridServer to work with a hub-and-spoke architecture by disabling Direct Data
Transfer.

Below is a network infrastructure guide with all the communication port used on a standard deployment
with SSL enabled to encrypt data in transit inside the VPC. We suggest you use a SSL certificate
manager like ​AWS Certificate Manager​ to simplify provision, management, and deployment of public and
private SSL/TLS certificates.
For detailed information manually enabling and creating certificates for GridServer follow the information
on ​this Knowledge Base Article​. For detailed information on the overall networking setup refer to
“Configure your network” in the ​GridServer Installation Guide
Costs

TIBCO GridServer allows you to run the Engines in any mix of on-demand, reserved or spot instances.
Since engines are by default fault tolerant they are a good target for EC2 Spot Instances. When an
Engine goes offline, the work assigned to it is requeued and assigned to another Engine.

GridServer Manager components (Directors and Brokers) should be run only in either on-demand or
reserved instances.

Running Brokers on the same OS as other components is not a requirement and gives no performance
benefit. However, running the same OS for all components does simplify administration and
troubleshooting.

AWS EC2 instances are charged per hour or per second depending on the instance type and OS. For
information about EC2 Pricing refer to AWS Amazon EC2 ​https://aws.amazon.com/ec2/pricing/

Sizing

You can deploy the GridServer Manager architecture to give varying degrees of redundancy and load
sharing, depending on the available computing resources.
To plan an architecture that best supports your needs:
● Determine how your facility wants to use GridServer.
● Estimate expected transaction volume and amount of work.
● Survey infrastructure and networking requirements for the expected workloads.
● Instead of using one grid for all types of Services, you might prefer to divide different subsets of
Services (for example, by size or priority) to different Directors.

Ensure that your environment meets the minimum hardware and software system requirements for
memory, disk space, and processor speed. Please see the ​readme file included in the product archive​ for
the complete list of all hardware and software requirements, including supported operating systems, Java
or .NET software, web browsers, and accompanying components. Note that depending on the
applications you will virtualize, you may need to exceed these requirements accordingly.

A minimal configuration of GridServer consists of a single Manager configured with a Primary Director
and a single Broker. To distribute load, add more Brokers to other Managers to your architecture by
adding additional Managers containing more Brokers or Directors to address redundancy, volume, or
other considerations.
In medium to larger grids, the number of Engines in the grid can require more capability than a single
Broker can provide depending on the tasks submitted. As a general rule you should never have more
than 3000 engines per broker.
Engine instance size will be dependant on the type of tasks and services that you will be running on the
grid and you can mix instance sized on the same Grid. You should use M, R and C instances based on
the memory vs. processor load of your tasks, if you have GridLibs that can leverage GPU processing P
instances are recommended.
Engines require a minimum of 2 cores and 2GB of RAM and 4 cores, 8GB RAM are recommended.

Burstable Performance EC2 Instances (T family) are not recommended for any of the TIBCO GridServer
components.

Please see the ​readme file included in the product archive​ for the complete list of all hardware and
software requirements

For detailed information on sizing refer to the ​GridServer Installation Guide​.

Deployment Guidance
For this section we assume that you have read and follow the ​Planning your Deployment​ section above.

GridServer provides a CloudFormation template allows you to automatically create all of the resources
needed for a standard Grid as shown in the ​GridServer Architecture on AWS​ section. These templates
are available with the product and are intended to give you a starting point that you can use to customize
for your specific deployment scenario.

This includes defining the AMI to be used for a new EC2 instances for managers and engines, defining
AutoScaling Groups, defining Security Group to be used for all resources, defining an Aurora Postgres
reporting database, defining an IAM Role to set the EC2 credentials, and defining policies to allow this
role to connect to AWS APIs.

Is important that you enable the proper logging for your Grid environment following the information on
Chapter 7 of the ​GridServer Manager Administration Guide​.

Deployment Assets

The provided CloudFormation template uses AMIs from AWS Marketplace if you are deploying this with
your own licenses you will have to install and build your own AMIs and modify the CloudFormation
Template accordingly. .
If you are using the Marketplace AMis make sure you have subscribed to GridServer in the AWS
Marketplace {LINK}, please launch GridServer using CloudFormation Stacks for improved security and
simpler connectivity.
If you launch directly with the EC2, then additional installation and configuration will be needed to build
the reference architecture. Please follow the information on the ​GridServer Installation Guide​.
Launching GridServer for AWS using CloudFormation

This section explains the best practices in launching GridServer's AMI on AWS Marketplace. The AMIs
should be launched via instructions on this page using CloudFormation Templates.
Follow the steps listed below to get your instance running.

1. Accept the License

2. Run CloudFormation Template for "GridServer for AWS"

Prerequisites

● Amazon AWS Account.

● Amazon Key Pair for security.
● Amazon Virtual Private Cloud with CIDR block large enough to take care of your needs for the
number of EC2 instances required for the Grid.
● Amazon Subnets. At least 2 subnets from different Availability Zones. It is recommended to use
private subnets so that GridServer Managers/Engines would be accessed only within VPC.
● Increased AWS Account limits to take care of your Grid needs for EC2 instances

Template Input Parameters

GridServer Configuration

➢ Admin User​ ​:​ ​Admin User for GridServer Manager login

➢ Admin Password : Admin password for GridServer Manager login
➢ Database User : Database user for internal database
➢ Database Password : Database password for internal database
➢ SSL Enabled ? : If the Managers will be SSL enabled or not.
➢ Manager Instance Type : Instance type for the Grid Manager (Directors and Brokers)
➢ Secured IP : IP address of machine from which all traffic would be allowed to Grid
Managers. If access from more machines need to be allowed, user would need to
update the security group after successful CFT deployment.
➢ Engine VM Count : Number of Engine VMs to be launched at template deployment time
➢ Minimum Engine VM Count : Minimum Number of VMs in an Engine Auto Scaling group
➢ Maximum Engine VM Count : Maximum Number of VMs in an Engine Auto Scaling
group
➢ On Demand Base Capacity : Number of first x instances designated as On-Demand VMs
from an Engine Auto Scaling Group
➢ On Demand Percentage Above Base Capacity : Percentage for On-Demand VMs from
total capacity of an Engine Auto Scaling Group
➢ Engine VM CPU Count : Number of CPU cores for the Engine VMs to be launched
Reporting Database Configuration

➢ Reporting Database Name : Database name for the PostgreSQL DB to be created

➢ Reporting Database User : Database user name to be created
➢ Reporting Database Password : Database password to be created
➢ Reporting Database Instance Type : Instance Type for the VM to create DB
➢ Reporting Database Subnet Ids * : Existing two subnet ids from different Availability
Zones where two DB instances would be created

Amazon EC2 Configuration

➢ Key Name * : Name of an existing EC2 KeyPair in current region

➢ KeyPair Content : KeyPair private key text contents with no spaces and \n added where
newline is required.This is required only when Grid is SSL enabled.
➢ VPC Id * : Id of an existing Virtual Private Cloud
➢ Subnet Ids * : Existing subnet ids where the Grid components would be launched. These
subnets should be from different availability zones.

2. Launching GridServer for AWS from EC2 Console

Launching a Grid manually in the EC2 console will work fine, but you'll miss out on several
features to make your life easier. Please note that launching a GridServer Manager manually in
the EC2 console is not supported for SSL enabled Grid However Engine can be launched
manually in the SSL enabled Grid.
It's recommended to always launch using the provided CloudFormation Templates.
Follow the steps listed below to get your instance running.

1. Accept the License

2. Launch EC2 instance based on GridServer Manager AMI
3. Launch EC2 instance based on GridServer Engine AMI

Prerequisites
➢ Amazon AWS Account.
➢ Amazon Key Pair for security.
➢ Amazon Virtual Private Cloud with CIDR block big enough to take care of your needs for
the number of ec2 instances.
➢ Amazon Subnets. At least 2 subnets from different Availability Zones.
 ➢ Increased account limits to take care of your Grid needs for ec2 instances
➢ Amazon Security Group to be assigned to the GridServer Manager/Engine
➢ Amazon AutoScaling Group to be used for launching Engines. This should have
instance types having the same number of CPU cores.
➢ Amazon Instance Profile with a Role with “AutoScalingFullAccess” and
“AmazonEC2FullAccess” permissions. Please assign this profile at the time of launching
Primary Broker.
➢ Amazon Aurora DB cluster for aurora-postgresql9.6.

2.a​. ​User Data for GridServer for AWS Manager AMI

Note : Reporting DB related data below needs to be provided from an existing Aurora DB for
aurora-postgresql9.6. Also note that all the values to be provided in double quotes as shown in
examples.

- Primary Director

Manager Type : “Director”

Manager Sub Type : “Primary”
Admin User : <admin user>
Admin Passowrd : <admin password>
Database User : <database user>
Database Password : <database password>
Reporting DB URL : <reporting db connection url>
Reporting DB User : <reporting db user name>
Reporting DB Password : <reporting db user password>

Example :
"Director","Primary","admin","admin","admin","admin",”jdbc:postgresql://localhost:5432/griddb”,”gsadmin”,”gsadmin”

- Secondary Director

Manager Type - “Director”

Manager Sub Type - “Secondary”
Admin User : <admin user>
Admin Passowrd : <admin password>
Database User : <database user>
Database Password : <database password>
Reporting DB URL : <reporting db connection url>
Reporting DB User : <reporting db user name>
Reporting DB Password : <reporting db user password>
Primary Director Hostname : FQDN of Primary Director
Example:
"Director","Secondary","admin","admin","admin","admin",”jdbc:postgresql://localhost:5432/griddb”,”gsadmin”,”gsadmin”,”​ip-172-31-1
1-245.us-east-2.compute.internal​”

- Primary Broker

Manager Type : “Broker”

Manager Sub Type : “Enabled”
Admin User : <admin user>
Admin Passowrd : <admin password>
Database User : <database user>
Database Password : <database password>
Reporting DB URL : <reporting db connection url>
Reporting DB User : <reporting db user name>
Reporting DB Password : <reporting db user password>
Primary Director Hostname : FQDN of Primary Director
Secondary Director Hostname : FQDN of Secondary Director
Wait handle : NA
Is SSL : NA
Key Name : NA
Key Pair : NA
SSL keystore Host : NA
SSL Hosts List : NA
AWS Region : <region>
Autoscaling Group : <autoscaling group name> → Name of an existing AutoScaling
Group created for Engine VMs
Launch Template Name : <launch template name> → Name of an existing
LaunchTemplate used for AutoScaling Group of Engine VMs.
Launch Template Version : <launch template version> → Version of an existing
LaunchTemplate.
Instance CPU Core : <number> → CPU core of instance types used while creating
AutoScaling Group
Instance Type List : <comma separated instance type list> → List of instance types used
while creating AutoScaling Group>

Example :
"Broker","Enabled","admin","admin","admin","admin",”jdbc:postgresql://localhost:5432/griddb”,”gsadmin”,”gsadmin”,"​ip-172-31-11-24
5.us-east-2.compute.internal​","​ip-172-33-10-100.us-east-2.compute.internal​","","","","","",””,"1","1000","us-east-2","GSAutoscalingGro
up","GSLaunchTemplateName","1","2","t2.medium,t2.large,t3a.small"

- Failover Broker

Manager Type : “Broker”

 Manager Sub Type : “Failover”
Admin User : <admin user>
Admin Password : <admin password>
Database User : <database user>
Database Password : <database password>
Reporting DB URL : <reporting db connection url>
Reporting DB User : <reporting db user name>
Reporting DB Password : <reporting db user password>
Primary Director Hostname : FQDN of Primary Director
Secondary Director Hostname : FQDN of Secondary Director

Example:
"Broker","Failover","admin","admin","admin","admin",”jdbc:postgresql://localhost:5432/griddb”,”gsadmin”,”gsadmin”,"ip-172-31-11-24
5.us-east-2.compute.internal","ip-172-33-10-100.us-east-2.compute.internal"

2.b. User Data for GridServer for AWS Engine AMI

Primary Director Hostname : FQDN of Primary Director

Is SSL ? : true
Key Name : <key name>
Key Pair Private Key Content : < key pair content> → as a string with no spaces and /n
added for new line wherever required.

Example for non-ssl:

Ip-172-31-11-245.us-east-2.compute.internal

Example for ssl:

Ip-172-31-11-245.us-east-2.compute.internal​,true,gskeypair,-----BEGIN RSA PRIVATE
KEY-----\nMIIEogIBchKCAQEAltuYqfH1vdiE/qMEOjUq+rJladJ2MHPVZjZFe3Cks32L83RcUSDaZd/0qzbn\nXoaZX9WijM4mtneSshI
dFWK0UgFk6DXit0QjPzkSWjVsV7R9SEbOUJqrdHwwUHjpjFTYRTzt0YhI\nTED4DHPhEzO8UG3KKZUSebonwiASm2/9D5rlpRnkb
aUiASXiZW0UC7TSQ1MEtL8gt/Ji0A/XlCSISJ8af15w2Zl93qn\ndCmvZfgEyIB5+A+km4jIksPOdqJ2B/G03ekX3SyK4pwTAoGAD4CT
I5kOTycZ2Ua+lVVZtuzfunx9\n4PfgDYJsLjqkpsT+6ewqeRRwf/tBamZMEjE5/N86tGQA1ZCuM8ioqdPUvdqcMl8NGoGDjlsHr9Bs\n3d
cNrAqGiqeq8IFZxjL3WpeDtwgt1aJSYjQw5IRqycZQbVSE6nHS5tdoOf1q3qmUUmHc=\n-----END RSA PRIVATE KEY-----

Operational Guidance

Health Check

TIBCO GridServer provides health check information on the GridServer Administration Tool, this is a
web-based tool that enables the GridServer administrator to monitor and manage the Manager, its
Engines, and Drivers.
For more information on the administration tool refer to “Chapter 2 The GridServer Administration Tool”
on the ​GridServer Manager Administration Guide

The GridServer Administration Tool will enable you to you can:

● Monitor Service and task execution and cancel Services
● Monitor Engine. Directors and Brokers activity, status, tasks, services and machine status,
including thread and memory information.
● Diagnose issues and monitor the grid
● View and modify Manager and Engine configuration
● Setup notification for specific events
● Edit Engine properties and change values
● Configure routing of Drivers and Engines to Brokers
Backup and Recovery

Backup of GridServer Managers requires an OS-level file copy of the DS_DATA directory, this directory is
the directory that is specified by DS_DATA_DIR configuration variable. Since you only need a OS-level
file copy, you can setup a cron job that runs on the Directors and Brokers that will backup the directory
and move it to S3 or other external storage. You can also use ​AWS DataSync​ to have this backups
shipped to another AWS account or another Region for DR purposes.

The Reporting Database, if you follow the Deployment instructions above it will be setup as an RDS
Aurora Postgres database so you can follow the standard for ​Amazon RDS Backup and Restore​.

GridServer Managers also support the ability to export the configuration and deployment of Directors and
Brokers into a signed JAR file format and later import this same format to another Manager.
You can migrate Manager configurations from a UAT or prototype grid to a production grid, simplify
administration of multiple Manager systems, or disseminate an organization’s default Manager
configuration among all clusters in the organization.

For a DR scenario in case of a complete AWS region failure, we recommend shipping the backups to
another AWS region as part of your backup strategy.
On that secondary region you can have a minimal Grid deployment (one director, one broker and 1
engine running as a “pilot light” deployment. When you need to bring the second region in production you
can scale up the grid. Note that you will need to reconfigure all your Grid Clients (Drivers) to point to the
Grid on the new region.
For detailed information on the backup and recovery procedures and strategies see “Backup / Restore”
Section on the ​GridServer Manager Administration Guide​.

Maintenance

For maintenance and troubleshooting information refer to Chapter 7 “Administration and Maintenance”
and Chapter 9 “Diagnosing GridServer Issues” of the ​GridServer Manager Administration Guide

When using SSL, you will need to keep track of certificate expiration and u​pdate the certificate store on
the GridServer components​ accordingly when you have newly issued certificates.

Product Support

For an overview of TIBCO Support including different support tiers and corresponding SLAs, visit
http://www.tibco.com/services/support.

To access the Support Knowledge Base and getting personalized content about the products you are
interested in, visit the TIBCO Support portal at https://support.tibco.com.

For creating a Support case, you must have a valid maintenance or support contract with TIBCO. You
also need a user name and password to log in to ​https://support.tibco.com​. If you do not have a user
name, you can request one by clicking “Register” on the website
Further reading material
● Full Product Documentation ​TIBCO GridServer Documentation
● Deploying Massively Scalable Computing on AWS with TIBCO GridServer
● Solution Brief "Managing Market Risk with Extreme Speed and Scalability at Lower Cost"
● Whitepaper "A Cost-effective Solution to New FRTB Computing Requirements"
● TIBCO GridServer Datasheet