Hivelocity Inc.

Livecot Architecture page 1

AWS

Traffic IN

Traffic OUT
Internet Gateway

public subnet : 172.31.60.0/24

IP1
54.238.144.233
IP3 IP4 livecot.com / api.livecot.com
IP1 IP2 Websites and API servers

IP2
54.238.159.40
mailer.livecot.com
manager DEV
HAProxy NAT
repo + redis sessions all NAT server for outbound traffic from
Load balancer
private subnet.

private subnet : 172.31.61.0/24

IP3
54.238.139.73
manager.livecot.com
Manager and Repo server

IP4
54.238.142.142
API API worker worker worker worker
website website dev.livecot.com / dev.api.livecot.com
auto scale depending on emails volume DEV server (does all)
databases

NOT YET

redis queues rdb master rdb slave

(emails) write only read only
 Hivelocity Inc.

Livecot Server Life Cycle page 2

Initialization from EC2 console

Intance creation from Amazon Machine Image (AMI)
Instance creation from AWS EC2 console Estimated time : 5 minutes

Instance has started

The server checks if there is already a livecot_server.php file.
If yes, the server contacts the manager to get the latest settings.
START
CRON jobs are running but not working
until the livecot_server.php has been received

YES
livecot_server.php
exists ?

NO UPDATE SETTINGS

Instance is running
The server checks if there is already a livecot_server.php file.
If yes, the server performs the CRON job unless a SHUTDOWN action
has been required.
wait next CRON job ...
If no, the server just wait until its running mode has been defined
In that case, this requires a manual action from an administrator :
artisan livecot:control start --mode=worker

CRON job

YES
livecot_server.php
exists ?

NO
YES

manually get settings is shutting down ?

somewhere here
artisan livecot:control start --mode=worker

mode is one of the following and won't NO

change until manual update :

- worker
- api perform CRON jobs
- rdb

wait next CRON job ...

Instance is shutting down

The server contacts the manager to let it know that it is shutting down.
SHUTDOWN The manager will try to contact the server later to see if it is still running or not.

Inform the manager

 Hivelocity Inc.

Livecot Architecture v2 page 3

Notes:
IP
1- Another HAProxy can be placed in front
Management Layer
public access Background of many mailer instances if necessary
MANAGER Tasks
HAPROXY
NAT + REPOSITORY
LOAD BALANCER 2- Many Getter instances may be
+ REDIS
necessary in case of high volume of
1- Check newsletters newsletter emails to send in a short period
Allows IN traffic Allows OUT traffic
2- Manage instances of time.
3- Store sessions
3- Logs saved in S3 must include
information about the instance, like the
Application Layer instance_id

API + WEBSITE API + WEBSITE MAILER #1 GETTER 4- Can we combine the NAT and HAProxy
#1 #2 POSTFIX + REDIS #1 in one instance?

1- Queue emails Get newsletter recipients 5- Instance role is defined by:

2- Sends emails a- property in text file?
(1 image per role)
b- environment variable?
(reliable? by api ok! by console??)
Data Layer c- property in php config received
from manager?
MySQL MySQL
MASTER (WRITE) SLAVE (READ)
AWS S3 (need manager but universal image)
d- private IP range?

1- Keep logs
 Hivelocity Inc.

Livecot Architecture : IP addresses ranges page 4

Private network: 172.31.62.0/23

IP count: 507 (510 - 3 for admin)
Min IP: 172.31.62.1
Max IP: 172.31.63.254

Master Slave API API Getter Getters Mailer Mailers

(write) (read) (#1) (#2) (persistent) (pool) (persistent) (pool)

1 10 20 50 100 125 254

Ctrl / Redis RDB API / Web getters mailers

1~10 2~30 1~50 1~25 1~129

Notes:
1- Based on DEV server alone, one Getter should be able to load 10,000 recipients in 2.5 min. It's 240,000 in one hour. 25 servers could get 6 million emails in one
hour. Taking into consideration a probably slower response time from clients,4 million emails could still be acceptable (see the sending capacity below).

2- Based on DEV server alone, one Mailer should be able to send at least 10,000 emails in 30 minutes. That's 20,000 emails per hour. 130 servers should be able to
send 2,600,000 emails per hour. Even more with better server specs (DEV server was only a c1.small)
 Hivelocity Inc.

Livecot Architecture : IP addresses page 5

Notes:
IP

Management Layer
public access
MANAGER
DON'T RELEASE THE
HAPROXY
LOAD BALANCER
NAT + REPOSITORY
+ REDIS
FOLLOWING IP
ADDRESSES !
54.238.144.233 54.238.159.40 54.238.139.73
172.31.60.40
! 172.31.60.10
! 172.31.61.60 AWS registered them in whitelists.

54.238.144.233
Application Layer livecot.com
api.livecot.com
API + WEBSITE API + WEBSITE MAILER #1 GETTER
#1 #2 POSTFIX + REDIS #1 54.238.159.40
mailer.livecot.com
172.31.61.160 172.31.61.161 172.31.61.100

Data Layer

MySQL MySQL
AWS S3
MASTER (WRITE) SLAVE (READ)

172.31.61.20 172.31.61.30
 Hivelocity Inc.

Livecot Transaction Email Flow Part 1: API Instance page 6

API Instance Flow

Handle request
a1 routing, validation, process
Management Layer
public access
MANAGER
HAPROXY Save transaction email into MySQL
NAT + REPOSITORY
LOAD BALANCER a2 receive id
+ REDIS

Allows IN traffic Allows OUT traffic 1- Check newsletters

2- Manage instances Put email into redis queue
3- Store sessions a3 mailer will send when idle
(email may not be sent right away)
a1 a4
Return response
Application Layer a4 response contains email information
with id but no send status
API + WEBSITE API + WEBSITE MAILER #1 GETTER
#1 #2 POSTFIX + REDIS #1

1- Queue emails Get newsletter recipients

a2 2- Sends emails
a3

Data Layer

MySQL MySQL
AWS S3
MASTER (WRITE) SLAVE (READ)

1- Keep logs
 Hivelocity Inc.

Livecot Transaction Email Flow Part 2: Mailer Instance page 7

Mailer Instance Flow

POP Email from redis queue
b1 insert custom fields, rewrite urls
Management Layer
public access
MANAGER
HAPROXY Send email to Postfix
NAT + REPOSITORY
LOAD BALANCER b2 receive queue id if email was sent
+ REDIS

Allows IN traffic Allows OUT traffic 1- Check newsletters

2- Manage instances Update MySQL
3- Store sessions b3 update the queue id, queued_at and
error_code if email was invalid
b2
Application Layer
b1
API + WEBSITE API + WEBSITE MAILER #1 GETTER
#1 #2 POSTFIX + REDIS #1

b3 1- Queue emails Get newsletter recipients

2- Sends emails

Data Layer

MySQL MySQL
AWS S3
MASTER (WRITE) SLAVE (READ)

1- Keep logs
 Hivelocity Inc.

Livecot Newsletter Email Flow Part 1: Prepare requests to client page 8

Instance Flow
Check newsletters
a1 CRON job every minute
Management Layer
prepare requests: offsets, limits
public access
MANAGER
HAPROXY Put get request in queue
NAT + REPOSITORY
LOAD BALANCER a2 getter can be dedicated instance or
+ REDIS
manager instance.
For now we consider a dedicated
Allows IN traffic Allows OUT traffic a1 1- Check newsletters a2 instance.
2- Manage instances
3- Store sessions

Application Layer

API + WEBSITE API + WEBSITE MAILER #1 GETTER

#1 #2 POSTFIX + REDIS #1

1- Queue emails Get newsletter recipients

2- Sends emails

Data Layer

MySQL MySQL
AWS S3
MASTER (WRITE) SLAVE (READ)

1- Keep logs
 Hivelocity Inc.

Livecot Newsletter Email Flow Part 2: Send requests to client page 9

Instance Flow
POP get request
b1 check client URL, prepare request with
Management Layer
credentials, offset and limit
public access
MANAGER
HAPROXY Request recipients
NAT + REPOSITORY
LOAD BALANCER b2 limit 1000
+ REDIS

Allows IN traffic Allows OUT traffic 1- Check newsletters

2- Manage instances Save recipient and total in MySQL
3- Store sessions b3 get email id,
compute sub-requests,
launch sub-process if necessary
b2
Application Layer Put each email in mailer queue
b4 mailer will send when idle
API + WEBSITE API + WEBSITE MAILER #1 GETTER (email may not be sent right away)
#1 #2 POSTFIX + REDIS #1
go back to b2 until all recipients are
received or the maximum amount of
1- Queue emails Get newsletter recipients failed requests is reached.
2- Sends emails
b4 b1 b3

Data Layer

MySQL MySQL
AWS S3
MASTER (WRITE) SLAVE (READ)

1- Keep logs
 Hivelocity Inc.

Livecot Newsletter Email Flow Part 3: Send newsletter emails page 10

See: Livecot Transaction Email Flow Part 2: Mailer Instance

We use a redis queue dedicated to newsletter emails but the process is the same.
Another difference is that the newsletter_emails table will be used when updating the email in MySQL
 Hivelocity Inc.

Livecot CRON jobs page 11

CRON Jobs
- CRON jobs are performed every minute
- scripts are loaded by task/taskboot.php
- scripts are located at Livecot/Task/...
- scripts are executed like this:
php path/to/task/taskboot.php [env] [type] [action]

Management Layer Mailer newsletter (1)

1 - listen newsletter emails queue
public access
MANAGER - send emails
HAPROXY
NAT + REPOSITORY
LOAD BALANCER
+ REDIS Mailer transaction (1)
2 - listen transaction emails queue
- send emails
Allows IN traffic Allows OUT traffic
4 5 6 7 8
Mailer delivery
3 - checks mail log and update emails
status

Application Layer Newsletter checkNewsletters (1)

4 - get newsletters that must be sent
API + WEBSITE API + WEBSITE MAILER #1 GETTER create processes
#1 #2 POSTFIX + REDIS #1
Newsletter checkProcesses (1)
5 - check processes and update statistics

7 7 1 2 3 7 6 7 NewsletterGetter run (1)

6 - listen to newsletter processes queue
- get newsletter recipients from client server

Backup logs (1) (2)

Data Layer 7 - backup logs and other files to AWS S3
(this script doesn't exist yet)
MySQL MySQL
AWS S3
MASTER (WRITE) SLAVE (READ) Backup database (1) (2)
8 - backup db dump to AWS S3

1- Keep logs
7 7
Notes:
(1) : The process should be unique. Every minute
a CRON job is launched but the process
performs a check based on the script name and
its own process id (pid). If another pid is found for
the same script name, the job is cancelled.
This ensure that only one process is doing the
job, but if that process dies, it will be replaced by
another one in less than one minute.

(2) : The CRON job doesn't run every minute.

Frequency depends on the job.
 Hivelocity Inc.

Livecot Transaction Email Flow Part 3: Detail page 12

Scenario 1: API REQUEST Scenario 2: API REQUEST

Direct Sending Queueing
Handle request Handle request
The email is sent before (ApiTransactionEmailController) The email is registered in (ApiTransactionEmailController)

the response is returned to database but not sent

the client. right away. It is put in a
NO NO
Fields Fields
Return Error Return Error
Validated ? Validated ?
Redis queue, and will be
This is the current YES
processed by a worker YES

scenario. Register in RDB server. This results in a Register in RDB

faster API response time.

Send Send
(TransactionSender)
This is a work in progress (TransactionSender)

NO
to improve the API speed. NO

Use template ? Use template ?

YES YES

NO NO
Template Template
Return Error Return Error
exists ? exists ?

YES YES

Get template
Put the email in
the Redis Queue

Insert custom fields Return Email Info

(including id)

NO
The email will be sent later.
Use URLs
redirection ?
Most of the time it will be sent
YES
instantly, unless there is a high
amount of traffic. In that case, the
sending server will launch or wake
Check and replace URLs
up other sending servers to help.

Send Email

Return Sent Email

 Hivelocity Inc.

Livecot Manager page 13

Hivelocity Admin