This​ ​is​ ​an​ ​example​ ​tech​ ​spec,​ ​written​ ​to​ ​illustrate​ ​[this​ ​blog​ ​post]​.

● Commentary​ ​(like​ ​this)​ ​has​ ​a​ ​yellow​ ​hilight.

● I’d​ ​suggest​ ​you​ ​read​ ​the​ ​spec​ ​one​ ​through​ ​ignoring​ ​the​ ​hilighted​ ​parts,​ ​then​ ​go​ ​back​ ​and
read​ ​the​ ​full​ ​document​ ​again.
● Please​ ​don’t​ ​get​ ​too​ ​worked​ ​up​ ​about​ ​the​ ​specific​ ​proposal​ ​here​ ​-​ ​the​ ​point​ ​is​ ​to​ ​illustrate
a​ ​tech​ ​spec​ ​in​ ​general,​ ​not​ ​to​ ​talk​ ​about​ ​technical​ ​decisions.
● Not​ ​all​ ​sections​ ​are​ ​required,​ ​or​ ​to​ ​this​ ​degree​ ​of​ ​chattiness.​ ​Yours​ ​might​ ​be​ ​much
shorter.

Tech​ ​Spec​ ​-​ ​Transactional​ ​Emails

November​ ​1,​ ​2017
Author:​ ​Chuck​ ​Groom​ ​[​ ​There​ ​should​ ​be​ ​only​ ​one​ ​author​ ​-​ ​see​ ​the​ ​blog​ ​post​ ​for​ ​details​ ​]
Team:​ ​Louise​ ​Wagner​ ​and​ ​Colin​ ​Mullen​ ​[​ ​…​ ​but​ ​of​ ​course,​ ​give​ ​credit​ ​where​ ​it’s​ ​due!​ ​]

Overview
[​ ​Link​ ​to​ ​the​ ​product​ ​spec​ ​and​ ​supporting​ ​documents,​ ​if​ ​they​ ​exist]

We​ ​currently​ ​use​ ​the​ ​default​ ​mailing​ ​emailer​ ​system​ ​included​ ​in​ ​our​ ​web​ ​framework.​ ​Templates
are​ ​checked​ ​in​ ​with​ ​code​ ​and​ ​are​ ​part​ ​of​ ​the​ ​release​ ​cycle;​ ​and​ ​we​ ​use​ ​our​ ​own​ ​SMTP​ ​server.

This​ ​solution​ ​has​ ​limitations,​ ​and​ ​should​ ​be​ ​improved;​ ​see​ ​ticket​ ​#1845​.​ ​We​ ​want​ ​to​ ​move
transactional​ ​emails​ ​to​ ​a​ ​hosted​ ​3rd​ ​party​ ​service​ ​that​ ​supports​ ​templates​ ​and​ ​tracking
delivery/open/click​ ​rates.​ ​We​ ​propose​ ​using​ ​Mailchimp.​ ​Send-email​ ​jobs​ ​will​ ​be​ ​enqueued​ ​to
RabbitMQ,​ ​and​ ​consumed​ ​by​ ​a​ ​new​ ​Email​ ​Sending​ ​Service.​ ​This​ ​is​ ​a​ ​medium-sized​ ​project​ ​that
will​ ​take​ ​about​ ​one​ ​engineer-month​ ​of​ ​effort​ ​from​ ​our​ ​team,​ ​and​ ​about​ ​two​ ​people-weeks​ ​of
combined​ ​work​ ​from​ ​other​ ​teams.

Goals
[​ ​You​ ​can​ ​skip​ ​Goals,​ ​Product​ ​Requirements​ ​if​ ​there​ ​is​ ​a​ ​product​ ​spec​ ​that​ ​explained​ ​these​ ​in
detail.​ ​ ​]

For​ ​transactional​ ​emails:

1. Allow​ ​marketing​ ​and​ ​design​ ​teams​ ​to​ ​modify​ ​email​ ​templates​ ​without​ ​engineering
involvement.
2. Make​ ​available​ ​email​ ​metrics​ ​on:​ ​open​ ​rates,​ ​click​ ​rates,​ ​and​ ​deliverability.
 3. Make​ ​it​ ​easier​ ​and​ ​faster​ ​to​ ​add​ ​new​ ​emails​ ​to​ ​our​ ​system.

Product​ ​Requirements
● Emails​ ​use​ ​a​ ​template​ ​(with​ ​variable​ ​substitution​ ​and​ ​basic​ ​if-then-else​ ​logic)​ ​which​ ​can
be​ ​edited​ ​directly​ ​by​ ​the​ ​marketing​ ​and​ ​design​ ​teams​ ​in​ ​a​ ​self-serve​ ​fashion.
● Emails​ ​designs​ ​can​ ​be​ ​tested​ ​in​ ​a​ ​staging​ ​environment​ ​that​ ​does​ ​not​ ​affect​ ​production.

Assumptions
● We​ ​must​ ​have​ ​the​ ​capacity​ ​to​ ​sent​ ​up​ ​to​ ​2​ ​emails​ ​to​ ​each​ ​customer​ ​per​ ​day​ ​(though​ ​we’ll
certainly​ ​not​ ​plan​ ​to​ ​send​ ​this​ ​many​ ​emails​ ​on​ ​a​ ​regular​ ​basis).​ ​At​ ​current​ ​growth
projections,​ ​assuming​ ​we​ ​are​ ​planning​ ​for​ ​a​ ​2-year​ ​horizon,​ ​that​ ​means​ ​we​ ​need
capacity​ ​for​ ​up​ ​to​ ​20,000​ ​emails/day.​​ ​[​ ​It’s​ ​a​ ​good​ ​idea​ ​to​ ​provide​ ​SLAs​ ​and​ ​capacity
whenever​ ​possible.​ ​]
● We​ ​only​ ​care​ ​about​ ​HTML​ ​email;​ ​sorry,​ ​Mutt/Pine​ ​users!
● We​ ​will​ ​want​ ​to​ ​segregate​ ​our​ ​email​ ​sending​ ​system​ ​(APIs,​ ​templates)​ ​so​ ​there​ ​is​ ​a
development/test​ ​sandbox​ ​which​ ​is​ ​separate​ ​from​ ​production;​ ​e.g.​ ​a​ ​change​ ​in​ ​a
development/test​ ​email​ ​template​ ​won’t​ ​impact​ ​production.
● It​ ​is​ ​the​ ​responsibility​ ​of​ ​the​ ​marketing​ ​and​ ​design​ ​teams​ ​to​ ​copy​ ​changes​ ​forward​ ​from
development/test​ ​to​ ​production.​ ​Note​ ​that​ ​this​ ​is​ ​potentially​ ​error-prone​ ​process​ ​that​ ​will
be​ ​difficult​ ​to​ ​test;​ ​this​ ​is​ ​likely​ ​something​ ​we’ll​ ​want​ ​to​ ​address​ ​soon.​ [​ ​ ​This​ ​is​ ​a​ ​gross
problem​ ​-​ ​it’s​ ​likely​ ​going​ ​to​ ​be​ ​a​ ​pain​ ​point,​ ​but​ ​I’m​ ​not​ ​sure​ ​we​ ​can​ ​solve​ ​this​ ​right
away,​ ​so​ ​I’d​ ​rather​ ​call​ ​it​ ​out​ ​and​ ​move​ ​on.​ ​]
● We​ ​don’t​ ​want​ ​to​ ​accidentally​ ​trigger​ ​emails​ ​to​ ​real​ ​people​ ​from​ ​our​ ​development/test
domains,​ ​so​ ​should​ ​add​ ​a​ ​filter​ ​that​ ​only​ ​sends​ ​to​ ​@ourcompany.com.
● Because​ ​it​ ​is​ ​possible​ ​that​ ​a​ ​bug​ ​could​ ​cause​ ​us​ ​to​ ​e.g.​ ​send​ ​50​ ​emails​ ​to​ ​the​ ​same
customer​ ​in​ ​5​ ​minutes,​ ​it​ ​is​ ​highly​ ​desired​ ​to​ ​include​ ​a​ ​safeguard​ ​throttle​ ​that​ ​(a)​ ​raises
an​ ​internal​ ​alarm​ ​if​ ​we​ ​send​ ​more​ ​than​ ​3​ ​emails​ ​to​ ​the​ ​same​ ​email​ ​address​ ​in​ ​one​ ​hour,
and​ ​(b)​ ​prevents​ ​sending​ ​more​ ​than​ ​5​ ​emails​ ​to​ ​the​ ​same​ ​email​ ​address​ ​in​ ​one​ ​hour.
[These​ ​numbers​ ​were​ ​pulled​ ​out​ ​of​ ​thin​ ​air.​ ​I​ ​don’t​ ​care​ ​about​ ​the​ ​actual​ ​numbers,​ ​but​ ​I
find​ ​that​ ​people​ ​are​ ​often​ ​more​ ​inclined​ ​to​ ​engage​ ​when​ ​there’s​ ​a​ ​specific​ ​stake​ ​in​ ​the
ground​ ​to​ ​talk​ ​about.]

Out​ ​of​ ​Scope

● This​ ​system​ ​only​ ​applies​ ​to​ ​transactional​ ​emails,​ ​triggered​ ​by​ ​in-app​ ​user​ ​events.​ ​Email
blasts​ ​for​ ​e.g.​ ​marketing​ ​purposes​ ​will​ ​be​ ​handled​ ​by​ ​a​ ​separate​ ​system.
● We​ ​will​ ​not​ ​support​ ​A/B​ ​testing​ ​at​ ​this​ ​stage,​ ​though​ ​it’s​ ​a​ ​likely​ ​future​ ​project.
● Cron-job​ ​triggered​ ​emails​ ​(e.g.​ ​send​ ​a​ ​customer​ ​daily​ ​summary​ ​at​ ​4am)​ m ​ ight​ ​be
handled​ ​by​ ​our​ ​system​ ​in​ ​the​ ​future,​ ​but​ ​this​ ​is​ ​not​ ​supported​ ​for​ ​now.
 ● We​ ​do​ ​not​ ​have​ ​a​ ​multilingual​ ​or​ ​localized​ ​solution​ ​yet;​ ​only​ ​English​ ​emails​ ​using​ ​a​ ​US
date​ ​format​ ​are​ ​supported.​ ​[It​ ​never​ ​hurts​ ​to​ ​think​ ​about​ ​international…]
● While​ ​we​ ​will​ ​certainly​ ​be​ ​able​ ​to​ ​more​ ​quickly​ ​add​ ​new​ ​emails​ ​than​ ​today​ ​(e.g.​ ​“let’s
send​ ​an​ ​thank-you​ ​email​ ​when​ ​a​ ​customer​ ​refers​ ​a​ ​friend”),​ ​this​ ​is​ ​not​ ​just​ ​a​ ​matter​ ​of
filing​ ​an​ ​engineering​ ​ticket​ ​because​ ​it​ ​requires​ ​coordination​ ​between​ ​three​ ​teams:​ ​the
email​ ​content​ ​owner;​ ​engineering;​ ​and​ ​QA.​ ​I’ve​ ​noted​ ​this​ ​as​ ​an​ ​open​ ​question​ ​below.
[Ugh,​ ​this​ ​will​ ​be​ ​a​ ​tricky​ ​process.​ ​For​ ​now,​ ​let’s​ ​just​ ​say​ ​it’s​ ​not​ ​all​ ​on​ ​us,​ ​and​ ​sort
through​ ​the​ ​details​ ​later.]

Open​ ​Questions
[​ ​It’s​ ​totally​ ​OK​ ​to​ ​publish​ ​a​ ​tech​ ​spec​ ​with​ ​holes​ ​in​ ​it,​ ​as​ ​long​ ​as​ ​you​ ​call​ ​out​ ​the​ ​holes.​ ​]

● What’s​ ​our​ ​process​ ​for​ ​adding​ ​new​ ​emails?​ ​Who​ ​owns​ ​streamlining​ ​this?
● How​ ​will​ ​we​ ​manage​ ​email​ ​retry?
● How​ ​should​ ​we​ ​throttle​ ​emails​ ​(so​ ​a​ ​bug​ ​doesn’t​ ​send​ ​a​ ​slew​ ​of​ ​emails​ ​to​ ​one
customer?)
● How​ ​should​ ​we​ ​configure​ ​our​ ​Nginx/Firewall​ ​rules​ ​to​ ​allow​ ​Mailchip​ ​to​ ​make​ ​callbacks​ ​to
our​ ​systems​ ​securely?
● What​ ​happens​ ​if​ ​someone​ ​messes​ ​up​ ​a​ ​Mailchimp​ ​template​ ​such​ ​that​ ​it​ ​e.g.​ ​has​ ​broken
if-then-else​ ​logic​ ​and​ ​cannot​ ​be​ ​sent?​ ​How​ ​is​ ​this​ ​monitored,​ ​and​ ​who​ ​is​ ​responsible​ ​for
checking?​ ​We​ ​are​ ​trusting​ ​marketing​ ​to​ ​own​ ​this,​ ​but​ ​does​ ​there​ ​need​ ​to​ ​be​ ​an
engineering​ ​safe-guard​ ​too?
● Do​ ​we​ ​need​ ​a​ ​manual​ ​“pause”​ ​switch​ ​that​ ​will​ ​turn​ ​off​ ​the​ ​email​ ​worker​ ​(so​ ​jobs​ ​enqueue,
but​ ​emails​ ​aren’t​ ​sent)?​ ​This​ ​might​ ​be​ ​useful​ ​for​ ​maintenance​ ​or​ ​bugs.
● How​ ​do​ ​we​ ​absolutely,​ ​100%​ ​guarantee​ ​that​ ​someone​ ​will​ ​stay​ ​on​ ​top​ ​of​ ​Mailchimp
billing?​ ​E.g.​ ​if​ ​someone​ ​in​ ​finance​ ​leaves​ ​the​ ​company,​ ​we​ ​don’t​ ​want​ ​an​ ​expired​ ​credit
card​ ​notice​ ​to​ ​go​ ​unnoticed.

Approach
[​ ​Here,​ ​I’m​ ​combining​ ​a​ ​presentation​ ​of​ ​the​ ​chosen​ ​approach​ ​alongside​ ​the​ ​alternatives​ ​we
considered.​ ​It’s​ ​would​ ​also​ ​be​ ​fine​ ​to​ ​separate​ ​these​ ​out​ ​into​ ​a​ ​section​ ​“Other​ ​Options
Considered”​ ​]

The​ ​first​ ​question​ ​we​ ​considered​ ​is:​ ​do​ ​we​ ​want​ ​to​ ​send​ ​email​ ​ourselves​ ​(from​ ​our​ ​own​ ​server)
or​ ​use​ ​a​ ​third​ ​party​ ​system?​ ​While​ ​it’s​ ​cheap​ ​to​ ​send​ ​emails​ ​using​ ​our​ ​own​ ​SMTP​ ​server,​ ​we
know​ ​from​ ​experience​ ​that​ ​maintaining​ ​email​ ​deliverability​ ​(IP​ ​address​ ​reputation,​ ​bounces,
unsubscribes)​ ​is​ ​a​ ​huge​ ​operational​ ​burden,​ ​and​ ​we​ ​strongly​ ​lean​ ​towards​ ​using​ ​a​ ​third​ ​party.

Our​ ​ideal​ ​third​ ​party​ ​system​ ​will​ ​handle​ ​both​ ​sending​ ​emails​ ​and​ ​templates​ ​(with​ ​a​ ​template
editor​ ​supporting​ ​variable​ ​substitution​ ​and​ ​if-then-else​ ​logic).​ ​We​ ​would​ ​prefer​ ​an​ ​all-in-one
solution​ ​to​ ​reduce​ ​the​ ​number​ ​of​ ​external​ ​touch​ ​points;​ ​this​ ​means​ ​we​ ​won’t​ ​consider​ ​solutions
like​ ​e.g.​ ​SendWithUs​ ​which​ ​just​ ​manage​ ​templates.

Two​ ​of​ ​the​ ​more​ ​popular​ ​solutions​ ​for​ ​transactional​ ​emails​ ​with​ ​templates​ ​are​ ​SendGrid​ ​and
MailChimp.​ ​Because​ ​our​ ​team​ ​has​ ​had​ ​experience​ ​with​ ​MailChimp​ ​and​ ​they​ ​have​ ​a​ ​solid
operational​ ​reputation,​ ​we​ ​suggest​ ​using​ ​MailChimp.​ ​(Note​ ​that​ ​their​ ​transactional​ ​system​ ​used
to​ ​be​ ​called​ ​Mandrill,​ ​but​ ​this​ ​was​ ​pulled​ ​into​ ​MailChimp​ ​brand​ ​name).

Components
[​ ​This​ ​kind​ ​of​ ​summary​ ​is​ ​optional,​ ​but​ ​sometimes​ ​it​ ​help​ ​the​ ​reader​ ​to​ ​re-state​ ​what​ ​we
proposed​ ​in​ ​terms​ ​of​ ​what​ ​we’ll​ ​be​ ​building.​ ​]

To​ ​recap,​ ​our​ ​plan​ ​is:

● Marketing​ ​creates​ ​email​ ​templates​ ​using​ ​MailChimp
● Clients​ ​will​ ​send​ ​emails​ ​by​ ​enqueuing​ ​them​ ​as​ ​async​ ​RabbitMQ​ ​jobs
○ Specify​ ​an​ ​email​ ​template,​ ​recipient
○ We​ ​will​ ​provide​ ​a​ ​small​ ​library​ ​that​ ​makes​ ​sending​ ​emails​ ​a​ ​simple​ ​API​ ​call
● There​ ​will​ ​be​ ​a​ ​new​ ​database​ ​table​ ​tracking​ ​the​ ​status​ ​of​ ​each​ ​email​ ​addresses​ ​we’ve
sent​ ​to​ ​(deliverable,​ ​#sent,​ ​etc).
● We​ ​will​ ​create​ ​two​ ​new​ ​services
○ Email​ ​Sending​ ​Service​ ​-​ ​consumes​ ​email-send​ ​jobs.
■ Makes​ ​API​ ​calls​ ​to​ ​MailChimp,​ ​using​ ​email​ ​templates
■ Populates​ ​additional​ ​standard​ ​variables​ ​(e.g.​ ​user​ ​first​ ​name)
■ Use​ ​Redis​ ​to​ ​throttle​ ​emails​ ​(​optional​)
○ Email​ ​Callback​ ​Service​ ​-​ ​processes​ ​async​ ​callbacks​ ​from​ ​MailChimp​ ​about​ ​email
delivery

Currently​ ​Supported​ ​Emails

These​ ​are​ ​the​ ​current​ ​emails​ ​we​ ​send.​ ​They​ ​will​ ​be​ ​moved​ ​to​ ​the​ ​new​ ​system.
● Welcome​ ​(on​ ​sign-up)
● Subscription​ ​confirmation
● Unsubscribe​ ​confirmation​ ​(with​ ​CTA​ ​to​ ​re-subscribe)
● Payment​ ​receipt

Send​ ​Queue,​ ​and​ ​Email​ ​Sender​ ​Service

We​ ​propose​ ​that​ ​all​ ​transactional​ ​emails​ ​are​ ​sent​ ​by​ ​putting​ ​them​ ​onto​ ​an​ ​async​ ​job​ ​queue.​ ​The
email​ ​job​ ​is​ ​a​ ​JSON​ ​blob​ ​that​ ​includes​ ​at​ ​least​ ​the​ ​following​ ​fields​ ​(we​ ​may​ ​add​ ​more):
● Recipient​ ​email​ ​address
● Template​ ​identifier
● User-id​ ​(may​ ​be​ ​empty​ ​if​ ​not​ ​associated​ ​with​ ​an​ ​account)
 ● Variables​ ​(key-value​ ​tuples)

We​ ​will​ ​write​ ​a​ ​new​ ​“Transactional​ ​Email​ ​Sending​ ​Service”​ ​that​ ​consumes​ ​from​ ​the​ ​jobs​ ​queue
to​ ​send​ ​emails.​ ​There​ ​will​ ​be​ ​at​ ​least​ ​4​ ​workers​ ​running​ ​on​ ​at​ ​least​ ​2​ ​different​ ​machines.​ ​These
workers​ ​are​ ​responsible​ ​for​ ​the​ ​(simple)​ ​mechanics​ ​of​ ​making​ ​remote​ ​API​ ​calls​ ​to​ ​MailChimp​ ​to
send​ ​emails;​ ​populating​ ​additional​ ​standardized​ ​variables​ ​(see​ ​below),​ ​and​ ​retry​ ​on​ ​failure​ ​(also
see​ ​below).

Because​ ​we​ ​already​ ​use​ ​RabbitMQ,​ ​we​ ​will​ ​also​ ​be​ ​using​ ​RabbitMQ​ ​for​ ​our​ ​email​ ​queues​ ​(with
a​ ​new​ ​queue).​ ​There​ ​will​ ​be​ ​separate​ ​instances​ ​of​ ​the​ ​queue​ ​and​ ​worker​ ​fleets​ ​for​ ​production
and​ ​development/staging.​ ​When​ ​an​ ​email​ ​queue​ ​worker​ ​is​ ​started,​ ​it​ ​will​ ​know​ ​whether​ ​it​ ​is
running​ ​in​ ​production​ ​or​ ​not.

Send​ ​Email​ ​Library

We​ ​will​ ​write​ ​a​ ​small​ ​library​ ​that​ ​any​ ​service​ ​can​ ​use​ ​to​ ​enqueuing​ ​transactional​ ​emails​ ​in​ ​this
standardized​ ​fashion.​ ​The​ ​API​ ​will​ ​be​ ​something​ ​like:

​ emplate_name:​ ​string,​ ​e.g.​ ​“welcome”

#​ t
#​ c​ ustomer:​ ​customer​ ​object
#​ p ​ arams:​ ​dict,​ ​optional

send_email(template_name,​ ​customer,​ ​params​ ​=​ ​None)

This​ ​method​ ​just​ ​adds​ ​the​ ​email​ ​to​ ​the​ ​send-queue.​ ​It​ ​is​ ​assumed​ ​that​ ​all​ ​internal​ ​applications
will​ ​use​ ​our​ ​standardized​ ​configuration​ ​management.​ ​We​ ​will​ ​add​ ​hooks​ ​into​ ​this​ ​system​ ​so​ ​on
application​ ​start,​ ​email​ ​configurations​ ​(how​ ​to​ ​connect​ ​to​ ​the​ ​right​ ​RabbitMQ​ ​queue)​ ​are
automatically​ ​set​ ​up.

Template​ ​Names
[​ ​I​ ​might​ ​be​ ​rambling​ ​a​ ​bit​ ​here​ ​and​ ​getting​ ​off​ ​into​ ​the​ ​weeds​ ​-​ ​but​ ​you​ ​know​ ​what?​ ​I’m​ ​talking​ ​to
myself​ ​and​ ​figuring​ ​out​ ​an​ ​approach,​ ​and​ ​that’s​ ​largely​ ​the​ ​point​ ​of​ ​the​ ​exercise.​ ​]

We​ ​suggest​ ​that​ ​we​ ​use​ ​ONE​ ​MailChimp​ ​account​ ​for​ ​both​ ​test​ ​and​ ​production​ ​emails.​ ​Each
email​ ​template​ ​must​ ​start​ ​with​ ​either​ ​“test_”​ ​or​ ​“prod_”​ ​to​ ​distinguish​ ​between​ ​the​ ​environment,
e.g.​ ​there​ ​would​ ​be​ ​both​ ​“test_welcome”​ ​and​ ​“prod_welcome”​ ​ ​templates.

When​ ​our​ ​software​ ​triggers​ ​an​ ​email,​ ​it​ ​would​ ​just​ ​specify​ ​the​ ​e.g.​ ​“welcome”​ ​template;​ ​if​ ​it​ ​was
in​ ​a​ ​staging​ ​environment,​ ​the​ ​email​ ​sending​ ​service​ ​will​ ​prepend​ ​“test_”​ ​to​ ​the​ ​template​ ​name.

Note​ ​that​ ​the​ ​marketing​ ​team​ ​owns​ ​creating​ ​and​ ​managing​ ​these​ ​email​ ​templates.
Standard​ ​Variables
We​ ​will​ ​want​ ​to​ ​provide​ ​all​ ​emails​ ​with​ ​a​ ​standard​ ​list​ ​of​ ​variables.​ ​These​ ​will​ ​be​ ​automatically
populated​ ​by​ ​the​ ​email​ ​sending​ ​service;​ ​clients​ ​do​ ​not​ ​need​ ​to​ ​specify​ ​these.​ ​Note​ ​that​ ​we​ ​can’t
expect​ ​to​ ​pull​ ​in​ ​the​ ​universe​ ​of​ ​all​ ​data​ ​about​ ​a​ ​customer​ ​(e.g.​ ​full​ ​click-stream​ ​history)​ ​because
some​ ​data​ ​might​ ​require​ ​complex​ ​queries.​ ​Instead,​ ​we​ ​will​ ​maintain​ ​a​ ​list​ ​of​ ​standard​ ​variables
on​ ​ ​this​ ​wiki​ ​page​.​ ​Initially​ ​our​ ​variables​ ​include:

● First​ ​name
● Date​ ​joined
● Membership​ ​tier
● User​ ​timezone​ ​(as​ ​both​ ​current​ ​UTC​ ​offset​ ​in​ ​hours,​ ​and​ ​name​ ​like​ ​“US/Pacific”)

Client-specified​ ​variables​ ​are​ ​merged​ ​into​ ​set​ ​of​ ​variables​ ​given​ ​by​ ​the​ ​email​ ​service.​ ​In​ ​the
case​ ​where​ ​there​ ​is​ ​a​ ​conflict​ ​(the​ ​client​ ​specifies​ ​a​ ​variable​ ​name​ ​which​ ​is​ ​provided​ ​by​ ​the
service)​ ​we​ ​will​ ​log​ ​an​ ​ERROR​ ​and​ ​override​ ​the​ ​client​ ​variable.

Email​ ​Retry
If​ ​the​ ​email​ ​sending​ ​service​ ​is​ ​unable​ ​to​ ​send​ ​an​ ​email​ ​via​ ​MailChimp​ ​due​ ​to​ ​connectivity
problems​ ​or​ ​because​ ​of​ ​some​ ​other​ ​outage,​ ​we​ ​will​ ​attempt​ ​to​ ​re-send​ ​the​ ​email​ ​using
backing-off​ ​retry​ ​logic.​ ​The​ ​mechanics​ ​are​ ​TBD.​ ​[​ ​It’s​ ​OK​ ​to​ ​not​ ​know​ ​this​ ​yet,​ ​let’s​ ​move​ ​on...​ ​]

Filter​ ​Non-Company​ ​Email​ ​Addresses

When​ ​the​ ​email​ ​sending​ ​service​ ​runs​ ​in​ ​non-production​ ​mode,​ ​we​ ​will​ ​log​ ​but​ ​skip​ ​sending
emails​ ​to​ ​all​ ​addresses​ ​that​ ​are​ ​outside​ ​of​ ​our​ ​company​ ​domain.​ ​We​ ​don’t​ ​want​ ​to​ ​accidentally
send​ ​real​ ​customers​ ​emails​ ​from​ ​staging!

Throttle
Note​ ​that​ ​this​ ​feature​ ​might​ ​not​ ​be​ ​included​ ​in​ ​the​ ​initial​ ​rollout.

It​ ​is​ ​highly​ ​desired​ ​to​ ​detect​ ​sending​ ​a​ ​slew​ ​of​ ​emails​ ​to​ ​the​ ​same​ ​email​ ​address​ ​in​ ​a​ ​short
period​ ​of​ ​time.​ ​We​ ​could​ ​use​ ​Redis​ ​counters​ ​with​ ​a​ ​TTL;​ ​but​ ​the​ ​specifics​ ​are​ ​TBD.​ [​ ​ ​Again,​ ​we
capture​ ​this​ ​as​ ​an​ ​open​ ​question​ ​and​ ​move​ ​on.]

Email​ ​Callback​ ​Service

We​ ​will​ ​a​ ​new,​ ​small,​ ​separate​ ​service​ ​for​ ​receiving​ ​MailChimp​ ​async​ ​callbacks.​ ​These
callbacks​ ​tell​ ​us​ ​whether​ ​an​ ​email​ ​was​ ​sent,​ ​delivered,​ ​opened,​ ​clicked;​ ​or​ ​if​ ​there​ ​is​ ​a
hard-bounced,​ ​soft-bounce,​ ​or​ ​unsubscribe.​ ​This​ ​service​ ​will​ ​write​ ​all​ ​such​ ​events​ ​to​ ​the​ ​log.​ ​In
addition,​ ​we​ ​will​ ​keep​ ​a​ ​new​ ​database​ ​table​ ​(e
​ mail_address_status​)​ ​ ​about​ ​the​ ​status​ ​of​ ​each
unique​ ​email​ ​address​ ​we​ ​have​ ​sent​ ​emails​ ​to,​ ​with​ ​fields​ ​like​ ​(this​ ​isn’t​ ​finalized):

● Email​ ​address​ ​-​ ​case​ ​insensitive​ ​string

● Active​ ​(has​ ​not​ ​hard-bounced,​ ​has​ ​not​ ​unsubscribed​ ​from​ ​all​ ​emails)​ ​-​ ​boolean,​ ​default​ ​‘t’
● Count​ ​sent​ ​-​ ​integer
● Count​ ​delivered​ ​-​ ​integer
● Count​ ​soft-bounce​ ​-​ ​integer
● Count​ ​hard-bounce​ ​-​ ​integer
● Count​ ​opened​ ​-​ ​integer
● Count​ ​clicked​ ​-​ ​integer
● Last​ ​sent​ ​date​ ​-​ ​timestamp

The​ ​email​ ​sending​ ​service​ ​will​ ​also​ ​check​ ​this​ ​table​ ​before​ ​send​ ​emails​ ​(to​ ​skip​ ​non-active
addresses)​ ​and​ ​will​ ​upsert​ ​on​ ​sending.

Schema​ ​Changes
[​ ​In​ ​this​ ​example,​ ​I’m​ ​being​ ​almost​ ​absurdly​ ​terse​ ​because​ ​I’m​ ​assuming​ ​I’m​ ​working​ ​with​ ​a
particular​ ​team​ ​and​ ​process​ ​where​ ​more​ ​details​ ​wouldn’t​ ​be​ ​necessary.​ ​]

We​ ​will​ ​create​ ​a​ ​new​ ​database​ ​table​ ​email_address_status​,​ ​discussed​ ​above.​ ​It​ ​will​ ​be
approximately​ ​the​ ​same​ ​size​ ​as​ ​the​ ​users​ ​table.​ ​The​ ​only​ ​two​ ​users​ ​are​ ​the​ ​email​ ​sending
service​ ​and​ ​the​ ​email​ ​callback​ ​service.

Security​ ​and​ ​Privacy

[​ ​Always​ ​consider​ ​customer​ ​data,​ ​encryption,​ ​PII,​ ​and​ ​potential​ ​vectors​ ​of​ ​attack​ ​or​ ​data
leakage.]

Our​ ​biggest​ ​risk​ ​is​ ​that​ ​we​ ​send​ ​too​ ​many​ ​emails,​ ​which​ ​annoy​ ​or​ ​alienate​ ​customers.​ ​And​ ​of
course,​ ​there​ ​must​ ​be​ ​an​ ​opt-out​ ​option​ ​for​ ​users​ ​so​ ​they​ ​are​ ​in​ ​control​ ​of​ ​how​ ​we​ ​use​ ​their
emails.

In​ ​terms​ ​of​ ​privacy,​ ​we​ ​will​ ​be​ ​sending​ ​some​ ​customer​ ​PII​ ​(email​ ​address,​ ​first​ ​name,​ ​pricing
tier)​ ​to​ ​a​ ​third​ ​party​ ​(Mailchimp)​ ​via​ ​an​ ​encrypted​ ​(HTTPS)​ ​API;​ ​this​ ​shouldn’t​ ​be​ ​big​ ​deal,​ ​but
the​ ​legal​ ​and​ ​devops​ ​teams​ ​should​ ​be​ ​made​ ​aware​ ​of​ ​this.

Our​ ​email​ ​callback​ ​service​ ​will​ ​need​ ​to​ ​receive​ ​external​ ​callbacks​ ​from​ ​mailchimp,​ ​so​ ​we’ll​ ​need
to​ ​be​ ​careful​ ​to​ ​ensure​ ​that​ ​these​ ​external​ ​endpoints​ ​are​ ​not​ ​a​ ​vector​ ​of​ ​attack.​ ​Precise
firewall/Nginx​ ​configuration​ ​TBD.
Test​ ​Plan
For​ ​major​ ​releases​ ​with​ ​a​ ​black-box​ ​(manual)​ ​QA​ ​cycle,​ ​we’d​ ​like​ ​to​ ​ask​ ​the​ ​QA​ ​team​ ​to​ ​add​ ​the
welcome​ ​email​ ​as​ ​part​ ​of​ ​their​ ​test​ ​plan.​ ​Signing​ ​up​ ​to​ ​create​ ​an​ ​account​ ​should​ ​trigger​ ​an
email,​ ​which​ ​the​ ​QA​ ​team​ ​will​ ​verify​ ​arrives​ ​and​ ​looks​ ​visually​ ​correct.

We​ ​would​ ​also​ ​like​ ​to​ ​include​ ​email​ ​integration​ ​tests​ ​to​ ​our​ ​Selenium​ ​automated​ ​tests.​ ​The
golden-path​ ​sign-up​ ​flow​ ​should​ ​include​ ​an​ ​end​ ​check​ ​of​ ​an​ ​email​ ​inbox​ ​to​ ​ensure​ ​a​ ​welcome
email​ ​arrived​ ​within​ ​a​ ​few​ ​seconds​ ​of​ ​sign-up.

Operations

Deployment
We​ ​will​ ​work​ ​with​ ​the​ ​devops​ ​team​ ​to​ ​include​ ​the​ ​email​ ​sending​ ​service​ ​and​ ​email​ ​callback
service​ ​as​ ​new​ ​components​ ​(along​ ​with​ ​config​ ​file​ ​management,​ ​secrets,​ ​etc)​ ​in​ ​our​ ​deployment
system.​ ​We​ ​will​ ​own​ ​adding​ ​the​ ​new​ ​DB​ ​schema​ ​ourselves​ ​(as​ ​a​ ​checked-in​ ​migration),​ ​but​ ​the
DBAs​ ​will​ ​be​ ​keeping​ ​an​ ​eye​ ​on​ ​things​ ​like​ ​e.g.​ ​query​ ​performance​ ​on​ ​prod.

Initially,​ ​our​ ​expected​ ​load​ ​is​ ​quite​ ​low​ ​(we’re​ ​just​ ​taking​ ​small​ ​data​ ​blobs​ ​off​ ​a​ ​queue​ ​and
making​ ​remote​ ​HTTPs​ ​calls),​ ​so​ ​we​ ​suggest​ ​deploying​ ​to​ ​a​ ​fleet​ ​of​ ​2​ ​separate​ ​VMs​ ​(on​ ​separate
physical​ ​services,​ ​of​ ​course)​ ​for​ ​each​ ​service.

RabbitMQ​ ​is​ ​owned​ ​by​ ​the​ ​infrastructure​ ​team,​ ​they​ ​are​ ​onboard​ ​with​ ​adding​ ​and​ ​supporting​ ​the
new​ ​email​ ​queues.

Rollout​ ​Plan
Because​ ​this​ ​is​ ​a​ ​new​ ​system,​ ​we​ ​can​ ​deploy​ ​the​ ​services​ ​and​ ​components​ ​in​ ​advance​ ​of
actually​ ​sending​ ​customer​ ​emails.​ ​The​ ​order​ ​of​ ​operation​ ​is:
● Deploy​ ​full​ ​infrastructure​ ​(but​ ​don’t​ ​send​ ​welcome​ ​emails)
○ Marketing​ ​creates​ ​templates
○ RabbitMQ​ ​setup
○ Database​ ​changes
○ Deploy​ ​email​ ​sending​ ​service
○ Deploy​ ​email​ ​callback​ ​service
● Manually​ ​test​ ​(trigger​ ​emails)
○ Validate​ ​integrations
○ Validate​ ​monitoring​ ​and​ ​metrics
● Deploy​ ​code​ ​that​ ​sends​ ​actual​ ​emails
Rollbacks
[​ ​Always​ ​include​ ​some​ ​kind​ ​of​ ​playbook​ ​for​ ​what​ ​people​ ​should​ ​do​ ​if​ ​something​ ​goes​ ​wrong​ ​with
the​ ​release​ ​-​ ​even​ ​if​ ​it’s​ ​just​ ​one​ ​line.​ ​]

If​ ​the​ ​are​ ​unforeseen​ ​issues,​ ​it​ ​should​ ​be​ ​easy​ ​to​ ​simply​ ​shut​ ​down​ ​the​ ​email​ ​workers.​ ​We​ ​will
be​ ​closely​ ​looking​ ​at​ ​logfile​ ​ERRORs​ ​when​ ​this​ ​goes​ ​live.

Monitoring​ ​and​ ​Logging

[​ ​Never​ ​fly​ ​blind!​ ​Always​ ​have​ ​a​ ​monitoring​ ​plan,​ ​even​ ​if​ ​it’s​ ​just​ ​“dump​ ​to​ ​a​ ​log​ ​file,​ ​and​ ​have
something​ ​that​ ​check​ ​for​ ​ERRORs”​ ​]

In​ ​addition​ ​to​ ​standard​ ​machine-health​ ​metrics​ ​and​ ​monitoring,​ ​we​ ​will​ ​use​ ​logfile​ ​monitoring
(with​ ​Graylog).

Email​ ​sending​ ​service:

● Log​ ​each​ ​email​ ​sent​ ​(email​ ​address,​ ​template,​ ​user​ ​information)
● Log​ ​emails​ ​suppressed​ ​(eg.​ ​known​ ​bounce,​ ​or​ ​in​ ​testing​ ​if​ ​out​ ​of​ ​domain)
● Log​ ​any​ ​remote​ ​errors/exceptions​ ​--​ ​add​ ​monitoring​ ​alert​ ​on​ ​these.​ ​This​ ​should​ ​include
missing​ ​or​ ​broken​ ​templates.
● Add​ ​abnormal​ ​level​ ​monitors
○ Production​ ​triggers:​ ​<​ ​1​ ​email​ ​successfully​ ​sent​ ​per​ ​half​ ​hour,​ ​>​ ​2000​ ​emails​ ​sent
per​ ​half​ ​hour​ ​(we’ll​ ​update​ ​this​ ​over​ ​time)

Email​ ​callback​ ​service:

● Log​ ​each​ ​callback​ ​(email,​ ​status​ ​reported​ ​by​ ​mailchimp)
● Trigger​ ​monitoring​ ​on​ ​remotely​ ​reported​ ​errors/exceptions

Metrics
The​ ​project​ ​success​ ​metrics​ ​(customer​ ​engagement​ ​rates​ ​by​ ​cohort,​ ​customer​ ​support​ ​request
rate)​ ​are​ ​provided​ ​by​ ​the​ ​BI​ ​team,​ ​and​ ​we’ll​ ​check​ ​regularly​ ​to​ ​see​ ​how​ ​this​ ​project​ ​is​ ​affecting
these.

We​ ​suggest​ ​that​ ​we​ ​start​ ​tracking​ ​some​ ​additional​ ​email-status​ ​metrics​ ​from​ ​logfile​ ​entries,​ ​with
a​ ​Graylog​ ​dashboard​ ​report:
● Count​ ​emails​ ​sent,​ ​by​ ​template
● Count​ ​bounces​ ​reported​ ​by​ ​Mailchimp

Note​ ​that​ ​MailChimp​ ​has​ ​good​ ​dashboards​ ​for​ ​open​ ​rate,​ ​click​ ​rates,​ ​etc.​ ​For​ ​now,​ ​we​ ​won’t​ ​pull
these​ ​into​ ​our​ ​system​ ​but​ ​will​ ​give​ ​the​ ​business​ ​team​ ​access​ ​to​ ​log​ ​into​ ​the​ ​MailChimp​ ​account.
Long-term​ ​Support

The​ ​platform​ ​team​ ​owns​ ​the​ ​email​ ​sending​ ​service​ ​and​ ​email​ ​callback​ ​service​ ​going​ ​forward;
infrastructure​ ​team​ ​owns​ ​the​ ​RabbitMQ.​ ​ ​Note​ ​that​ ​while​ ​we’re​ ​using​ ​Mailchimp​ ​for​ ​now,​ ​our
system​ ​is​ ​generic​ ​enough​ ​that​ ​there’s​ ​not​ ​all​ ​that​ ​much​ ​vendor​ ​lock-in.​ ​We​ ​expect​ ​to​ ​spend​ ​on
the​ ​order​ ​of​ ​$2500/month​ ​sending​ ​emails.​ ​[​ ​N.b.​ ​this​ ​number​ ​is​ ​totally​ ​made​ ​up​ ​].

As​ ​noted​ ​in​ ​the​ ​“Out​ ​of​ ​Scope”​ ​section​ ​in​ ​the​ ​beginning,​ ​adding​ ​new​ ​email​ ​templates​ ​is​ ​not​ ​just
an​ ​engineering​ ​project,​ ​but​ ​a​ ​process​ ​involving​ ​several​ ​teams.​ [​ ​ ​It’s​ ​OK​ ​to​ ​repeat​ ​yourself​ ​to
CYA​ ​].

Timeline​ ​and​ ​Components

● MARKETING
○ Set​ ​up​ ​MailChimp​ ​account,​ ​add​ ​company​ ​credit​ ​card,​ ​create​ ​initial​ ​email
templates​ ​for​ ​development​ ​and​ ​production​ ​(one​ ​person,​ ​3​ ​days)
● ENGINEERING​ ​(platform)
○ Write​ ​library​ ​for​ ​enqueuing​ ​emails​ ​(one​ ​person,​ ​3​ ​days)
○ Write​ ​new​ ​Email​ ​Sending​ ​Service,​ ​integrate​ ​with​ ​MailChimp​ ​(one​ ​person,​ ​5​ ​days)
○ Write​ ​new​ ​Email​ ​Callback​ ​Service​ ​(one​ ​person,​ ​4​ ​days)
○ Work​ ​with​ ​DevOps​ ​team​ ​to​ ​configure,​ ​deploy,​ ​troubleshoot​ ​services​ ​(one​ ​person,
2​ ​days)
○ Set​ ​up​ ​monitoring​ ​alerts​ ​and​ ​metrics​ ​dashboards​ ​(one​ ​person,​ ​1​ ​day)
● ENGINEERING​ ​(infrastructure)
○ Add​ ​and​ ​support​ ​new​ ​email​ ​queues​ ​for​ ​RabbitMQ​ ​(one​ ​person,​ ​1​ ​day)
● QA
○ Black-box​ ​plan​ ​for​ ​testing​ ​emails​ ​(1​ ​day)
○ Add​ ​welcome​ ​email​ ​check​ ​to​ ​golden-path​ ​automated​ ​Selenium​ ​tests​ ​(2​ ​days)
○ Manually​ ​test​ ​initial​ ​rollout​ ​(1​ ​day)
● DEVOPS
○ Support​ ​setting​ ​up​ ​new​ ​services​ ​(2​ ​days)