SCAL ABLE GAME DEVELOPMENT WITH AWS

Guide to Scalable Data

Storage for Games on AWS
 2
GUIDE TO SCAL ABLE DATA STOR AGE FOR GAMES ON AWS

Quick jump
Supporting a global audience
of millions of online players 1.0

with request rates that easily

Relational vs. NoSQL databases

MySQL...................................................................................4
reach millions per second Amazon Aurora...................................................................6

means you need the ability Redis......................................................................................7

to accommodate significant MongoDB.............................................................................8

spikes in traffic. Amazon DynamoDB...........................................................9

Other NoSQL options......................................................12

Caching...............................................................................13
We hope this guide gives you the information you need
to become more knowledgeable about different types of
2.0
purpose-built data storage for games that AWS has to
Binary game content with Amazon S3
offer—plus how to optimize performance to give your
players the best experience possible. We even threw in Content delivery and Amazon CloudFront.................17
a few architectural diagrams to help you get started.
Uploading content to Amazon S3................................18

Amazon S3 performance considerations....................22

 3

With modern game applications that scale

horizontally and globally with your players, the
traditional approach of using a single, large relational
database becomes less tenable. It is important
to spend time thinking about your overall game
data architecture—where data lives, how to query
it, and how to efficiently update it. A number of
new databases have become popular that eschew
traditional atomicity, consistency, isolation, and
durability (ACID) concepts in favor of lightweight

1.0
access, distributed storage, and eventual consistency.
These NoSQL databases can be especially beneficial
for games, where data structures tend to be lists
and sets—like friends, levels, and items—as opposed
to complex relational data.

As a general rule, the biggest bottleneck for online

games tends to be database performance. A typical
web-based app has a high number of reads and few
writes—think of reading blogs, watching videos, and
so forth. Games are quite the opposite, with reads
and writes frequently hitting the database due to
The advent of horizontal scaling
constant state changes in the game.
has changed the application tier as
THE BATTLE OF THE CENTURY well as the traditional approach of a There are many database options out there for both

Relational
single large relational database. relational and NoSQL flavors, but the ones used most
frequently for games on AWS are Amazon Aurora,
Amazon ElastiCache for Redis, Amazon DynamoDB,
Amazon RDS for MySQL, and Amazon DocumentDB

vs. NoSQL
(with MongoDB capability).

First, we will cover MySQL because it is both popular

and applicable to gaming. Combinations such

databases
as MySQL and Redis, or MySQL and DynamoDB,
are especially successful on AWS. All database
alternatives described in this section support atomic
operations, such as increment and decrement, which
are crucial for gaming.
 4
1.1 MYSQL

As an ACID-compliant relational database,

MySQL has the following advantages:

TRANSACTIONS
MySQL provides support for grouping multiple These advantages continue to make MySQL attractive,
changes into a single atomic transaction especially for aspects of gaming like account records,
that must be committed or rolled back. in-app purchases, and similar functionality where
NoSQL stores typically lack multi-step transactions and data consistency are paramount.
transactional functionality. Even gaming companies using NoSQL offerings, such
as Redis and DynamoDB, frequently put transactional
ADVANCED QUERYING data like accounts and purchases in MySQL.

MySQL
MySQL speaks SQL, which provides the
flexibility to perform complex queries that If you are using MySQL on AWS, we recommend
evolve over time. NoSQL databases typically that you use Amazon Relational Database Service
only support key-value access or access by a (RDS) to host MySQL. This can save you valuable
single secondary index, meaning you must deployment and support cycles. Amazon RDS for
make careful data design decisions up front. MySQL automates the time-consuming aspects of
database management, like launching Amazon Elastic
MySQL is the most widely adopted SINGLE SOURCE OF TRUTH Compute Cloud (EC2) instances, configuring MySQL,
MySQL guarantees internal data consistency. attaching Amazon Elastic Block Store (EBS) volumes,
open- source relational database. With Part of what makes many NoSQL solutions setting up replication, running nightly backups, and so
more than 20 years of community-backed faster is distributed storage and eventual on. In addition, Amazon RDS offers advanced features,
consistency. (Eventual consistency means you including synchronous Multi-AZ replication for high
development and support, MySQL is a
could write a key on one node, fetch that key availability, automated master-slave failover, and read
reliable, stable, and secure SQL-based on another node, and have it not appear replicas for increased performance.
database management system. there immediately.)

To get started with Amazon RDS,

EXTENSIVE TOOLS see Getting Started with RDS.
MySQL has been around since the 1990s,
and there are extensive debugging and data
analysis tools available for it. In addition, SQL
is a general-purpose language that is
widely understood.
 5
1.1 MYSQL

The following are some configuration options that we recommend you As your game grows and your write
implement when you create your RDS MySQL DB instances: load increases, resize your RDS DB
instances to scale up.

Resizing an RDS DB instance requires some

downtime, but if you deploy it in Multi-
DB INSTANCE CLASS ALLOCATED STORAGE SLOW SQL QUERIES
AZ mode (as you would for production), it
± Micro instance in development/test ± 5 GB in development/test environments. To find and analyze slow SQL queries in is limited to the time it takes to initiate a
environments. production, ensure you have enabled the failover (typically a few minutes). For more
± 100 GB minimum in production
MySQL slow query log in Amazon RDS as information, see Modifying an Amazon
± Medium or larger instance in environments to enable provisioned input/
shown in the following list. These settings RDS DB Instance in the RDS User Guide. In
production environments. output operations per second (IOPS).
are configured using Amazon RDS DB addition, you can add one or more Amazon
Parameter Groups. Note that there is a minor RDS read replicas to offload reads from your
performance penalty for the slow query log. master RDS instance, leaving more cycles for
database writes.
MULTI-AZ DEPLOYMENT PROVISIONED IOPS ± Set SLOW_QUERY_LOG = 1 to enable. In
Amazon RDS, slow queries are written to
± No in development/test environments. ± Yes for production environments. For instructions on deploying replicas with
the MYSQL.SLOW_LOG table.
Provisioned IOPS guarantees you a Amazon RDS, see Working with
± Yes in production environments to
certain level of disk performance, which ± The value set in LONG_QUERY_TIME Read Replicas.
enable synchronous Multi-AZ replication
is important for large write loads. determines that only queries that take
and failover.
longer than the specified number of seconds
± For more information, see Amazon RDS
For best performance, always launch are included. The default is 10. Consider
Provisioned IOPS Storage in the Amazon
production on an RDS DB instance that is decreasing this value to 5, 3, or even 1.
RDS User Guide.
separate from any of your Amazon RDS
± Make sure to periodically rotate the slow
development/test DB instances.
query log as described in Common DBA
Tasks for MySQL DB Instances in the
Amazon RDS User Guide.

AUTO MINOR VERSION UPGRADE BACKUP SNAPSHOTS

± Yes for hands-off upgrades. ± Schedule Amazon RDS backup snapshots

and upgrades during your low player
count times, such as early in the morning.
If possible, avoid running background
jobs or nightly reports during this window
to prevent a query backlog.
 6
1.2 AMAZON AUROR A

There are several key features that Amazon Aurora brings to a The following are some recommendations for using
gaming workload: Amazon Aurora in your gaming workload:

HIGH PERFORMANCE Use the following DB instance classes: t2.small

Amazon Aurora is designed to provide up to five times the throughput instance in your development/test environments
of standard MySQL running on the same hardware. This performance and r3.large or larger instance in your
is on par with commercial databases but at a significantly lower cost. production environment.
On the largest Amazon Aurora instances, it is possible to provide up
to 500,000 reads and 100,000 writes per second, with 10 millisecond Deploy read replicas in at least one additional

Amazon
latency between read replicas. Availability Zone to provide for failover and read
operation offloading.
DATA DURABILITY

Aurora
In Amazon Aurora, each 10 GB chunk of your database volume is Schedule Amazon RDS backup snapshots and
replicated six ways across three Availability Zones, allowing for the upgrades during low player count times. If possible,
loss of two copies of data without affecting database write availability, avoid running jobs or reports against the database
and three copies without affecting read availability. Backups are done during this window to prevent backlogging.
automatically and continuously to Amazon Simple Store Service
(S3), which is designed for 99.999999999 percent durability with a If your game grows beyond the bounds of a
retention period of up to 35 days. You can restore your database to traditional relational database, like MySQL or
Amazon Aurora is a MySQL- any second (up to the last five minutes) during the retention period. Amazon Aurora, we recommend that you perform a
performance evaluation, including tuning parameters
compatible relational database SCALABILITY and sharding. In addition, you should look at using
engine that combines the speed Amazon Aurora is capable of automatically scaling its storage a NoSQL offering, such as Redis or DynamoDB,
subsystem out to 64 TB of storage. This storage is automatically to offload some workloads from MySQL. In the
and availability of high-end
provisioned for you, so you do not have to provision storage ahead of following sections, we will cover a few popular
commercial databases with the time. As an added benefit, this means you pay only for what you use, NoSQL offerings.
simplicity and cost-effectiveness reducing the costs of scaling. Amazon Aurora also can deploy up to 15
read replicas in any combination of Availability Zones, including cross-
of open-source databases. region where Amazon Aurora is available. This allows for seamless
failover in case of an instance failure.

See how The Pokémon Company

International uses Aurora.
 7
1.3 REDIS

Redis provides foundational data types, including MySQL or DynamoDB, Redis can provide a highly
counters, lists, sets, and hashes, which are accessed scalable solution for game data. Redis plus MySQL
using a high-speed text-based protocol. For more is a popular solution for gaming.
details, see the Redis data types documentation and
introduction to Redis data types and abstractions. Redis uses minimal CPU but a lot of memory. As a
These unique data types make Redis an ideal choice result, it is best suited to high-memory instances,
for leaderboards, game lists, player counts, stats, such as the Amazon EC2 memory optimized instance
inventories, and similar data. Redis keeps its entire family (that is, R3). AWS offers a fully managed Redis
data set in memory, so access is extremely fast. For service, Amazon ElastiCache for Redis. ElastiCache
comparisons with Memcached, check out the for Redis can handle clustering, master-slave
Redis benchmarks. replication, backups, and many other common Redis
maintenance tasks.
There are a few caveats concerning Redis that you

Redis
should be aware of. First, you need a large amount For a deep dive on how to get the most out of
of physical memory because the entire dataset is ElastiCache, see the Performance at Scale with
memory resident (that is, there is no virtual memory Amazon ElastiCache whitepaper.
support). Replication support is also simplistic, and
debugging tools for Redis are limited. Redis is not
suitable as your only data store. However, when used
in conjunction with a disk-backed database, such as
Best described as an atomic data
structure server, Redis has unique
features not found in other databases.

Photo of developer
Benefits of Redis

± In-memory data store

± Flexible data structures

± Simplicity and ease of use

± Replication and persistence

± High availability and scalability

± Extensibility
 8
1.4 MONGODB

MongoDB uses a binary variant of JSON called BSON for Benefits of MongoDB
communication, which makes programming against it
a matter of storing and retrieving JSON structures. This ± Data flexibility
has made MongoDB a popular choice for games and web
applications, since server APIs are usually JSON too. ± High performance

± High availability
MongoDB also offers a number of interesting hybrid features,
including SQL-like syntax that enables you to query data by ± Continuous scale
range and composite conditions. MongoDB supports atomic
operations, such as increment/decrement and add/remove
from list. This is similar to Redis support for these operations.
For examples of atomic operations that MongoDB supports,
see the MongoDB findAndModify documentation.

MongoDB
MongoDB is widely used as a primary data store for games
and is frequently used in conjunction with Redis because
the two complement each other well. Transient game data,
sessions, leaderboards, and counters are kept in Redis. Then
progress is saved to MongoDB at logical points (for example,
at the end of a level or when a new achievement is unlocked).
MongoDB is a document-oriented database, Redis yields high-speed access for latency-sensitive game
which means data is stored in a nested data data, and MongoDB provides simplified persistence.

structure similar to a structure you would use in MongoDB supports native replication and sharding as
a typical programming language. well, although you do have to configure and monitor these
features yourself. For an in-depth look at deploying MongoDB
on AWS, see the MongoDB on AWS whitepaper.

Amazon DocumentDB (with MongoDB compatibility) is

a fully managed document database service that supports
MongoDB workloads. It is designed for high availability and
performance at scale and is highly secure.
 9
1.5 AMAZON DYNAMODB

DynamoDB manages tasks such as synchronous

replication and input/output (I/O) provisioning,
automatic scaling, and managed caching. DynamoDB
uses a provisioned throughput model in which you
specify how many reads and writes you want per
second. The rest is handled for you under the hood.

To set up DynamoDB, see the Getting Started guide.

Amazon
DynamoDB Games frequently use DynamoDB features in the
following ways:

Key-value store for user data, items, friends,

and history
Finally, DynamoDB is a fully managed NoSQL
solution provided by AWS. Range key store for leaderboards, scores, and
date-ordered data

Atomic counters for game status, user counts,

and matchmaking

Like MongoDB and MySQL, DynamoDB can be paired

with a technology such as Redis to handle real-
time sorting and atomic operations. Many game
developers find DynamoDB to be sufficient on its
own, but the point is you still have the flexibility to
add Redis or a caching layer to a DynamoDB-based
The Pokémon Company architecture. Let’s review our reference diagram with
International saved over $80K/ DynamoDB to see how it simplifies the architecture. See how CAPCOM uses DynamoDB.
month when migrating their
280+ million users onto Aurora,
DynamoDB, and ElastiCache.
 10
1.5 AMAZON DYNAMODB

A production-ready game
backend running on AWS
using DynamoDB
Table structure and queries
Elastic Load
Balancing
DynamoDB, like MongoDB, is a loosely structured
Stateful TCP socket HTTP/S HTTP/S TCP
NoSQL data store that allows you to save different sets
of attributes on a per-record basis. You only need to
CloudFront CDN predefine the primary key strategy you are going to use:

PARTITION KEY
The partition key is a single attribute that
DynamoDB uses as input to an internal hash
function. This could be a player name, game ID,
UUID, or similar unique key. Amazon DynamoDB
builds an unordered hash index on this key.
Stateful Game Servers HTTP/JSON Servers HTTP/JSON Servers Stateful Game Servers
Security Group Auto Scaling Group Auto Scaling Group Security Group PARTITION KEY AND SORT KEY
Referred to as a composite primary key, this
type of key is composed of two attributes: the
Queue Mobile push Broadcast partition key and the sort key. DynamoDB uses
Game data async notifications message for
the partition key value as input to an internal
job game
Reads Reads hash function, and all items with the same
partition key are stored together in sorted order
S3 for binary
SQS for job game assets
by sort key value. For example, you could store
CACHE CACHE Writes queues game history as a duplet of [user_id, last_login].
SNS for Push Amazon DynamoDB builds an unordered hash
Messages index on the partition key attribute, and a
Push message
sorted range index on the sort key attribute.
ElastiCache for Redis
Security Group Only the combination of both keys is unique
Job results in this scenario.

Writes Run job For best querying performance, you should maintain
each DynamoDB table at a manageable size. For
example, if you have multiple game modes, it is
DynamoDB
better to have a separate leaderboard table for each
Job Workers Auto
game mode, rather than a single giant table. This also
Scaling Group
gives you the flexibility to scale your leaderboards
separately in the event that one game mode is more
Availability Zone A Availability Zone B
popular than the others.

Single Region (Oregon, Singapore, etc.)

 11
1.5 AMAZON DYNAMODB

Provisioned throughput Amazon DynamoDB Accelerator (DAX)

DynamoDB shards your data behind the scenes to To get the best performance from DynamoDB, make DAX allows you to provision a fully managed, in-
give you the throughput you requested. DynamoDB sure your reads and writes are spread as evenly as memory cache for DynamoDB that speeds up the
uses the concept of read and write units. One read possible across your keys. Using a hexadecimal string, responsiveness of your DynamoDB tables from
capacity unit represents one strongly consistent such as a hash key or checksum, is one easy strategy millisecond-scale latency to microseconds. This
read per second, or two eventually consistent reads to inject randomness. acceleration comes without requiring any major
per second, for an item up to 4 KB in size. One write changes in your game code, which simplifies
capacity unit represents one write per second for deployment into your architecture. All you have
For more details on optimizing DynamoDB
an item up to 1 KB in size. The defaults are five to do is re-initialize your DynamoDB client with
performance, see Best Practices for Designing
read and five write units, which means 20 KB of a new endpoint that points to DAX, and the
and Architecting with DynamoDB in the Amazon
strongly consistent reads per second and 5 KB of rest of the code can remain untouched. DAX
DynamoDB Developer Guide.
writes per second. You can increase your read and/ handles cache invalidation and data population
or write capacity at any time by any amount up without your intervention. This cache can help
to your account limits. You can also decrease the speed responsiveness when running events that
read and/or write capacity by any amount, but this might cause a spike in players, such as a seasonal
cannot exceed more than four decreases in one day. downloadable content (DLC) offering or a new
Scaling can be done using the AWS Management patch release.
Console or Command Line Interface (CLI) by selecting
the table and modifying it appropriately. You can !
also take advantage of DynamoDB Auto Scaling by 3
using the Auto Scaling service to dynamically adjust
Amazon SNS
provisioned throughput capacity on your behalf Amazon
in response to actual traffic patterns. DynamoDB CloudWatch
Auto Scaling works in conjunction with Amazon 2
CloudWatch alarms that monitor the capacity units. 1
4
Auto Scaling scales according to your defined rules.

There is a delay before the new provisioned

throughput is available while data is repartitioned in
6 UpdateTable 5
the background. This does not cause downtime, but
it does mean that the DynamoDB Auto Scaling is best
DynamoDB Application
suited for changes over time, such as the growth of a Table Auto Scaling
game from 1,000 to 10,000 users. It is not designed
to handle hourly user spikes. For this, as with other
databases, you need to use some form of caching
A high-level overview of how
to add resiliency. DynamoDB auto scaling manages
throughput capacity for a table
 12
1.6 OTHER NOSQL OPTIONS

You can use any of these for gaming, and there are
examples of gaming companies using them on AWS
with success. As with choosing a server programming
language, there is no perfect database—you need to
weigh the pros and cons of each one.

Other RIAK

Riak KV is a flexible key-value data

NoSQL
model for web scale profile and session
management, real-time big data, data
cataloging, content management, customer
360 data management, digital messaging,

options
and more.

COUCHBASE

There are a number of other Couchbase Cloud is a fully managed,

automated database that simplifies
NoSQL alternatives, including Riak, database management for deploying,
Couchbase, and Cassandra. managing, and operating Couchbase Server
across multi-cloud environments.

CASSANDRA

Apache Cassandra is an open-source,

distributed, NoSQL database that presents
a partitioned wide-column storage model
with eventually consistent semantics.
 13
1.7 CACHING

Even a short-lived cache of just a few seconds for ELASTICACHE

data such as leaderboards, friend lists, and recent ElastiCache groups servers in a cluster to simplify
activity can greatly offload your database. Adding management. Most ElastiCache operations like
cache servers is cheaper than adding additional configuration, security, and parameter changes are
database servers, so it also lowers your AWS costs. performed at the cache cluster level. Despite the use
of the cluster terminology, ElastiCache nodes do not
MEMCACHED talk to each other or share cache data. ElastiCache
Memcached is a high-speed, memory-based key- deploys the same versions of Memcached and Redis
value store that is the gold standard for caching. that you would download yourself, so existing client
In recent years, Redis has also become extremely libraries written in Ruby, Java, PHP, Python, and so on
popular because it offers advanced data types and are completely compatible with ElastiCache.
features similar performance to Memcached. Both
options perform well on AWS. You can choose to The typical approach to caching is known as lazy

Caching
install Memcached or Redis on EC2 instances yourself, population or cache aside. This means the cache is
or you can use Amazon ElastiCache for Redis, the checked, and if the value is not in cache (a cache
AWS managed caching service. Like Amazon RDS miss), the record is retrieved, stored in cache,
and DynamoDB, ElastiCache completely automates and returned.
the installation, configuration, and management of
Memcached and Redis on AWS. LAZY POPULATION
Lazy population is the most prevalent caching
For gaming, adding a caching layer in For more details on setting up ElastiCache, see strategy because it only populates the cache
when a client requests the data. This way, it avoids
front of your database for frequently What is Amazon ElastiCache for Redis? in the
extraneous writes to the cache in the case of
Amazon ElastiCache User Guide.
used data can alleviate a significant records that are infrequently (or never) accessed
or that change before being read. This pattern is
number of scalability problems.
so ubiquitous that most major web development
frameworks, such as Ruby on Rails, Django, and
Grails, include plugins that wrap this strategy. The
downside to this strategy is that when data changes,
the next client that requests it incurs a cache miss,
resulting in a slower response time because the new
record needs to be queried from the database and
populated into cache.

See how Directive Games uses ElastiCache.

 14
1.7 CACHING

WHEN RECORDS ARE SAVED Amazon ElastiCache scaling

This downside leads us to the second-most
prevalent caching strategy. For data that you ElastiCache simplifies the process of scaling your In general, monitoring hits, misses, and evictions is ElastiCache for Redis version 3 or higher also
know will be accessed frequently, populate cache instances up and down. ElastiCache provides sufficient for most applications. If the ratio of hits to supports sharded clusters. You can create
the cache when records are saved to avoid access to a number of Memcached metrics in misses is too low, you should revisit your application clusters with up to 15 shards, expanding the
unnecessary cache misses. This means client CloudWatch at no additional charge. Based on these code to make sure your cache code is working as overall in-memory data store to more than 3.5 TiB.
response times will be faster and more metrics, you should set CloudWatch alarms to alert expected. As mentioned, evictions should typically Each shard can have up to five read replicas,
uniform. In this case, you simply populate you to cache performance issues. You can configure be zero 100 percent of the time. If evictions are allowing you to handle 20 million reads and 4.5
the cache when you update the record rather these alarms to send emails when the cache memory nonzero, either scale up your ElastiCache nodes to million writes per second.
than when the next client queries it. The is almost full or when cache nodes are taking a long provide more memory capacity or revisit your caching
tradeoff here is that if your data is changing time to respond. We recommend monitoring the strategy to ensure you are only caching what you The sharded model, in conjunction with the
rapidly, it can result in an unnecessarily high following metrics: need to cache. read replicas, improves overall performance and
number of cache writes. And writes to the availability. Data is spread across multiple nodes, and
database can appear slower to users, since CPU UTILIZATION Additionally, you can configure your cache node the read replicas support rapid, automatic failover in
the cache also needs to be updated. The amount of CPU Memcached or Redis is cluster to span multiple Availability Zones to provide the event that a primary node has an issue.
using. Very high CPU could indicate an issue. high availability for your game’s caching layer. This
To choose between these two strategies, ensures that in the event of an Availability Zone To take advantage of the sharded model, you must
you need to know how often your data is EVICTIONS being unavailable, your database is not overwhelmed use a cluster-aware Redis client. The client will treat
changing versus how often it is Number of keys that must be forced out of by a sudden spike in requests. When creating a cache the cluster as a hash table with 16,384 slots spread
being queried. memory due to lack of space. This should be cluster or adding nodes to an existing cluster, you can equally across the shards and will map the incoming
zero. If it is not near zero, you need a larger choose the Availability Zones for the new nodes. You keys to the proper shard. ElastiCache for Redis treats
TIMED REFRESH ElastiCache instance. can either specify the requested number of nodes in the entire cluster as a unit for backup and restore
The final popular caching alternative is a each Availability Zone or select the option to spread purposes. You do not have to think about or manage
timed refresh. This is beneficial for data feeds GETHITS/CACHEHITS AND nodes across zones. backups for the individual shards.
that span multiple different records, such as GETMISSES/CACHEMISSES
leaderboards or friend lists. In this strategy, How frequently your cache has the keys you With Amazon ElastiCache for Redis, you can create
you would have a background job that need. The higher percentage of hits, the more a read replica in another Availability Zone. Upon a
queries the database and refreshes the cache you are offloading your database. failure of the primary node, AWS provisions a new
every few minutes. This decreases the write primary node. In scenarios where the primary node
load on your cache and enables additional CURRCONNECTIONS cannot be provisioned, you can decide which read
caching to happen upstream (for example, at The number of clients currently connected (this replica to promote to be the new primary.
the CDN layer) because pages remain depends on the application).
stable longer.
 15
GUIDE TO SCAL ABLE DATA STOR AGE FOR GAMES ON AWS

“Having the freedom to explore and experiment—we can

do so much more today running on AWS as we execute
against our mission to connect the world through games.
Using AWS allows Zynga to focus on developing great
games, investing in product innovation, and improving
player experiences.”

Dorion Carroll, Chief Information Officer, Zynga

Read the Zynga case study.

 16

Your database is responsible for While you could theoretically store this type of
data in a database, using Amazon S3 has a number
storing user data, including accounts, of advantages, including:
stats, items, purchases, and more.
But for game-related binary data, ± Storing binary data in a database is memory and
Amazon S3 is a better fit. disk intensive, consuming valuable
query resources.

2.0
± Clients can directly download the content from
Amazon S3 provides a simple HTTP-based API to Amazon S3 using a simple GET operation.
PUT (upload) and GET (download) files. With Amazon
± Amazon S3 is designed for 99.999999999 percent
S3, you pay only for the amount of data you store
durability and 99.99 percent availability of objects
and transfer. Using Amazon S3 consists of creating
over a given year.
a bucket to store your data in and making HTTP
requests to and from that bucket. For a walkthrough ± Amazon S3 natively supports features such as
of the process, see Creating a bucket in the Amazon ETag, authentication, and signed URLs.
S3 Getting Started Guide. ± Amazon S3 plugs into the Amazon CloudFront
CDN for distributing content quickly to large
numbers of clients.
Amazon S3 is ideally suited for a variety of gaming
use cases, including the following: With these factors in mind, let’s look at the aspects of
Amazon S3 that are most relevant for gaming.

Binary game
± Content downloads: Game assets, maps,
patches, and betas

± User-generated files: Photos, avatars,

content with
user-created levels, and device backups

± Analytics: Storing metrics, device logs,

and usage patterns

± Cloud saves: Game save data and syncing

Amazon S3
between devices (AWS AppSync would also
be a good choice)
 17
2.1 CONTENT DELIVERY AND AMAZON CLOUDFRONT

Users expect an ongoing stream of new characters, Easy versioning with ETag
levels, and challenges for months—if not years—after
a game’s release. The ability to deliver this content As mentioned earlier, Amazon S3 supports HTTP
quickly and cost-effectively has a big impact on the ETag and the If-None-Match HTTP header, which
profitability of a DLC strategy. are well known to web developers but frequently
overlooked by game developers. These headers
Although the game client itself is typically distributed enable you to send a request for a piece of Amazon

Content
through a given platform’s app store, pushing a new S3 content and include the MD5 checksum of the
version of the game just to make a new level available version you already have. If you already have the
can be onerous and time consuming. Promotional latest version, Amazon S3 responds with an HTTP
or time-limited content, such as Halloween-themed 304 Not Modified status code (or an HTTP 200 status

delivery
assets or a long weekend tournament, are usually code along with the file data) if you need it. For an
easier to manage yourself in a workflow that mirrors overview of this call flow, read about typical usage
the rest of your server infrastructure. of HTTP ETag.

and Amazon
If you are distributing content to a large number
of clients (for example, a game patch, expansion, Using ETag in this manner makes any future use of
or beta), we recommend using CloudFront in front CloudFront more powerful because CloudFront also

CloudFront
of Amazon S3. CloudFront has points of presence supports the Amazon S3 ETag. For more information,
(POP) located throughout the world, which improves see Request and Response Behavior for Amazon S3
download performance. In addition, you can configure Origins in the Amazon CloudFront Developer Guide.
which Region’s CloudFront serves to optimize your
costs. For more information, see How does CloudFront You can also geo target or restrict access to your
lower my costs? in the CloudFront FAQs. content through CloudFront’s Geo Targeting
feature. CloudFront detects the country where your
DLC is a huge aspect of modern games from an Finally, if you anticipate significant CloudFront usage, customers are located and will forward the country
contact our sales team. Amazon offers reduced code to your origin servers, allowing your origin
engagement perspective, and it is becoming a
pricing that is even lower than our on-demand pricing server to determine the type of personalized content
primary revenue stream. for high-usage customers. that will be returned to the customer based on their
geographic location. This content could be anything
from a localized dialog file for an RPG to localized
asset packs for your game.
 18
2.2 UPLOADING CONTENT TO AMAZON S3

There are two strategies for uploading to Amazon S3.

You can either upload directly to Amazon S3 from the
game client or upload by first posting to your REST
API servers and then having your REST servers
upload to Amazon S3. While both methods work,
we recommend uploading directly to Amazon S3 An upload using Amazon S3 POST

Uploading
if possible, since this offloads work from your
REST API tier.

Uploading directly to Amazon S3 is straightforward

content to
Using Amazon S3 POST Using Amazon S3 POST
and can even be accomplished directly from a web
browser. For more information, see Browser-based
uploads using POST (AWS signature version 2) in

Amazon S3
the Amazon S3 Developer Guide. You can even create Amazon S3 Amazon S3

secure URLs for players to upload content (say from

an out-of-game tool) using presigned URLs. File transfer

To protect against corruption, you should consider

calculating an MD5 checksum of the file and File transfer

including it in the Content-MD5 header. This will

Our other gaming use cases for Amazon enable Amazon S3 to automatically verify the file was
Your web server Your web server

S3 revolve around uploading data from not corrupted during upload. For more information,
File transfer
see PutObject in the Amazon S3 API Reference. Web request

the game, be it user-generated content,

analytics, or game saves. User-generated content (UGC) is a great use case
for uploading data to Amazon S3. A typical piece
of UGC has two parts: binary content (for example,
Your customer Your customer
a graphic asset) and its metadata (for example,
name, date, author, and tags). The usual pattern is
to store the binary asset in Amazon S3 and store
the metadata in a database. You can then use the
database as our master index of available UGC that
others can download.
 19
2.2 UPLOADING CONTENT TO AMAZON S3

The following figure shows an example call flow that This simple call flow handles the case where
you can use to upload UGC to Amazon S3. the asset data is stored verbatim in Amazon S3,
which is usually true of user-generated levels or
In this example, you PUT the binary game asset characters. This same pattern works for game
(for example, the avatar or level) to Amazon S3, saves as well—store the game save data in Amazon
which creates a new object in Amazon S3. After you S3, and index it in your database by user_id, date,
receive a success response from Amazon S3, you and any other important metadata. If you need to
make a POST request to our REST API layer with the do additional processing of an Amazon S3 upload
metadata for that asset. The REST API needs to have (for example, generating preview thumbnails),
a service that accepts the Amazon S3 key name make sure to read about asynchronous jobs in the
plus any metadata you want to keep, and then Introduction to Scalable Game Development
it stores the key name and the metadata in the Patterns on AWS ebook. There, you will learn
database. The game’s other REST services can then about adding Amazon SQS to queue jobs to
query the database to find new content, popular handle these types of tasks.
downloads, and so on.

HTTP/JSON Data

1. PUT multi-part file data to S3

2. HTTP 200 OK from S3

S3 Bucket

4. INSERT data + S3 key

3. POST metadata to REST API tier

M

See how Epic Games uses S3.

ELB REST API Master DB
Instances

A simple workflow for transfer of game content

 20
2.2 UPLOADING CONTENT TO AMAZON S3

Analytics and A/B testing After you identify the data, follow these steps to track it:

Collecting data about your game is one of 1. Collect metrics in a local data file on the user’s 3. For each file you upload, put a record somewhere For both analytics and A/B testing, the data
the most important and easiest things you device (for example, mobile, console, or PC). To indicating there is a new file to process. Amazon flow tends to be unidirectional—that is,
can do. Perhaps the trickiest part is deciding make things easier later, we recommend using a S3 event notifications provide an excellent way metrics flow in from users, are processed, and
what to collect. Because Amazon S3 storage CSV format and a unique file name. For example, to support this. To enable notifications, you must then a human makes decisions that impact
is cheap, you should consider keeping track of a given user might have their data tracked in 241- first add a notification configuration identifying future content releases or game features. In
any reasonable metrics you can think of for a game_name-user_idYYYYMMDDHHMMSS.csv or the events you want Amazon S3 to publish, such A/B testing, for example, when you present
user (for example, total hours played, favorite something similar. as a file upload and the destinations where you users with different items, screens, and so
characters or items, and current and highest want Amazon S3 to send the event notifications. forth, you can make a record of the choice
level) if you are not sure what to measure or 2. Periodically persist the data by having the client We recommend Amazon SQS, as you can have they were given along with their subsequent
have a client that is not updated easily. upload the metrics file directly to Amazon S3. a background worker listen to Amazon SQS for actions (such as purchase or cancel). Then,
Alternatively, you can integrate with Amazon new files and process them as they arrive. For periodically upload this data to Amazon S3,
However, if you are able to formulate Kinesis and adopt a loosely coupled architecture, more details, see the Amazon SQS section of the and use Amazon EMR to create reports. In the
questions that you want answered as discussed in the Introduction to Scalable Game Introduction to Scalable Game Development simplest use case, you can generate cleaned
beforehand or if client updates are easy, you Development Patterns on AWS ebook. When you Patterns on AWS ebook. up data from Amazon EMR in CSV format in
can focus on gathering the data that helps go to upload a given data file to Amazon S3, open another Amazon S3 bucket and then load it
you answer those specific questions. a new local file with a new file name. This keeps 4. As part of a background job, process the data into a spreadsheet program.
the upload loop simple. using a framework like Amazon EMR or another
framework that you choose to run on Amazon A proper treatment of analytics and Amazon
EC2. This background process can look at new data EMR is beyond the scope of this ebook.
files that have been uploaded since the last run
and perform aggregation or other operations on For more information, see Data Lakes and
Upload Processing the data. (Note: If you are using Amazon EMR, you Analytics on AWS and the Best Practices
might not need to follow step 3 because Amazon for Amazon EMR whitepaper. To contact us,
EMR has built-in support for streaming new files.)
2. PUT file to S3 please fill out the form at the AWS Game
5a. EMR cluster Tech website.
3. HTTP 200 OK from S3 5. Optionally, feed the data into Amazon Redshift
1. Write local
metrics file
S3 Bucket EMR for additional data warehousing and analytics
OR flexibility. Amazon Redshift is an ANSI SQL-
compliant, columnar data warehouse that you pay
for by the hour. This enables you to perform queries
5b. Non-EMR workflow across large volumes of data, such as sums and
min/max using familiar SQL-compliant tools.
DynamoDB EC2 Workers
OR
Repeat these steps in a loop, uploading and
processing data asynchronously.
4. PUT new S3 key is ready

SQS The figure on this page shows how this pattern works.

A simple pipeline for analytics and A/B testing

 21
2.2 UPLOADING CONTENT TO AMAZON S3

Amazon Athena There are, however, a few things to keep in mind to

optimize performance while using Athena for your
queries, including:
Gleaning insights quickly and cheaply is one
of the best ways developers can improve
on their games. Traditionally this has been
relatively difficult because data normally
must be extracted from game application
servers, stored somewhere, transformed,
and loaded into a database in order to AD-HOC QUERIES PROPER PARTITIONING UNDERSTANDING PRESTO
be queried later. This process can take a
Because Athena is priced at a base of $5 Partitioning data divides tables into Athena uses Presto, an open-source
significant amount of time and compute
per TB of data scanned, this means you parts that keep related entries together. distributed SQL query engine for running
resources, greatly increasing the cost of
incur no charges when there are no queries Partitions act as virtual columns. You define interactive analytic queries against data
running such tasks.
being run. Athena is ideally suited for them at table creation, and they can help sources of all sizes (ranging from gigabytes
running queries on an ad-hoc basis when reduce the amount of data scanned per to petabytes). An under-the-covers
Amazon Athena assists with your analytical
information needs to be gleaned from query, thereby improving performance understanding of Presto can help you
pipeline by providing the means of querying
data quickly without running an extract, and reducing the cost of any particular optimize the various queries that you run
data stored in Amazon S3 using standard
transform, and load (ETL) process first. query. You can restrict the amount of data on Athena.
SQL. Because Athena is serverless, there is
scanned by a query by specifying filters
no infrastructure to provision or manage,
based on the partition. For example, the ORDER BY clause returns
and generally, there is no requirement to
the results of a query in sort order. To do
transform data before applying a schema
For example, in the following query: the sort, Presto must send all rows of data
to start querying. COMPRESSION SELECT count(*) FROM lineitem WHERE to a single worker and then sort them. This
Just like partitioning, proper compression l_gamedate = ‘2019-10-31’, a non- could cause memory pressure on Presto,
of data can help reduce network load and partitioned table would have to scan the which could cause the query to take a long
costs by reducing data size. It is also best to entire table, looking through potentially time to execute. Or, even worse, the query
make sure that the compression algorithm millions of records and gigabytes of could fail. If you are using the ORDER
you choose allows for files to be split, so data. This slows down the query and BY clause to look at the top or bottom N
Athena’s execution engine can increase adds unnecessary costs. A properly values, use a LIMIT clause to reduce the
parallelism for additional performance. partitioned table can help speed queries cost of the sort significantly by pushing the
and significantly reduce cost by cutting the sorting and limiting to individual workers
amount of data queried by Athena. For a (rather than having a single worker
See how Rovio uses Athena.
detailed example, see Top 10 Performance do the sorting).
Tuning Tips for Amazon Athena on the
AWS Big Data Blog.
 22
2.3 AMAZON S3 PERFORMANCE CONSIDER ATIONS

To achieve this scale and get the best performance Let’s assume your bucket is called mygame-ugc and that you store files
out of Amazon S3, there are a few guidelines you based on a sequential database ID:
need to follow. First, as with DynamoDB, make
sure that your Amazon S3 key names are evenly
distributed, as Amazon S3 determines how to
partition data internally based on the first few
characters in the key name.

Amazon S3 In this case, all of these files would likely live in the same internal partition

performance
within Amazon S3 because the keys all start with 107. This limits your
scalability because it results in writes that are sequentially clustered
together. A simple solution is to use a hash function to generate the first
part of the object name in order to randomize the distribution of names.

considerations
One easy strategy is to use an MD5 or SHA1 filename and prefix the Amazon
S3 key with that, as shown in the following code example:

Amazon S3 can scale to tens of thousands of

PUTs and GETs per second.
Here is a variation with a Python SHA1 example:

#!/usr/bin/env python
import hashlib
sha1 = hashlib.sha1(filename).hexdigest()[0:3]
path = sha1 + “-” + filename 39

For more guidelines about maximizing S3

performance, see Best Practices Design Patterns:
Optimizing Amazon S3 Performance in the Amazon
S3 Developer Guide. If you anticipate a particularly
high PUT or GET load, file an AWS Support
Ticket. We can help ensure your buckets are
well architected.
SCAL ABLE GAME DEVELOPMENT WITH AWS