Storage on AWS

Raul Hugo, [email protected]

Nov 2019
1 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Agenda

• Introduction
• Storage Primer
• Block Storage
• Shared File Systems
• Object Store
• On-Premises Storage Integration

2 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Introduction: Why choose AWS for storage

Compelling Speed,
Economics Easy to Use Reduce risk Agility, Scale
Pay as you go Self service Durable and Secure Reduce time to
administration market
Avoid risks of
No risky capacity SDKs for simple physical media Focus on your
planning integration handling business, not your
No need to infrastructure
No Commitment
provision Global Scale
for redundancy
or overhead

3 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 0
Storage Primer
4 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Block vs File vs Object
Block Storage
Raw Storage
Data organized as an array of unrelated blocks
Host File System places data on disk
Ex: Hard Disks, Storage Area Network (SAN) Storage Arrays

File Storage
Unrelated data blocks managed by a file (serving) system
Native file system places data on disk
Ex: Network Attached Storage (NAS) Appliances, Windows File Servers

Object Storage
Stores Virtual containers that encapsulate the data, data attributes, metadata and Object IDs
API Access to data
Metadata Driven, Policy-based, etc.
Ex: Ceph, OpenStack Swift

5 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Storage - Characteristics
Some of the ways we look at storage
Durability Availability Security Cost Scalability Performance Integration
Measure of Measure of Security Amount per Upward Performance Ability to
expected data expected measures for storage unit, flexibility, metrics interact via API
loss downtime at-rest and in- e.g. $ / GB storage size, (bandwidth or with other
transit data number of services
users

6 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Understanding Durability

Standard IA Glacier

Two copies on one site Copies on two sites copies in three AZ

designed for designed for designed for

99.99% 99.999% 99.999999999%
durability durability durability

7 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Availability vs Durability

% Availability Durability
99.999 5 minutes 15 seconds 1 in 100,000
99.9999 31 seconds 1 in 1,000,000
99.99999 3 seconds 1 in 10,000,000
99.999999999 300 uSeconds 1 in 100,000,000,000

8 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 More choice for more applications
Block storage File storage
General Purpose SSD EFS Standard
Provisioned IOPS SSD EFS Infrequent Access
Amazon AWS Storage
Throughput-Optimized HDD EFS Gateway Family FSx for Windows
Amazon EC2
Cold HDD FSx for Lustre
Amazon FSx
for Windows
Amazon EBS
File Server
Amazon FSx Object storage
for Lustre
Backup S3 Standard
S3 Standard-IA
S3 One Zone-IA
S3 Intelligent-Tiering
Amazon S3
S3 Glacier
S3 Glacier Deep Archive

9 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 1
Block Storage
10 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 What is Amazon EBS?

EC2
instance
• Block storage as a service
• Create, attach volumes through an API
• Service accessed over the network

EBS
volume

11 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 AWS EBS Features
Durable Secure Scalable
Designed for 99.999 Identity and Capacity when
reliability Access Policies you need it
Redundant storage Encryption Easily scale up
across multiple devices and down
within an AZ

Performance Backup
Low-latency SSD Point-in-time Snapshots
Consistent I/O Performance Copy snapshots across AZ
Stripe multiple volumes for and Regions
higher I/O performance

12 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon EBS
Network attached block device
• Independent data lifecycle
• Multiple volumes per EC2 instance
• Only one EC2 instance at a time per volume
• Can be detached from an instance and attached to a different one

Raw block devices

• Unformatted block devices
• Ideal for databases, filesystems

Multiple Drive Types

• SSD (gp2 & io1) and HDD (st1 & sc1)

13 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon EBS volume types

SSD HDD

gp2 io1 st1 sc1

General Purpose Provisioned IOPS Throughput Optimized HDD Cold HDD
SSD SSD

14 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon EBS use cases

SSD HDD

Relational Databases NoSQL Databases Big Data , Analytics File / Media

MySQL, SQL Server, Cassandra, MongoDB, Kafka, Splunk, CIFS/NFS,
PostgreSQL, SAP, CouchDB Hadoop, Data Transcoding,
Oracle Warehousing Encoding, Rendering

15 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon EBS volume types: General Purpose SSD

Baseline: 100 to 16,000 IOPS; 3 IOPS per GiB

Burst: 3,000 IOPS (for volumes up to 1,000 GiB)

Throughput: Up to 250 MiB/s

gp2 Latency: Single-digit ms

General Purpose SSD
Capacity: 1 GiB to 16 TiB
Great for boot volumes, low-latency applications, and bursty databases

16 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon EBS volume types: Provisioned IOPS

Baseline: 100–64,000 IOPS

Throughput: Up to 1,000 MiB/s

Latency: Single-digit ms

Capacity: 4 GiB to 16 TiB

io1
Provisioned IOPS Ideal for critical applications and databases with sustained IOPS

17 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon EBS volume types: Throughput
Provisioned
Baseline: 40 MiB/s per TiB up to 500 MiB/s

Burst: 250 MiB/s per TiB up to 500 MiB/s

Capacity: 500 GiB to 16 TiB

Ideal for large-block, high-throughput sequential workloads

st1
Throughput Optimized HDD

18 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon EBS volume types: Cold HDD

Baseline: 12 MiB/s per TiB up to 192 MiB/s

Burst: 80 MiB/s per TiB up to 250 MiB/s

Capacity: 500 GiB to 16 TiB

Ideal for sequential throughput workloads, such as logging and backup

sc1
Cold HDD

19 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 EBS Snapshots
• Point-in-time snapshots of volume blocks
• Stored in Amazon S3 and accessed via EBS APIs
• Key Features:
• Immediate access to EBS volume data
• Resizing EBS volumes
• Sharing EBS Snapshots
• Copying EBS Snapshots across AWS regions

21 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 EBS Snapshots – Delta Block Changes

AWS Cloud
EC2 Availability Zone Amazon S3
Create Snapshot
EBS EBS EBS EBS EBS EBS
EBS Snapshot

Clone From EBS Snapshot

Snapshot EBS Snapshot

EC2 EC2 EC2

Internet
22 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 How Do EBS Snapshots Work?
Time

X
Snapshot 1 Snapshot 2 Snapshot 3

S3
EBS Volume
Block 11
Chunk
Block 22
Chunk
Block 33
Chunk
Block 44
Chunk
23 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 What is Amazon EC2 instance store?

EC2 instances
• Local to instance
• Non-persistent data store
• Available on certain EC2
families
Instance Store – Ex: d2, i3, r5d, m5d
• Data is not replicated (by
or
default)
• No snapshot support
Physical Host • SSD or HDD

24 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 2
Shared file system
25 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Elastic File System (EFS)

• Fully managed file system for EC2 instances

• Provides standard file system semantics
• Works with standard operating system APIs
• Sharable across thousands of instances
• Elastically grows to petabyte scale
• Delivers performance for a wide variety of workloads
• Highly available and durable
• NFS v4–based
• Accessible from on-premise servers

26 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 1 Amazon EFS is Simple

Fully managed
- No hardware, network, file layer
- Create a scalable file system in seconds!
Seamless integration with existing tools and apps
- NFS v4.1—widespread, open
- Standard file system access semantics
- Works with standard OS file system APIs
Simple pricing = simple forecasting

27 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 2 Amazon EFS is Elastic

• File systems grow and shrink

automatically as you add and remove
files
• No need to provision storage capacity or
performance
• You pay only for the storage space you
use, with no minimum fee

28 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 3 Amazon EFS is Scalable

• File systems can grow to petabyte scale

• Throughput and IOPS scale
automatically as file systems grow
• Consistent low latencies regardless of
file system size
• Support for thousands of concurrent
NFS connections

29 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Highly Durable and Highly Available

• Designed to sustain AZ offline

conditions
• Resources aggregated across multiple
AZ’s
• Superior to traditional NAS availability
models
• Appropriate for Production / Tier 0
applications

30 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Example use cases

Big Data Analytics

Media Workflow Processing
Web Serving
Content Management
Home Directories
31 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 EFS – Mounting
EFS

EC2 EC2 EC2 EC2 EC2 EC2

EFS DNS Name

availability-zone.file-system-id.efs.aws-region.amazonaws.com

Mount on machine
sudo mount -t nfs4 mount-target-DNS:/ ~/efs-mount-point

32 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 EFS Lifecycle Management

• EFS offers both Standard and Infrequent Access (IA)

storage classes
• With Lifecycle Management enabled, EFS automatically
moves files not accessed for 30 days from the Standard
storage class to the EFS IA storage class

34 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 FSx for Windows

Fully managed Windows Integrated with

file systems … AWS
… built on Windows
Server
35 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Native Windows compatibility and features

SMB AD
NTFS Native SMB Integrates with DFS
Microsoft AD Namespaces
2.0 to 3.1.1 and supports and
Native Windows Windows ACLs DFS Replication
compatibility

Windows Server

36 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 3
Object Stores
38 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon S3 (Simple Storage Service)

• Web accessible object store (through API or HTTPS)

• Highly durable (99.999999999% design)
• Limitlessly scalable
• Multiple Tiers to match your workload
• Data Lifecycle Rules
• Static Website Hosting
• Security, Compliance, and Audit capabilities
• Standard Storage Pricing (us-east-1) - $0.023 per GB

39 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Your choice of object storage classes

S3 Glacier
S3 Standard S3 Intelligent-Tiering S3 Standard-IA S3 One Zone-IA S3 Glacier
Deep Archive
Frequent Access Frequency Infrequent

• Active, frequently • Data with changing • Infrequently • Re-creatable, less • Archive data • Archive data
accessed data access patterns accessed data accessed data • Select minutes or • Select 12 or 48
• Milliseconds access • Milliseconds access • Milliseconds access • Milliseconds access hours hours
• > 3 AZ • > 3 AZ • > 3 AZ • 1 AZ • > 3 AZ • > 3 AZ
• $0.0210/GB • $0.0210 to • $0.0125/GB • $0.0100/GB • $0.0040/GB • $0.00099/GB
$0.0125/GB • Retrieval fee per GB • Retrieval fee per GB • Retrieval fee per GB • Retrieval fee per GB
• Monitoring fee per • Min storage duration • Min storage duration • Min storage duration • Min storage duration
Obj. • Min object size • Min object size • Min object size • Min object size
• Min storage duration

40 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
S3 Management Features

41 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
S3 Access Management & Security
• Deep integration with AWS Identity and Access Management (IAM)
• Access Control Lists (ACLs) & S3 bucket policies
• Query String Authentication
• Audit Logs
• S3 supports both server-side & client-side encryption
• S3 Inventory to check the encryption status of S3 objects
• S3 Block Public Access to ensure S3 buckets and objects do not have
public access
• Amazon Macie to discover, classify, and protect sensitive data stored
in Amazon S3

42 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 4
On-Premises Storage
45
Integration
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Many Options for Data Transfer

AWS Amazon Amazon Amazon Kinesis Amazon S3 AWS

Direct Connect Kinesis Kinesis Video Streams Transfer Storage
Firehose Data Streams Acceleration Gateway

AWS AWS AWS AWS AWS AWS

Database Snowball Snowball Edge Snowmobile DataSync Transfer
Migration for SFTP
Service

46 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Storage Gateway hybrid storage solutions
Enables using standard storage protocols to access AWS storage services

Files Amazon S3

Volumes Amazon Glacier

Amazon EBS
Tapes
snapshots

47 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Storage Gateway – Files, volumes, and tapes

File gateway NFS (v3 and v4.1) interface

On-premises file storage backed by Amazon S3 objects

Volume gateway iSCSI block interface

On-premises block storage backed by S3 with EBS snapshots

Tape gateway iSCSI virtual tape library interface

Virtual tape storage in Amazon S3 and Glacier with VTL management

48 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Storage Gateway – Common capabilities

Standard storage protocols integrate with on-premises applications

Local caching for low-latency access to frequently used data

Efficient data transfer with buffering and bandwidth management

Native data storage in AWS

Stateless virtual appliance for resiliency

Integrated with AWS management and security

49 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Tape gateway
Virtual tape storage in Amazon S3 and Glacier with VTL management
Customer Premises

CHANGER
MEDIA
iSCSI HTTPS

Tape Virtual Archived Tapes

Backup

DRIVE
TAPE
Gateway Tapes stored in
Server stored in Amazon Glacier
Amazon S3
Virtual tape storage in S3 and Glacier accessed via tape gateway
Data compressed in-transit and at-rest
Unlimited virtual tape storage, with up to 1PB of tapes active in library
Supports leading backup applications:

50 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 File gateway
On-premises file storage maintained as objects in Amazon S3
Customer Premises

NFS
v3 / v4.1
HTTPS

Application File S3 S3 Glacier

Server Gateway Standard Standard -
Infrequent
Access
• Data stored and retrieved from your S3 buckets
• One-to-one mapping from files-to-objects
• File metadata stored in object metadata
• Bucket access managed by IAM role you own and manage
• Use S3 Lifecycle Policies, versioning, or CRR to manage data

51 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Volume gateway
On-premises volume storage backed by Amazon S3 with EBS snapshots
Customer Premises

iSCSI HTTPS

Application Volume Storage Gateway Amazon EBS

Server Gateway bucket in snapshots
Amazon S3

Block storage in S3 accessed via the volume gateway

Data compressed in-transit and at-rest
Backup on-premises volumes to EBS snapshots
Create on-premises volumes from EBS snapshots
Up to 1PB of total volume storage per gateway
52 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Hybrid storage use cases with Storage Gateway

Enabling cloud workloads

Move data to AWS storage for Big Data, cloud bursting, or
migration

Backup, archive, and disaster recovery

Cost effective storage in AWS with local or cloud restore

Tiered cloud storage

Easily add AWS storage to your on-premises environment

53 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 EW

AWS DataSync
N

Online transfer service that simplifies, automates, and

accelerates moving data between on-premises storage and AWS

AWS

Fast data Easy to use Secure and Cloud Cost-

transfer reliable integrated effective

Combines the speed and reliability of network acceleration

software with the cost-effectiveness of open source tools

54 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 EW

How AWS DataSync works

N

On-Premise Region
AWS storage resources

Amazon S3
bucket
NFS TLS
AWS
Shared AWS DataSync DataSync
file systems agent Amazon EFS
file system

Deploy on- Data transfer Service in AWS Managed from

premises agent over the WAN writes or reads AWS Console or
for fast access to using purpose- data from AWS Command Line
local storage built protocol storage services Interface (CLI)

55 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Snowball & Snowball Edge

• Terabyte scale data transport

• Uses secure appliances
• Faster than Internet for significant data sets
• Import into S3
• HIPAA Compliant

56 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is Snowball?
Terabyte scale data transport
Ruggedized Rain & dust
case resistant
“8.5G Impact”

E-ink shipping
label Tamper-resistant
case & electronics

All data encrypted

end-to-end
80 TB
57
10G network
© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 How fast is Snowball?

• Less than 1 day to transfer 250TB via 5x10G connections with 5 Snowballs,
less than 1 week including shipping
• Number of days to transfer 250TB via the Internet at typical utilizations

Internet Connection Speed

Utilization 1Gbps 500Mbps 300Mbps 150Mbps
25% 95 190 316 632
50% 47 95 158 316
75% 32 63 105 211

58 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Amazon Snowmobile

https://www.youtube.com/watch?v=8vQmTZTq7nw

59 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 Any Questions?

60 © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.