0% found this document useful (0 votes)
17 views100 pages

Module 4 - Storage in AWS

The document provides an introduction to AWS storage solutions, covering various types of storage including block, file, and object storage, along with their characteristics and use cases. It emphasizes the benefits of AWS storage such as durability, availability, security, and cost-effectiveness. Additionally, it outlines the AWS storage portfolio and specific services like Amazon EBS and S3 for different application needs.

Uploaded by

chiakeehong5
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
17 views100 pages

Module 4 - Storage in AWS

The document provides an introduction to AWS storage solutions, covering various types of storage including block, file, and object storage, along with their characteristics and use cases. It emphasizes the benefits of AWS storage such as durability, availability, security, and cost-effectiveness. Additionally, it outlines the AWS storage portfolio and specific services like Amazon EBS and S3 for different application needs.

Uploaded by

chiakeehong5
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 100

Introduction to Storage on AWS

Evangeline Rose Andal


29 Apr 2022

1 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Introduction
• Storage Primer
• Block Storage
• Shared File Systems
• Object Store
• Data Transfer and Edge Processing
• Backup

2 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Introduction: Why choose AWS for storage

Compelling Speed,
Economics Easy to Use Reduce risk Agility, Scale
Pay as you go Self service Durable and Secure Reduce time to
administration market
Avoid risks of
No risky capacity SDKs for simple physical media Focus on your
planning integration handling business, not your
No need to infrastructure
No Commitment
provision Global Scale
for redundancy
or overhead

3 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
0
Storage Primer
4 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Block vs File vs Object
Block Storage
Raw Storage
Data organized as an array of unrelated blocks
Host File System places data on disk
Ex: Hard Disks, Storage Area Network (SAN) Storage Arrays

File Storage
Unrelated data blocks managed by a file (serving) system
Native file system places data on disk
Ex: Network Attached Storage (NAS) Appliances, Windows File Servers, NetApp OnTap

Object Storage
Stores Virtual containers that encapsulate the data, data attributes, metadata and Object IDs
API Access to data
Metadata Driven, Policy-based, etc.
Ex: Ceph, OpenStack Swift

5 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage - Characteristics
Some of the ways we look at storage
Durability Availability Security Cost Scalability Performance Integration
Measure of Measure of Security Amount per Upward Performance Ability to
expected data expected measures for storage unit, flexibility, metrics interact via API
loss downtime at-rest and in- e.g. $ / GB storage size, (bandwidth or with other
transit data number of services
users

6 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Understanding Durability

Standard IA Glacier

Two copies on one site Copies on two sites copies in three AZ

designed for designed for designed for


99.99% 99.999% 99.999999999%
durability durability durability

7 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Availability vs Durability

% Availability Durability
99.999 5 minutes 15 seconds 1 in 100,000
99.9999 31 seconds 1 in 1,000,000
99.99999 3 seconds 1 in 10,000,000
99.999999999 300 uSeconds 1 in 100,000,000,000

8 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
More choice for more applications
Block storage File storage
General Purpose SSD EFS
Provisioned IOPS SSD FSx for OnTap
Amazon AWS Storage
Throughput-Optimized HDD EFS Gateway Family FSx for Windows
Amazon EC2
Cold HDD FSx for Lustre
FSx for OpenZFS
Amazon FSx
Amazon EBS

Amazon FSx Object storage


for Lustre
Backup S3 Standard
S3 Standard-IA
S3 One Zone-IA
S3 Intelligent-Tiering
Amazon S3 S3 Glacier Instant Retrieval
S3 Glacier Flexible Retrieval
S3 Glacier Deep Archive

9 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS delivers broadest storage portfolio in industry

OBJECT BLOCK FILE BACKUP

T T T T
h h h h
e e e e
p p p p
i
Amazon Amazoni i
Amazon FSx for Amazon FSx Amazon FSx for Amazon FSx for AWSi
c Amazon EFS
S3 t EBSct c
t Windows File Server for Lustre NetApp ONTAP OpenZFS c
Backup
t
u u u u
r r r r
e e e e
c
a
c
a DATA TRANSFER AND EDGE PROCESSING
c
a
c
a
n n n n
't 't 't 't
b b b b
e e e e
d d d d
i i i i
s s s s
p p p p
l l l l
a a a a
y AWS Storage y AWS yAWS Transfer AWS AWS Snowball AWS y
e Gateway e DataSync e Family Snowmobile Edge Snowcone e
d d d d
. . . .
10 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1
Block Storage
11 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Block storage portfolio

Instance storage Amazon EBS Snapshots


Temporary block-level storage Easy to use, high performance block Incremental, point-in-time copies
attached to host hardware that is storage service designed for use with of your EBS data that can be used
ideal for storage of information that Amazon Elastic Compute Cloud (EC2) to restore new volumes, expand the
frequently changes or is replicated for both throughput and transaction size of a volume, or move volumes
across multiple instances intensive workloads across Availability Zones

12 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS is designed for a wide range of workloads on EC2
Relational Non-relational/
Enterprise applications databases NoSQL databases Big data analytics File/media

SAP ERP, Oracle MySQL, PostgreSQL, Cassandra, Kafka, Splunk, Hadoop, CIFS/NFS, transcoding,
ERP, Microsoft SQL Server, Oracle DB, MongoDB, CouchDB Data warehousing encoding, rendering
SharePoint, SAP HANA
Microsoft Exchange

LOW LATENCY AND CONSISTENT, HIGH IOPS AND THROUGHPUT


SCALABLE WITHOUT DISRUPTION TO YOUR WORKLOAD
99.999% AVAILABILITY AND AN ANNUAL FAILURE RATE (AFR) OF BETWEEN 0.1%–0.2%

13 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Easy to use, high performance block storage
at virtually any scale

Performance for any workload Easy to use High reliability


Up to 256,000 IOPS, single digit Easily add/remove capacity, 99.999% availability and
millisecond latency, 4,000 MiB/s or change volume types with an annual failure rate of
Throughput Elastic Volumes between 0.1%–0.2%

Virtually unlimited scale Secure Cost-effective


Use a single gigabyte or less, Encrypt all new volumes and Pay as low as $0.015/GB-
or scale up to petabytes of data for a region by default month for highly cost-effective
data with a single setting dollar per gigabyte block
14 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
storage
Six different volume types for optimal use

SSD HDD

gp2 – gp3 io1 – io2 st1 sc1


General Purpose Provisioned IOPS Throughput Cold
SSD SSD Optimized HDD HDD

15 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
General Purpose SSD - gp2

Use cases: Most workloads, relational and non-relational


database workloads, boot volumes, development and test
environments, virtual desktops

Volume size: 1 GiB–16 TiB

Durability: 99.8% - 99.9%

gp2 IOPS/volume*: 16,000


General Purpose SSD
Max throughput/volume*: 250 MiB/s

Pricing**: $0.10 per GB-month of provisioned storage


*Throughput limit is between 128 MiB/s & 250 MiB/s, depending on volume size
**Pricing is for US East (N. Virgina) Region
16 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
General Purpose SSD gp3

Use cases: Latest generation general-purpose SSD-based


EBS volumes. Provision performance independent of storage
capacity, while providing up to 20% lower pricing per GB
than existing gp2 volumes.

Volume size: 1 GiB–16 TiB

Durability: 99.8% - 99.9%


gp3
General Purpose SSD IOPS/volume*: 16,000

Throughput/volume*: 1,000 MiB/s

**Pricing is for US East (N. Virgina) Region


Pricing**: $0.08 per GB-month of provisioned storage,
3,000 IOPS free and $0.005/provisioned IOPS-month over
17 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 3,000
Provisioned IOPS SSD io1

Use cases: Large database workloads, mission-critical


business applications requiring sustained high performance –
Supports Multi-Attach

Volume size: 4 GiB–16 TiB

Durability: 99.8% - 99.9%

io1 Max IOPS/volume*: 64,000


Provisioned IOPS SSD
Max throughput/volume*: 1,000 MiB/s

Pricing**: $0.125 per GB-month of provisioned storage


$0.065 per provisioned IOPS-month
**Pricing is for US East (N. Virgina) Region

18 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Provisioned IOPS SSD io2
Use cases: Latest generation of the Provisioned IOPS SSD
volumes that is designed to provide 100X durability of 99.999%
as well as a 10X higher IOPS to storage ratio of 500 IOPS for
every provisioned GB –at the same price as the previous
generation (io1) – Supports Multi-Attach

Volume size: 4 GiB–16 TiB

Durability: 99.999%

io2 Max IOPS/volume*: 64,000


Provisioned IOPS SSD
Max throughput/volume*: 1,000 MiB/s

Pricing**: $0.125 per GB-month of provisioned storage


$0.065 per provisioned IOPS-month up to 32,000
**Pricing is for US East (N. Virginia) Region IOPS, $0.046 per provisioned IOPS-month up to
64,000 IOPS, $0.032 per provisioned IOPS-month
19 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
greater than 64,000 IOPS
Provisioned IOPS SSD io2 Block Express
Use cases: io2 Block Express offers the highest
performance block storage in the cloud with 4x higher
throughput, IOPS, and capacity than io2 volumes, along with
sub-millisecond latency. Block Express is the next generation
of Amazon EBS storage server architecture purpose-built to
meet the performance and latency requirements of the most
demanding applications.

Volume size: 4 GiB–64 TiB

Durability: 99.999%
io2 Block Express
Provisioned IOPS SSD Max IOPS/volume*: 256,000

Max throughput/volume*: 4,000 MiB/s

**Pricing is for US East (N. Virginia) Region Pricing**: $0.032/provisioned IOPS-month for greater than
64,000 IOPS
20 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Throughput Optimized HDD for frequently
accessed, throughput-intensive workloads
Use cases: st1 is backed by hard disk drives (HDDs) and is
ideal for frequently accessed, throughput-intensive
workloads with large datasets and large I/O sizes, such as
MapReduce, Kafka, log processing, data warehouse, and ETL
workloads.

Volume size: 125 GiB–16 TiB

st1 Durability: 99.8% - 99.9%


Throughput Optimized HDD
Max IOPS/volume*: 500

Max throughput/volume*: 500 MiB/s


**Pricing is for US East (N. Virginia) Region
Pricing**: $0.045 per GB-month of provisioned storage
21 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cold HDD for infrequently accessed workloads

Use cases: sc1 is backed by hard disk drives (HDDs) and


provides the lowest cost per GB of all EBS volume types. It is
ideal for less frequently accessed workloads with large, cold
datasets.

Volume size: 125 GiB–16 TiB

Durability: 99.8% - 99.9%


sc1
Cold HDD Max IOPS/volume*: 250

Max throughput/volume*: 250 MiB/s

Pricing**: $0.015 per GB-month of provisioned storage


**Pricing is for US East (N. Virginia) Region
22 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Fully managed backup with EBS Snapshots
Backup Restore

Low cost Protection Agility


Incremental backups do not duplicate Snapshots are stored Quickly restore volumes across
data and reduce storage costs in Amazon S3 Availability Zones within a region

23 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
EBS Snapshots – Delta Block Changes

AWS Cloud
EC2 Availability Zone Amazon S3
Create Snapshot
EBS EBS EBS EBS EBS EBS
EBS Snapshot

Clone From EBS Snapshot

Snapshot EBS Snapshot

EC2 EC2 EC2

Internet
24 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How Do EBS Snapshots Work?
Time

X
Snapshot 1 Snapshot 2 Snapshot 3

S3
EBS Volume
Block 11
Chunk
Block 22
Chunk
Block 33
Chunk
Block 44
Chunk
25 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is Amazon EC2 instance store?
• Local to instance
EC2 instances
• Non-persistent data store
• Available on several EC2
families
• Data is not replicated (by
Instance Store default)
• No snapshot support
or
• SSD or NVMe
• Pricing is included in instance
Physical Host cost

26 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2
Shared file system
27 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Elastic File System

28 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Elastic File System (Amazon EFS)
Simple, serverless, set-and-forget, elastic file system for AWS compute
Serverless shared Simple and highly Performant and cost-
storage reliable optimized

Serverless and scalable Elastic Performant


No provisioning, scale capacity, Pay only for capacity used
10s of GB/s of throughput and 500,000+ IOPS
connections, and IOPS Performance built-in, scales with capacity

Full AWS compute integration Highly durable and available Four storage classes
Designed for 11 9s of durability
EC2 Instances, containers, and serverless 99.99% availability SLA Automatic lifecycle-based cost optimization
Supports 10,000s of connections

29 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon EFS meets you where you are today and
tomorrow
Migrate
Migrate: lift and shift to AWS cloud without
refactoring application

Optimize: enable cost efficiency

Innovate Operate Optimize Modernize: build micro-services into


application with common data platform

Innovate: improve development efficiency,


build new features, enter new markets

Modernize

30 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Use cases for Amazon EFS

Business Criticality
Home directories Enterprise apps Analytics
DevOps Database backups Machine learning
Application dev. & test Web serving & content Media workflows
mgmt.

Metadata-intensive jobs Scale-out jobs

Low latency and serial I/O High throughput and parallel I/O

31 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Accelerate modernization and innovation
Highly integrated, serverless shared access
VPC VPC

Intra-region
The VPC Peering
pictu
re
can't
be
NFS displ NFS
Clients Amazon
ayeEC2 Amazon EKS Clients
d.
Other AWS
The The Regions
pict T pictu
ure he re
Inter-region can' pi can't
VPC Peering t be ct be
displ ur displ
AWS Fargate
aye Amazone EFS Amazon ECS
ayed
d. ca .
n't AWS VPN
be
di
sp AWS
la Direct On-
ye Connect Premises
AWS Lambda d. Amazon SageMaker Servers

32 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Build and deploy with confidence
Highly available and durable

The The
pictu Amazon EFS Amazon EFS pictu
re Standard One Zone re
can't can't
be be
displ displ
ayed ayed
. .

AWS Backup
Availability Zone
Availability Zone
Availability Zone Availability Zone

99.99% availability SLA 99.9% availability SLA

Designed for 11 9’s durability Designed for 11 9’s* durability

* Data stored in these storage classes may be lost in the event of a disaster or other fault that affects all copies of the data within the Availability Zone (AZ), or in the event of AZ destruction.
33 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance that scales with your application
Amazon EFS can scale up to 10s of GB/s of throughput and 500,000+ IOPS

Performance Modes Throughput Modes


General Purpose Bursting Throughput
Up to 35K read and 7K write IOPS Auto-scale throughput based on storage

Max I/O Provisioned Throughput


Up to 500,000+ IOPS User-defined throughput independent of storage.
Additional charges apply.

34 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Automatic cost optimization
Using EFS storage classes and lifecycle management

$0.043/GB-Month* $0.08/GB-Month*
Effective storage cost Effective storage cost

EFS One Zone EFS Standard


$0.30/GB-month*
$0.043/GB-month*

EFS One Zone-IA EFS Standard-IA


Cost-optimized for less accessed files
Cost-optimized for less accessed files
$0.01333/GB-month* for storage
$0.025/GB-month* for storage
$0.01/GB* for access
$0.01/GB* for access

*Pricing in the US East (N. Virginia) Region. Assumes 80% of the files are infrequently accessed
35 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon FSx

36 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon FSx for Windows File Server

Fully managed file storage Easy migration to AWS


built on Windows Server

37 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Fully managed Windows file storage means you no
longer have to …

Managed hardware Managed software


Plan capacity
Procure and purchase hardware
Set up storage servers
and volumes
Detect and address
hardware failures
Incur high upfront costs
38 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agile, scalable, and automated
Monitoring and automation

Amazon AWS AWS Live storage and


CloudWatch CloudTrail CloudFormatio throughput
n scaling

New New

AWS AWS
Amazon VMware Cloud Amazon Amazon Amazon
DataSync Backup
EC2 on AWS AppStream 2.0 WorkSpaces ECS

Compute instances Data management


39 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Flexible price and performance options
Storage type flexibility (SSD / HDD)
Deployment type flexibility (Single-AZ / Multi-AZ)
Select throughput and storage independently
Choice of in-line snapshots and backups stored in S3
Data deduplication and compression

40 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Fully featured, secure, reliable, and scalable
Accessibility Availability and durability Cost optimization
✓ Full SMB protocol support ✓ High availability – automatic recovery ✓ Storage type flexibility (SSD / HDD)
✓ Windows Server 2008+, Windows 7+, ✓ High durability – automatic replication ✓ Deployment type flexibility (Single-AZ /
Linux, and MacOS Multi-AZ)
✓ Multi-AZ deployment option
✓ EC2, WorkSpaces and AppStream 2.0 ✓ Live scaling of storage capacity
✓ SMB continuous availability (CA)
✓ VMware Cloud on AWS ✓ Data deduplication and compression
✓ Amazon ECS and Amazon EKS containers Performance and scale Data protection
✓ Cross-VPC / Account / Region access ✓ Consistent, sub-millisecond latencies ✓ Snapshots (with end-user file restore)
✓ On-premises access (DirectConnect / ✓ PB-scale storage scalability ✓ Backups
VPN)
✓ Tens of GB/s throughput scalability
Administration
✓ Millions of IOPS scalability Security and compliance
✓ Active directory integration
✓ Select throughput and storage ✓ Encryption at rest and in transit
✓ Managing file shares independently
✓ Kerberos authentication
✓ Monitoring user sessions and open files ✓ Server-side and client-side caching
✓ Access controls via NTFS ACLs, share
✓ Restoring locked files ✓ SMB Multichannel ACLs,
✓ User storage quotas ✓ Performance monitoring via CloudWatch VPC, and IAM
✓ Monitoring actions via AWS CloudTrail ✓ Live scaling of throughput capacity ✓ PCI DSS, ISO, SOC, GDPR, IRAP, and
HIPAA compliances

41 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon FSx for Lustre
Fully managed Lustre file system for high performance workloads

Massively scalable Seamless access to Simple and


performance your data repositories fully managed

Native file Cost-optimized for Secure


system interface compute-intensive workloads and compliant

42 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customers continue to increase the size of their workloads on AWS
across industry verticals and application areas
Industries and example use cases Application areas

Financial services: Life Sciences: Media and Entertainment: Big data Machine
Modeling and analytics Genome analysis Rendering and transcoding analytics learning

For every $1 spent on


high performance
computing, businesses
see $463 in incremental
revenues and $44 in
Automotive: Semiconductor: Oil and gas: High-performance incremental profit1
ECU simulations and Electronic design Seismic data processing computing
object detection automation

43 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FSx for Lustre deployment options

Optional

High and scalable Amazon FSx for Lustre Amazon FSx for Lustre Amazon FSx for Lustre
performance SSD Scratch file system SSD Persistent file system HDD Persistent file system

In all options, we support encryption at-rest and in-transit*

44 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multiple FSx for Lustre throughput options and deployment types
allow customers to optimize storage cost and performance
Storage type Baseline throughput Price per GB-month (in IAD) 1
Persistent storage Scratch • Scratch file
Storage systems are ideal
12 MB/s per TiB - for temporary
HDD (New!) $0.025
storage and
$0.041 (with SSD cache)
shorter-term
40 MB/s per TiB $0.042 - processing of
$0.083 (with SSD cache) data.

125 MB/s per TiB $0.073 - • Data is not


SSD replicated and
250 MB/s per TiB $0.105 - does not persist if
a file server fails.
500 MB/s per TiB $0.170 $0.14 • File systems with
SSD storage can
200 MB/s per TIB $0.070 burst up to 1.3
GB/s per TiB
Sample pricing for - US East (N. Virginia)
1 Prices are subject to change without notice. Pricing varies by AWS Region. For current pricing information, see the

Amazon FSx for Lustre Pricing page on the AWS website.


45 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is Amazon FSx for NetApp ONTAP?

Fully-featured NetApp ONTAP With the simplicity, agility,


and scalability of an AWS
service

46 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
6
Amazon FSx for ONTAP: Benefits
Launch and run fully managed file storage built on NetApp ONTAP

Familiar and fully Fast and cost effective Accessible and Integrated Secure and Compliant
managed

• Fully Managed • Multiple GB/s • Multi-protocol (NFS, • Encrypted at-rest and


throughput, 100K+ SMB, iSCSI) in-transit
• Use AWS and NetApp
tools IOPS, sub-ms latencies • Accessible from Linux, • Integration with
• Data replication • Automatic tiering to Windows, MacOS software for Anti-Virus
low cost, reduce costs • EC2, EKS, Workspaces, and Auditing
(SnapMirror),snapshot
(SnapVault), caching 90% Appstream 2.0, • Active Directory for
(FlexCache), and • Deduplication, VMware Cloud identity-based
cloning (FlexClone) compression, • Concurrent, multi- authentication
capabilities compaction, thin protocol access • ISO, PCI-DSS, SOC
provisioning compliant and HIPPA
eligible

47 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon FSx for NetApp ONTAP: Automatic performance and cost optimization
Intelligent policy-based data movement between tiers

Primary Tier
SSD, Multi-AZ ~20%
Up to 192 TB
Automated Tiering Policies
Optimized for performance
• Snapshot-only (default)
• None
Bi-directional data movement • Auto
based on access patterns • All
(hot/cold)

Capacity Pool Tier


Elastic, Multi-AZ
Unlimited capacity (PB+ file systems) ~80%
Cost-optimized for less accessed files

© 2022, Amazon Web Services, Inc. or its Affiliates.


14
Amazon FSx for ONTAP: Getting Started
Backup, DR Hybrid: Cloud Hybrid: On-prem Cross-region
Migration bursting caching
Replication (in-cloud DR)
Source
volume
SnapVault
SnapMirror
In-AWS
compute (e.g.,
SnapVault EC2) Region A Region B
SnapMirror

Low-latency caching
FlexCache
origin
(source) FlexCache
volume
On-prem ONTAP On-prem ONTAP On-prem FlexCache On-prem
or Global File Cache compute Region A Region B

© 2022, Amazon Web Services, Inc. or its Affiliates. 17


Amazon FSx for NetApp ONTAP: Pricing Dimensions

Provisioned pricing dimensions


• SSD storage ($0.25/GiB-mo provisioned)
• Throughput capacity ($1.20/MBps-mo provisioned)
• [Optional] SSD IOPS ($0.034/IOPS-mo provisioned)

Elastic pricing dimensions


• Capacity pool storage ($0.0438/GiB-mo stored)
• Capacity pool requests $0.0004/1,000 read requests
• Capacity pool requests $.005 per 1,000 write requests
• Backup storage ($0.05/GiB-mo stored)

© 2022, Amazon Web Services, Inc. or its Affiliates.


15
Introducing Amazon FSx for OpenZFS

Shared file storage Built on the AWS


that delivers Graviton family of
high speeds at a low processors
cost, accessible and the popular open-
through NFS source OpenZFS file
system

51 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deliver results faster with high-performance
storage

Latency IOPS Throughput

Average time to return Number of operations Amount of data


What is it?
the first byte of data (I/O) per second transferred per second

What can Latencies of a


Up to Up to
FSx for OpenZFS few hundred
1 million IOPS 12.5 GB/s
deliver? microseconds
Scalable data-intensive
Sequential transaction-heavy applications
When is it workloads like
like code repositories (Git)
important? data and analytics, as well
and content management systems (Wordpress)
as ML
52 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Do more with less and adapt faster to
changing
business needs
SSD storage
$0.09/GB-mo
($0.045/GB-mo w/ compression*)

Throughput
$0.26/MBps-mo
($0.13/MB-ps-mo w/ compression*)

*Pricing assumes average compression savings of ~50% and is an effective price.

53 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
3
Object Stores
54 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon S3

55 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon S3 (Simple Storage Service)

• Web accessible object store (through API or HTTPS)


• Highly durable (99.999999999% design)
• Limitlessly scalable
• Multiple Tiers to match your workload
• Data Lifecycle Rules
• Static Website Hosting
• Security, Compliance, and Audit capabilities
• Standard Storage Pricing (us-east-1) - $0.023 per GB

56 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Your choice of object storage classes

S3 Glacier S3 Glacier S3 Glacier


S3 Standard S3 Intelligent-Tiering S3 Standard-IA S3 One Zone-IA Instant Retrieval
Flexible Retrieval Deep Archive
Frequent Access Frequency Infrequent
• Data with changing • Infrequently accessed • Re-creatable, less • Archive data instant • Archive data • Archive data
• Active, frequently
access patterns data accessed data retrieval • Select minutes or • Select 12 or 48
accessed data
• Milliseconds access • Milliseconds access • Milliseconds • Milliseconds access hours hours
• Milliseconds
• > 3 AZ • > 3 AZ access • > 3 AZ • > 3 AZ • > 3 AZ
access
• $0.0210 to $0.0125/GB • $0.0125/GB • 1 AZ • $0.0040/GB • $0.0036/GB – • $0.00099/GB -
• > 3 AZ
($0.004 to $0.00099/GB • Retrieval fee per GB • $0.0100/GB • Retrieval fee per GB ($4.10/TB) ($1.01/TB)
• $0.0210/GB
Archive) • Min storage duration • Retrieval fee per • Min storage • Retrieval fee per GB • Retrieval fee per GB
• No retrieval fees • Min object size GB duration • Min storage duration • Min storage duration
• Monitoring fee per Obj. • Min storage • Min object size • Min object size • Min object size
• Min storage duration duration
• Min object size • Min object size
57 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
S3 Management Features

58 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
S3 Access Management & Security
• Deep integration with AWS Identity and Access Management (IAM)
• Access Control Lists (ACLs), S3 bucket policies, and S3 Access Points
• Query String Authentication
• Audit Logs
• S3 supports both server-side & client-side encryption
• S3 Block Public Access to ensure S3 buckets and objects do not have
public access
• Amazon Macie to discover, classify, and protect sensitive data stored
in Amazon S3
• Access Analyzer for S3
• Amazon S3 Object Lock
• AWS PrivateLink for S3

59
Amazon GuardDuty for S3
© 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
S3 Replication
Replicate within the same Replicate to a bucket with
AWS Region retention controls (in the same
or different AWS Region)

Replication Time Control


(RTC) 15 minute replication
SLA

Replicate faster to a different


Replicate to a Amazon Simple Storage AWS Region, backed by an
different AWS Region Service
SLA + replication metrics

60 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
4
Data Transfer and
63
Edge Processing
© 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Many Options for Data Transfer

AWS Amazon Amazon Amazon Kinesis Amazon S3 AWS


Direct Connect Kinesis Kinesis Video Streams Transfer Storage
Firehose Data Streams Acceleration Gateway

AWS AWS AWS AWS AWS AWS


Database Snowcone Snowball Edge Snowmobile DataSync Transfer
Migration Family
Service

64 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Storage Gateway

65 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Storage Gateway
On-premises access to virtually unlimited cloud storage

On-premises AWS Cloud

Management
Storage
Application & Monitoring
Server
Services

Storage Gateway
Protocols
Standard

HTTPS
User
Workstation Optimized
uploads Storage
Gateway
Local cache for
low latency access Managed
Service
Backup
Server

Deployment options: VMware, Hyper-V,


KVM, Amazon EC2, Hardware appliance

66 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hybrid storage use cases with Storage Gateway

Enabling cloud workloads


Move data to AWS storage for Big Data, cloud bursting, or
migration

Backup, archive, and disaster recovery


Cost effective storage in AWS with local or cloud restore

Tiered cloud storage


Easily add AWS storage to your on-premises environment

67 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Storage Gateway Family

Amazon S3 File Amazon FSx File Tape Gateway Volume Gateway


Gateway Gateway
Store and access objects Native access to Amazon Drop-in replacement for Block storage on-premises
in Amazon S3 from file- FSx for on-premises physical tape backed by cloud storage
based applications with group file shares and infrastructure backed by with local caching, Amazon
local caching home directories cloud storage with local EBS snapshots, and
caching clones, integrated with AWS
Backup

File-based Access FSx for


Easily switch tape SAN-like with
applications work Windows File Server
backups to AWS cloud recovery
without change from on-premises

68 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Customer’s Amazon S3
AWS SFTP server
Bucket
Connect using NFS v3/v4 or SMB v2/v3 protocols
Files stored as native S3 objects
Metadata is preserved as object user metadata

Object-level encryption with SSE-S3 or SSE-KMS

Fully managed local cache


Read-through, write-back, LRU managed

Notifications through Amazon CloudWatch (e.g., upload complete)

Optimized data transfers


Uploads only sends changes, downloads retrieve file parts needed

File Refresh cache by prefix


Optimizes content distribution workloads

Gateway Performance
Client writes at up to 4 Gbps

69 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
File Gateway
Store and access objects in Amazon S3 from file-based applications with local caching

On premises AWS Cloud

NFS/SMB HTTPS lifecycle


File
Gateway
Amazon S3 Any S3 storage class

• Backup on-premises data to the cloud


Use cases • Shift on-premises storage to cloud-backed file shares
• Low-latency on-premises access to cloud storage

70 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Connect using the iSCSI block protocol
Volumes stored in AWS reducing on-premises SAN footprint

Thin-provisioned (cached) or local (stored) volume types

Fully managed local cache


Read-through, write-back, LRU managed

Configurable encryption with SSE-S3 or SSE-KMS

Volume snapshots stored in Amazon EBS


Volume Gateway
Volume Gateway support for AWS Backup

Volume detach/attach

71 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Volume Gateway
Block storage on-premises backed by cloud storage

On premises AWS Cloud

Application iSCSI Volume HTTPS


Server Gateway
Amazon S3 AWS Backup Amazon EBS

• Backup on-premises data to the cloud


Use cases • Migration of volumes to the cloud
• DR to the cloud

72 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Emulates physical tape library through iSCSI VTL protocol

Fully managed local cache for recent backups


Read-through, write-back, LRU managed

Virtual tapes stored in Amazon S3

Ejected virtual tapes archived read-only in Amazon S3 Glacier or


Amazon S3 Glacier Deep Archive

Retrieve archived tapes to library (3–5 hours)

Configurable encryption SSE-S3 or SSE-KMS


Tape Gateway Compatible with all leading backup software

Performance
Client writes up to 2.3 Gbps, downloads up to 0.6 Gbps

73 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Tape Gateway
Virtual tapes presented to on-premises backup applications

On premises AWS Cloud

Tape Library Tape Archive

HTTPS eject
Backup iSCSI VTL Tape
Server
Gateway
Amazon S3 S3 Glacier / GDA

• On-premises backup to cloud


Use cases • Drop-in replacement for physical tape libraries
• Archive to Glacier or Glacier Deep Archive

74 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon FSx File Gateway

Extend FSx for Windows File Server into your


NEW
datacenter and remote locations

Deployed as a VM, hardware appliance, or in EC2

Local cache of recently used files to improve


performance and reduce latency
Amazon FSx File
Optimized cloud connectivity
Gateway
Centrally managed from a single pane of glass in the
AWS console

75 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon FSx File Gateway architecture
Low-latency access to Amazon FSx File Shares

On-Premises AWS Cloud


VPC

Storage
Gateway
Endpoint
AWS Site-to-Site VPN

AWS Direct Connect


FSx File Gateway
Amazon FSx
for Windows
SMB Clients File systems File Server

• On-premises cache of commonly accessed files backed by Amazon FSx for Windows File Server
• Deploy multiple FSx File Gateways in multiple offices or remote sites
Features
• Up to 64TB cache and up to 500 clients per gateway
• High availability with on-premises cache on VMware

76 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS DataSync

77 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS DataSync
Online transfer service that simplifies, automates, and
accelerates moving data between on-premises storage and AWS

AWS

Fast data Easy to use Secure and Cloud Cost-


transfer reliable integrated effective

Combines the speed and reliability of network acceleration


software with the cost-effectiveness of open source tools

78 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How does AWS DataSync work?
Simplifies, automates, and accelerates data transfer to or from AWS

AWS Region

Management &
Storage Services Monitoring

On-Premises AWS
Amazon KMS
S3
AWS IAM

Amazon AWS
EFS CloudTrail
Shared AWS DataSync agent TLS DataSync
AWS
file system deploys on VMware Service
FSx for CloudWatch
NFS/SMB
Windows AWS
(Object coming File Server Management
soon)
Console

79 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The benefits of AWS DataSync

Fast data Easy to Use Secure and Fully Cost-


transfer Reliable managed effective
• 10x faster than • No in-cloud • Securely transfer • Integrates with • Pay only for
open-source tools infrastructure to any storage AWS data
tier Management and transferred
• Highly optimized • Schedule monitoring
network transfer transfers • End-to-end services • $0.0125/GB or
encryption $12.50/TB
• Up to 10 Gbps per • Throttle
agent (100 TB/day) bandwidth • End-to-end data
verification

80 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Snow Family

81 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Snowball Edge and Snowcone

• Terabyte scale data transport


• Uses secure appliances
• Faster than Internet for significant data sets
• Import into S3
• HIPAA Compliant

82 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is Snowball?
Terabyte scale data transport
Ruggedized Rain & dust
case resistant
“8.5G Impact”

E-ink shipping
label Tamper-resistant
case & electronics

All data encrypted


end-to-end
80 TB
83
10G network
© 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2 Snowball Edge options

Compute optimized Storage optimized


• 42 TB usable S3 compatible storage • 80 TB usable S3-compatible storage
• 52 vCPUs, 208 GiB of memory • sbe1 instances (equivalent to C5)
• Optional NVIDIA Tesla V100 GPU • Up to 40 vCPUs, 80 GiB of memory, 1
• sbe-c and sbe-g instances TB SATA SSD
(equivalent to C5, M5a, G3, P3) • Object storage clustering available

Long-term deployment options: 1- and 3-year discounted pricing


84 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Introducing AWS Snowcone
Small, portable, rugged, and secure edge computing and data transfer device

• Military-grade security
• 4.5 pounds (2.1 kg)
• Portable computing, anywhere
• Withstands harsh environments
• Offline & online data transfer
• 8 TB of storage
• 2 CPU, 4 GB compute
Use cases
Industrial IoT, healthcare IoT, content distribution, content aggregation, data
migration, logistics, autonomous vehicles, and transportation
85 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How fast is Snowball?

• Less than 1 day to transfer 250TB via 5x10G connections with 5 Snowballs,
less than 1 week including shipping
• Number of days to transfer 250TB via the Internet at typical utilizations

Internet Connection Speed


Utilization 1Gbps 500Mbps 300Mbps 150Mbps
25% 95 190 316 632
50% 47 95 158 316
75% 32 63 105 211

86 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Snow Family for data collection & data movement

Snowball Edge
Snowcone Snowmobile
Storage Optimized
Up to 24 TB, Up to petabytes,
Migration size Up to exabytes offline
online and offline offline
Rugged 8.5 G impact cases that are rain and dust 45-foot container,
Form factor
resistant, E Ink label for shipping automation scheduled delivery
Encryption, security staff, GPS
Security 256-bit encryption, tamper detection
tracking, video surveillance, alarms
Storage capacity 8 TB usable 80 TB usable <100 PB
DataSync agent Pre-installed - -
40 vCPU, 80 GB RAM, 1 TB SSD
Compute 2 vCPU, 4 GB RAM usable -
usable
Onboard computing AWS IoT Greengrass functions
options Amazon EC2 AMIs
Wireless Wi-Fi - -

Portable or Mobile use Battery based operation - -

Clustering - Up to 15 nodes -

87 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Visit the AWS Snow Family page for a more detailed comparison »
Amazon Snowmobile

https://www.youtube.com/watch?v=8vQmTZTq7nw

88 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Transfer Family

89 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Managed file transfer – what is it?

Managed file transfer (MFT) provides secure and reliable


transfer of data between systems through a network (e.g.,
the Internet), enabling exchange of data to meet business
critical needs

- Focused on secure and flexible file transfers


- Support 3rd party integrations and connectors
- Provide pre-processing for data filtering and encryption
- May support additional protocols (AS2/HTTPS)

90 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Transfer Family benefits

Fully Seamless Secure & AWS


Managed Migration Compliant Integrated

• Highly available • Import host keys • IP filtering • Store and access


across 3 • Use your own IP • Support for VPC data natively using
Availability Zones addresses and endpoints and AWS services
• Scales on hostnames shared VPC • AWS CloudTrail and
demand • Use existing environments Amazon CloudWatch
• Supports authentication • Data encryption for auditing/logging
thousands of systems options via SSE- • Custom
concurrent users S3 and AWS authentication using
KMS Amazon API
• PCI, HIPAA, Gateway and AWS
SOC3, FIPS Lambda
compliance
91 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Transfer Family Value
Seamlessly migrate without impacting your workflows

AWS Cloud
Users and clients Data available for
use with other
AWS services
Machine Learning

SFTP, FTPS, FTP Data


Lake

AWS Transfer AWS Storage Analytics


Family Services

No change to existing
workflows, applications,
or clients No
Database
infrastructure to
manage

92 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Common use cases

Data lakes and Customer relationship Subscription based data


analytics management applications products
platforms

Digital media content Enterprise resource IoT services used in


aggregation and distribution planning and electronic remote locations for
data interchange for monitoring
supply chain logistics
93 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Get started with AWS Transfer Family

1 Launch a server 2 Select your target 3 Configure your users


endpoint S3 bucket(s) or EFS
file share(s)

Visit console.aws.amazon.com/transfer to get started today!

Pricing Examples:
Server endpoint time: • 10 GB/day costs $2.7K/year
$0.30/protocol/hour
• 100 GB/day costs $3.6K/year
Data uploads and downloads: $0.04/GB
Learn more: aws.amazon.com/aws-transfer-family/pricing/

94 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
5
Backup
95 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Backup

96 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Backup – meeting the challenges
Backup operations unified across AWS services

Complexity Compliance Cost

Simple & Performant Reliable & Secure Cost Effective

Policy- and tag-based Centralized backup activity No added cost


backup solution monitoring and logs for
Automated orchestration
Backup backup
access retention
Automated backup Backup policies managemen
scheduling encryption t

97 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How AWS Backup works
FSx for
Supported AWS Services With AWS Backup, you can:
Lustre EBS EC2 EFS • Create backup plans that orchestrate
T backup operations per policy
h Operators
e
p • Secure backups and manage restore
i processes
c
t Storage
u Gatewa • Control access for managed resources
r y
FSx for RDS Aurora
e
DynamoDB via IAM
Windows c AWS IAM
a
T • Setup notifications/logging via Amazon
n
' h SNS, AWS CloudTrail & AWS
e
t
b AWSp
CloudWatch
T i
e Organizations
h
e
d c
t
• Scale through AWS Organizations
T Backu
p h Backups i u Admin
s p
AWS CloudTrail
i e p r • Protect EC2 Windows/SQL with app
c p l Plan e
consistency
t Amazoni a c
Cross Account
u CloudWatch
c y Backupa
r t e n • Backup to other accounts to protect
e u 't
c
Amazon SNS r
d
. b against multiple threats
a e AWS Backup e
n c d
't a Backup
i Vault
b n s
e 't p
98 ©d2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
b l
i e a
s d y
p i e
l s d
a p .
y l
e a
d y
DR & Ransomware Recovery with AWS Backup

Recovery options:
• Supports 1-to-many, many-
to-many, many-to-1, etc.
AWS Backup
• Recover from same account
locally or from across region
• Recover from cross-account
Vault characteristics: locally or across region
• Backups are highly efficient incremental forever • Recover from RPOs that are
• Backup copies cannot be changed or encrypted hours, days, weeks or months
old
• Manage with vault specific CMK/KMS best practices
• Simple workflow to apply any
• Air-gapped backups using vault access policies forensic analysis
• Prescriptive guidance for vault account access provided
99 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Backup Pricing
US East (N.Virginia)

Amazon Amazon Amazon DynamoDB AWS Storage FSx Amazon


EFS EBS RDS Gateway Aurora

Warm storage ($ per GB / month) $0.05 $0.05 $0.095 $0.10 $0.05 $0.05 $0.021

Cold storage ($ per GB / month) $0.01 na na na na na na

Restore from warm ($ per GB) $0.02 Free Free $0.15 Free Free Free

Restore from cold ($ per GB) $0.03 na na na na na na

*Minimum storage period for cold tier is 90 days


100 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customers who can benefit the most with AWS Backup
Customers who:
• …want to optimize for compliance and want to standardize the way data
protection is performed
• …want to optimize for efficiency to reduce their own script dependencies to
protect AWS resources (EBS, RDS, EC2, etc.)
• …want to improve productivity with re-use of copies for things like test/dev,
QA and CI/CD
• …want to to protect against insider threats and ransomware
• …want centralized control and security for extensive recovery options in the
cloud
• External Website:
• https://aws.amazon.com/backup/resources/
• Blogs, Videos, Documentation, FAQs, articles
101 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Any Questions?

102 © 2022, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

You might also like