AWS Builders Online Series

How to get your data

into the cloud
Naim Mucaj
Senior Solutions Architect
Amazon Web Services

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda

- AWS Services to help you transfer/migrate data

- Common customer use cases

- Demo of Customer Scenarios using AWS Services, AWS DataSync,

and AWS Storage Gateway

- Resources

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How much data is out there?

175 ZB
Zettabytes of data created by
1021
1 zettabyte is
2025, globally 1,000,000,000,000,000,000,000

Sources:
https://www.idc.com/getdoc.jsp?containerId=prUS48247121
https://www.idc.com/getdoc.jsp?containerId=US47397521

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Are you planning or currently
involved in a cloud migration
project?

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Portfolio
Intelligent- Compression
Data services Tiering and dedupe
AWS Amazon EBS AWS Transfer Lifecycle
Replication Metrics reporting
Backup snapshots Family management
Core services

Object Block File

Amazon S3
Amazon Amazon
and Amazon Amazon EBS FSx Family EFS
S3 Glacier

Hybrid/edge storage
AWS AWS Storage AWS AWS
DataSync Gateway
Data movement services Snow Family Outposts

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Streamline data and application migrations
AWS provides a range of data migration services matched to your migration needs

AWS Transfer AWS CloudEndure AWS Database AWS Snow AWS Storage
Family DataSync Migration Migration Service Family Gateway
(AWS DMS)
Regular file transfers Move data between Simplifies, expedites, Migrate data to and Offline transfer of Hybrid cloud
directly into and out of on-premises storage and automates large- from most of the large amounts of storage for on-
Amazon Simple Storage systems and AWS, as scale migrations from widely used data into and out of premises access to
Service (Amazon S3) or well as between AWS physical, virtual, and commercial and AWS regardless of virtually unlimited
Amazon Elastic File storage services cloud-based open-source network connectivity cloud storage
System (Amazon EFS) infrastructure to AWS databases
using SFTP, FTPS, or FTP

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Streamline data and application migrations
AWS provides a range of data migration services matched to your migration needs

AWS Transfer AWS CloudEndure AWS Database AWS Snow AWS Storage
Family DataSync Migration Migration Service Family Gateway
Regular file transfers Simplifies, expedites, Migrate data to and
Move data between Offline transfer of Hybrid cloud
directly into and out and automates large- from most of the
on-premises storage large amounts of storage for on-
of Amazon S3 or scale migrations from widely used
systems and AWS, as data into and out of premises access to
Amazon EFS using physical, virtual, and commercial and open
well as between AWS AWS regardless of virtually unlimited
SFTP, FTPS, or FTP cloud-based source databases
storage services network connectivity cloud storage
infrastructure to AWS

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DataSync
overview

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is AWS DataSync?

Online data transfer service

that simplifies, automates, and accelerates
copying file and object data to and from AWS storage

Fast data Easy to use Secure and Fully Cost-effective

transfer reliable managed

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DataSync use cases

Transfer
Migrate Archive Replicate
data for timely
application data on-premises for data protection
in-cloud
to AWS storage to free and recovery
processing
up capacity

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What can you do with DataSync?
Quickly, easily, and securely transfer your data to and from AWS storage services

On Premises

NFS,
SMB,
Object

Hadoop AWS DataSync

Amazon S3 Amazon S3
(HDFS) (all storage classes) (all storage classes)

Edge
AWS DataSync
Amazon FSx Amazon FSx

AWS
Snowcone

Amazon S3 on
AWS DataSync Amazon EFS Amazon EFS
Outposts

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
On-premises transfers
Transfer data between on-premises storage systems and AWS Storage services

NAS array, Window

Server, etc. NFS, SMB

Amazon S3
(all storage classes)

S3 API

Object Storage
DataSync Agent Amazon FSx
AWS DataSync
(Deploys on VMware, Managed Service
HDFS Hyper-V, KVM)

Amazon EFS

Hadoop Cluster (HDFS)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DataSync: How it works
Transfer data between on-premises storage systems and AWS Storage services

2
NAS array, Window
Server, etc. NFS, SMB

Agents are deployed Amazon S3

2 Locations control to connect to on- (all storage classes)
how AWS DataSync 1 premises storage
connects to storage
S3 API

Object Storage
DataSync Agent Amazon FSx
AWS DataSync
(Deploys on VMware, Managed Service
HDFS Hyper-V, KVM)
3
AWS DataSync managed service
connects to AWS storage and EFS
4
coordinates the transfer
Hadoop Cluster (HDFS) For transfers between on premises
and AWS, internet, AWS Direct
Connect, and AWS Virtual Private
Network (VPN) are Amazon
© 2023, supported
Web Services, Inc. or its affiliates. All rights reserved.
Archive data to the cloud
Move cloud data from expensive on-premises storage systems to durable, secure long-term storage

On premises AWS Cloud

AWS Storage Resources

NAS File NFS/SMB
Storage
Amazon S3
DataSync Agent AWS DataSync
Service

Amazon FSx for

Application Windows File
Server(s) Server

AWS Storage AWS Storage

Gateway Gateway Service

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
FORMULA 1 uses AWS DataSync to replicate
video archive to Amazon S3

Challenge Solution Results

• 400 TB video archive that needs • Using AWS DataSync, FORMULA 1 • AWS DataSync transfers data
to be backed up regularly while copies up to 4 TB per day to quickly and securely to Amazon S3.
maintaining high accessibility Amazon S3.
• Data is stored durably and reliably
• FORMULA 1 uses AWS Storage in Amazon S3, and all data is
Gateway to provide standard file encrypted at rest.
access to data in Amazon S3 with • AWS Storage Gateway gives users
a local cache for low latency on-premises access to the video
access. archive in Amazon S3, providing fast
recovery from failure scenarios.

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“AWS DataSync synced data
from our on-premises file server
to Amazon S3 at an average of 4
TB per day, and did so with
encryption during transfer and
at rest.”
Martynas Juras
Cloud Engineer , FORMULA 1

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The hybrid challenge
You have on-premises applications…

Existing
applications &
data archives

Group file shares

& user directories
AWS
New data
sources

… that want to store data and

access services in the cloud

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway
Providing on-premises access to virtually unlimited cloud storage

Migration Modernization Continuous

reinvention

Backup and archive Shift on-premises Access and share

data to AWS network-attached storage cloud data across
to cloud-backed file shares edge locations

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway architecture
Providing on-premises access to virtually unlimited cloud storage

Remote and Branch Offices AWS Cloud

Storage Management
Services & Monitoring
Application
Server
Standard Protocols AWS Storage Gateway
NFS /
SMB Secure
connection
User iSCSI
Workstation
Optimized
VTL - uploads
Tape AWS Storage
Local cache for Gateway
low latency access Managed Service
Backup
Server
Deployment options: VMware, Hyper-V,
KVM, Amazon EC2, Hardware Appliance

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway: Gateway types
Providing cloud-backed file shares, virtual tapes, and block volumes

Amazon S3 Amazon FSx Tape Volume

File Gateway File Gateway gateway gateway

Block-level backups
Native file access to Native access to Amazon FSx Replace physical tape
of volumes with
Amazon S3 for backups, for Windows File Server for infrastructure leveraging
Amazon EBS Snapshots,
archives, and ingest on-premises group file shares Amazon S3 archive tiers for
AWS Backup integration,
for data lakes and home directories long-term retention
and cloud recovery

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway benefits
Providing on-premises application access to virtually unlimited cloud storage

Standard protocols Local caching Optimized data transfers

No changes to existing apps Low-latency access to Minimized network traffic
frequently used data

Designed to be secure Cost-effective AWS integrated

and compliant Pay-as-you-go pricing Management, monitoring,
FIPS, HIPAA-eligible, PCI, SOC*, and in-cloud workloads
ISO**, and encryption
*SOC (1, 2, 3) **ISO (9001, 27001, 27017, 27018)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo Scenarios – Sophia, Joe, and Ahmad

Sophia has large data repositories stored on NAS Storage presented via NFS
which need to be archived and migrated into AWS.

Joe manages the Surveillance Security Systems and has Video Repository
presented via SMB and has 5 years of data which no longer needs to be
online and can be archived.

Ahmad manages Backups and has run out of on-premises backup storage – and
has realized the SQL backups are taking up a large portion of their expensive
primary storage.

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo Set up 1 – AWS DataSync
On-premises AWS Region

Data Repositories
(NFS)

Amazon S3
AWS DataSync (all storage classes)
Video Surveillance
Agent
Archive
(SMB)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo Set up 2 – AWS Storage Gateway
On-premises AWS Region

SQL Backups SMB

Amazon S3
File Gateway (all storage classes)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo Set up – DataSync and Storage Gateway
On-premises AWS Region

Data Repositories
(NFS)

Amazon S3
AWS DataSync (all storage classes)
Video Surveillance
Agent
Archive
(SMB)

SQL Backups SMB

Amazon S3
File Gateway (all storage classes)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo recap
• AWS DataSync was used to transfer / migrate data from different
sources on premises (NFS and SMB) to Amazon S3

• Easily manage data movement workloads with bandwidth throttling,

migration scheduling, and task filtering.

• Amazon S3 File Gateway was used to provide low latency access to an

SMB Share for on-premises access.

• A seamless way to connect to the cloud (via SMB / NFS) in order to

store backup images as durable objects in Amazon S3 cloud storage.

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Snow Family
overview

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is AWS Snow Family?
Move petabytes of data to and from AWS,
or process data at the edge.

AWS Snowcone AWS Snowcone AWS Snowball

SSD

<1TB 10s PB

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How data gets imported to AWS Snow Family

AWS
Management
Console or
Track Connect to Ship Your data is
Create job shipment AWS Snowcone or device loaded into
AWS Snowball Edge, to AWS your bucket!
transform and/or copy
Job Shipped Delivered data Send Ingest Secure
created to you to you to AWS to AWS device wipe

1–N days
1–3 days 1–2 days (file transfer speed varies) 1–2 days 1–N days

Job provisioned At customer site At sorting facility Job completed

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrations with AWS Snow Devices
and AWS DataSync

Initial transfer with

Snowball devices

On-premises Amazon S3
storage systems

Incremental transfers
using AWS DataSync
AWS DataSync

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 LotteON provides 39 million members with a unified
storefront to 7 Lotte-owned stores.
Challenge
Lotte needed to transform and transfer 140 million
files consisting of product images and media assets.

Lotte used AWS Snowball Edge for secure, and fast

offline data transfer.
Solution
Lotte did not have to provision additional network
” With AWS Snowball, LotteON was quickly and capacity.
reliably able to transfer 140 million product
images and assets by combining 7 individual
web stores within 2 weeks. Snowball helps meet AWS Snowball Edge enabled Lotte to meet its timeline
the tight project launching schedule without to support a consistent shopping experience of 39
worrying about network ” Benefits million Lotte members.

Hyeongkil Na, The data was leveraged by 70+ AWS services, including
Amazon S3, Amazon EKS, Amazon DynamoDB, AWS
Platform Operation Team leader
Lotte e-commerce Lambda, Amazon Elasticsearch, Amazon Redshift,
Amazon EMR, and Amazon Athena.

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DataSync use cases

• SMB/NFS data transfer → Amazon S3 storage classes

• NFS data transfer → Amazon EFS, Amazon FSx for Luster,

Amazon FSx for OpenZFS

• SMB data transfer → Amazon FSx for Windows File Server

• Object Storage → Amazon S3 storage classes

• Hadoop Distributed FS (HDFS) → Amazon S3 storage classes

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway use cases
• Store and retrieve objects in Amazon S3 using file protocols
such as NFS and SMB

• Move backups and archives to the Cloud

• Reduce on-premises storage with cloud-backed file shares

• Provide on-premises applications low-latency access to data

stored in AWS (Amazon S3 and Amazon FSx for Windows)

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Snow Family use cases

• Migrate large amounts of data into and out of the cloud

without depending on networks

• You can use AWS Snowcone to collect, process, and move

data to AWS, either offline by shipping the device, or online
with AWS DataSync

• AWS Snowball Edge Storage Optimized devices are well

suited for large-scale data migrations

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recap
• Streamline Data and Application Migrations

• AWS DataSync

• AWS Storage Gateway

• Snow Family Devices

• Demo of using AWS DataSync and AWS Storage Gateway helping

transferring/migrating data to AWS

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Additional resources

• AWS DataSync : https://aws.amazon.com/datasync

• AWS Storage Gateway : https://aws.amazon.com/storagegateway

• AWS Snow Family : https://aws.amazon.com/snow

• AWS Storage : https://aws.amazon.com/products/storage/

Get hands-on experience with the AWS online data migration workshop

https://github.com/aws-samples/aws-online-data-migration-workshop

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS TRAINING & CERTIFICATION
Access 500+ Free Digital Courses
with AWS Skill Builder
Focus on the cloud skills and services that are most relevant
to you across 30+ AWS solutions, including digital self-
paced learning plans and Ramp-Up Guides.

• Build your future in the AWS Cloud at your own pace

• Advance your skills and knowledge with learning plans
• Validate your cloud expertise with AWS Certification

LEARN YOUR WAY EXPLORE.SKILLBUILDER.AWS »

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you for attending AWS Builders Online Series
We hope you found it interesting! A kind reminder to complete the survey.
Let us know what you thought of today’s event and how we can improve the event
experience for you in the future.

[email protected]
twitter.com/AWSCloud

facebook.com/AmazonWebServices
youtube.com/user/AmazonWebServices

linkedin.com/company/amazon-web-services
twitch.tv/aws

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.