IT Infrastructure Architecture

Infrastructure Building Blocks

and Concepts

Storage – Part 2
(chapter 9)
 Network Attached Storage (NAS)
• A NAS, also known as a File Server, is a
network device that provides a shared file
system to operating systems over a standard
TCP/IP network
– NFS (UNIX and Linux)
– SMB/CIFS (Windows)
• A NAS is often an appliance that implements
the file services and holds the disks on
which data is stored
• A NAS appliance could also use external disk
storage provided by a SAN
• Can provide snapshot and clone technology
at a file level, enabling features like “un-
erasing” deleted files by end users
 Network Attached Storage (NAS)
• The difference between a SAN and NAS:
– SAN:
• Offers disk blocks (unformatted disks called Logical unit
numbers) that can be used by only one server
• Uses iSCSI, Fibre Channel or FCoE as the communication layer
– NAS:
• Offers a shared filesystem to store files that can be used by
multiple servers
• Connects to for instance to an LDAP or Active Directory
service in order to set file and/or folder permissions
• Uses SMB/CIFS or NFS over TCP/IP as the communication
layer
 Network Attached Storage (NAS)
• A clustered NAS is a NAS that uses a
distributed file system running simultaneously
on multiple servers
– Distributes data and metadata across storage
devices
– Still provides unified access to the files from any of
the cluster nodes, unrelated to the actual location
of the data
 Object Storage
• Object storage is a storage architecture that manages data as
objects, where an object is defined as a file with its metadata,
and a globally unique identifier called the object ID
• Examples of metadata:
– Filename
– Date and time stamps
– Owner
– Access permissions
– The level of data protection
– Replication settings to for instance a different geography
• Object storage stores and retrieves data using a REST API over
HTTP, served by a webserver, and is designed to be highly
scalable
 Object Storage
• A traditional file system provides a structure that
simplifies locating files
– For example, a log file is stored in /var/log/proxy/proxy.log
• In object storage, a file’s object ID must be administered
by the application using it
– Using the object ID, the object can be found without knowing
the physical location of the data
– For example, an application has administered that its log file is
stored in object ID 8932189023
• Using object IDs enables simplicity and massive scalability
of the storage system
– The object ID is a link to an object that can be stored anywhere
 Object Storage
• Data in object storage can’t be modified
– The original file must be deleted, and a new file must be
created, leading to a new object ID
• This makes object storage unsuitable for frequently
changing data
• It is a good fit for data that doesn't change much,
like:
– Backups
– Archives
– Video and audio files
– Virtual machine images
 Object Storage
• Some systems emulate a file system using
object storage
– For instance, Amazon’s S3FS creates a virtual
filesystem, based on S3 object storage, that can be
mounted to an operating system in the traditional
way, however, with significant performance
degradation
– A much better solution is to use object storage
with applications designed for it
 Software Defined Storage
• Software
Defined Storage
(SDS) abstracts
data and storage
capabilities (also
known as the
control plane)
from the
underlying
physical storage
systems (the
data plane)
 Software Defined Storage
• SDS virtualizes all physical storage into one large
shared storage pool
– Data can be stored in a variety of storage systems
while being presented and managed as one storage
pool to the servers consuming the storage
• Storage can be implemented as software running
on commodity x86-based servers with direct
attached disks
• Physical storage can also be a SAN, a NAS, or an
Object storage system
 Software Defined Storage
• From the shared storage pool, software
provides data services like:
– Deduplication
– Compression
– Caching
– Snapshotting
– Cloning
– Replication
– Tiering
 Software Defined Storage
• SDS provides servers with virtualized data storage pools
– With the required performance, availability and security
– Delivered as block, file, or object storage
– Based on policies
• Example:
– A newly deployed database server can invoke an SDS policy that mounts
storage configured to have its data striped across a number of disks,
creates a daily snapshot, and has data stored on tier 1 disks
• APIs can be used to provision storage pools and set the
availability, security and performance levels of the virtualized
storage
• Using APIs, storage consumers can monitor and manage their
own storage consumption
Storage availability
 Redundancy and data replication
• To increase availability in a SAN, components like HBAs and switches
can be installed redundantly
• Using multiple paths between HBAs and SAN switches, failover can be
instantiated automatically when a failure occurs
• Multiple storage systems can be used. Using replication, changed disk
blocks from the primary storage system are continuously sent to the
secondary storage system, where they are stored as well
 Redundancy and data replication
• Synchronous replication:
– Each write to the active storage system and the replication to
the passive storage system must be completed before the write
is confirmed to the operating system
– Ensures data on both storage systems is synchronized at all
times and data is never lost
– When the physical cable length between the two storage
systems is more than 100 km, latency times get too long,
slowing down applications, that have to wait for the write on
the secondary storage system to finish
– Risk: a failing connection between both storage systems a write
is never finished, as the data cannot be replicated. This
effectively leads to downtime of the primary storage system
 Redundancy and data replication
• Asynchronous replication:
– After data has been written to the primary storage
system, the write is immediately committed to the
operating system, without having to wait for the
secondary storage array to finish its writes as well
– Asynchronous replication does not have the latency
impact that synchronous replication has
– Disadvantage: potential data loss when the primary
storage system fails before the data has been
written to the secondary storage system
 Backup and recovery
• Backups are copies of data, used to restore data
to a previous state in case of data loss, data
corruption or a disaster recovery situation
• Backups are always a last resort, only used if
everything else fails, to save your organization in
case of a disaster
• A well-designed system should have options to
repair incorrect data from within the system or
by using systems management tools (like
database tools)
 Backup and recovery
• In general, backups should not be kept for a long
time
– Because the data copies are only relevant in the
event of a disaster, organizations will typically have
little use to restore a data backup that is more than
a few weeks old
– Restoring a backup takes you back in time
• Like a time machine, but without the rest of the world –
like your business partners and customers – going back in
time as well
 Backup and recovery
• A common mistake is to mix up backup with archiving
– Backup is about protection against data loss
– Archiving deals with long term data storage, in order to
comply with law and regulations
• Backups should not be used to view the status of
information from the past
– It should be possible to retrieve these statuses from the
system itself
– No data should ever be deleted in a typical production
system
– Older data could be archived to a secondary system or
database
 Backup and recovery
• Backups need to be made at a regular basis
– Usually daily
– Sometimes more often – every hour, or even
continuously in highly critical environments
• 3-2-1 rule:
– Keep three copies of your data
– on two different media types
– with one copy stored at a separate location
 Backup and recovery
• Backups must be available at a secondary site for
restore
– Experience with real world disasters shows it is good
practice to have a distance of at least 5 km between the
main site and the backup data
• Apart from application data, a copy must be available
on the secondary site of:
– Operating system installation disks
– Printed procedures on how to build up a new system using
the backups
– License keys of the software (including the restore software)
 Backup and recovery
• Test the restore procedure at least once a year to
ensure restores work as planned
– Include building up new hardware
– Have restore procedures tested by a third party, or at
least by people that have not performed a restore before
– In case of a real disaster we cannot assume that systems
managers are able to restore data again
• Restore tests should be performed each month to
ensure backup media still work as expected
– Restore some files
– Do the tapes really contain the expected data?
 Backup schemes
• A backup scheme describes what data is backed-
up, when, and how
• Backup schemes can become very complex in
large environments with many applications
• Four basic backup schemes
 Backup schemes
• Full backup
– A complete copy of all data
– Full backups are only created at relatively large
intervals (like a week or a month)
– Creating them takes much time, disk or tape
space, and bandwidth
– Restoring a full backup takes the least amount of
time
 Backup schemes
• Incremental backup
– Save only newly created or changed data since the
last backup, regardless of whether it is a previous
incremental backup or a full backup
– Restoring an incremental backup can take a long
time
• Especially when the last full backup is many
incremental backups ago
 Backup schemes
• Differential backup
– Save only newly created or changed data since the
last full backup
– Restoring a differential backup is quite efficient, as
it implies restoring a full backup and only the most
recent differential backup
 Backup schemes
• Continuous Data Protection (CDP)
– Guarantees that every change in the data is also
simultaneously made in the backup system
– The RPO (Recovery Point Objective) is set to zero,
because each change immediately triggers a
backup process
– Expensive technology, and therefore only used in
specific situations
 Backup data retention time
• Backup data retention time is the amount of time in which a given set
of data will remain available for restore
• Defines how long backups are kept and at which interval
• In practice, a Grandfather-Father-Son (GFS) based schedule is often
used:
– Each day a backup is made
– After a week, there are seven backups, of which the oldest backup is
renamed to a weekly backup
– After the second week, the same is done and the daily backups of the week
before are deleted
– Now there are eight backups: seven daily, two weekly
– Every four weeks, the weekly backup is renamed as a monthly backup and
the weekly backups are reused
– The daily backups are the son, the weekly backups are the father, and the
monthly backups are the grandfather
 Archiving
• Archiving is mostly done for compliancy and
regulation reasons
• Example:
– US regulations require all medical records to be
retained for 30 years after a person's death
– This means that X-rays taken when a child was born
must be kept for as much as 130 years!
• Noncompliance to law and regulation can lead
to serious business disruption, fines, and even
jail time
 Archiving
• Archived data is read-only to protect it from
being altered
– Very important for regulatory compliance and non-
repudiation
– Some archiving systems store data in an encrypted
form and use digital signatures to prove data is not
tampered with
– Some systems allow data to be written to it for
archiving, but disallow changing or deleting data
• CD / DVD/ Blu-ray
• WORM tapes
 Archiving
• Data must be kept in such a way that it is guaranteed the data
can be read after a long time
– Digital format (like a Microsoft Word file or a JPG file)
– Physical format (like a DVD or a magnetic tape)
– Storage environment (temperature, humidity)
• Use open standards for storing archived data
– Open standards are well documented
– Reading data will always be feasible, using emulation software if
needed
– Storing all documents in structured human-readable XML text files is
one way to ensure data can be read for many decades
• Transfer data that is to be kept for a long time to the latest
storage media standard every 10 years
Storage performance
 Disk performance
• Disk performance is dependent on:
– Disk rotation speed
– Seek times
– Interface protocol

• Some common examples of rotation

delay:
Disk RPM Average rotational delay (ms)
5,400 5.6
7,200 4.2
10,000 3
15,000 2
 Disk performance
• Disks cannot spin much faster than 15,000 RPM
– At this speed the velocity at the edge of a 3.5” disk
is 250 km/h!
– Increasing this velocity would physically destroy the
disk
• Seek time is the time it takes for the head to get
to the right track
– Average seek times:
• 3 ms for high-end disks
• 9 ms for low-end disks
 IOPS
• Input/output Operations Per Second (IOPS) is a measure of how
many read and write operations a disk can complete in one second

• Writing is typically a bit slower than reading

• Typical IOPS:
Disk RPM I OPS
7,200 50
10,000 120
15,000 160
SSD 2,500 to 10,000
 RAID penalty
• In RAID sets multiple disks are used to form one
virtual disk (Logical unit number)
• Writing data on multiple disks introduces some
delay, known as the RAID penalty
 Interface throughput
• Storage performance is also dependent on how fast the interface can
move data from the disks to the systems consuming the data and vice
versa
• An overview of the various interface speeds:
I nterface Speed
100 MB/ s
I DE (Parallel ATA)
133 MB/ s
1.5 Gbit/ s (192 MB/ s)
SATA 3 Gbit/ s (384 MB/ s)
6 Gbit/ s (768 MB/ s)
160 MB/ s (Ultra-160)
SCSI
320 MB/ s (Ultra-320)
1.5 Gbit/ s (192 MB/ s)
SAS 3 Gbit/ s (384MB/ s)
6 Gbit/ s (768 MB/ s)
1 Gbit/ s (128 MB/ s)
2 Gbit/ s (256 MB/ s)
FC 4 Gbit/ s (512 MB/ s)
8 Gbit/ s (1024 MB/ s)
16 Gbit/ s (2048 MB/ s)
 Caching
• A caching system in disk controllers can improve performance by
several orders of magnitude
– Read-cache acts as a buffer for reads. When the same data is read
multiple times, it is served from cache
– Write-through cache: data is written to cache and then to disk, and only
acknowledged as written when the data is physically written on the disk
– Write-through cache: allows the disk controller to acknowledge the
data as written as soon as it is held in cache. This allows the cache to
buffer writes quickly and then write the data to the slower disk when
the disk is ready to accept new I/O operations
• The type and amount of cache needed depends on what
applications need
– A web server, for instance, will mostly benefit from read-cache,
whereas most databases are better off with write cache
 Storage tiering
• Tiered storage creates a hierarchy of storage media, based on
cost, performance requirements, and availability requirements
• Example:
– Tier 1: Production data (SSD and SAS disks)
– Tier 2: Seldom used data, like email archives (NL-SAS disks)
– Tier 3: Backups (Virtual Tape Libraries on NL-SAS disks)
– Tier 4: Archived data (Tape or NL-SAS disks)
• The more tiers are used, the more effort it takes to manage
the tiers
• Automated tiering usually checks for file access times, file
creation date, and file ownership, and automatically moves
data to the storage medium that fits best
 Load optimization
• Storage performance is highly dependent on
the type of load
• Most vendors recommend a specific storage
configuration for their systems or applications
– For example, Oracle recommends a combination of
RAID 1 and 5 for its database in order to optimize
performance
Storage security
 Protecting data at rest
• Data can be:
– In transit (transported over a network)
– In use (by an application or a cache)
– At rest (on a disk or a tape)
• Data at rest can be secured using encryption techniques
– Prevent reading or writing data to disk or tape without the correct
encryption/decryption key
• Disk encryption in the datacenter has limited benefits:
– Databases and applications need to work with unencrypted data to
perform useful work
– Disk encryption is only useful when the disks are physically lost or
stolen (laptops, desktops, or removable media)
– Disks in the datacentre are in a physically secure area
 Protecting data at rest
• Disk encryption in the datacenter is useful:
– A disk drive might get in the wrong hands – for instance because it
was removed after it was marked "faulty" and was never destroyed
– In case of disk failure, having the data encrypted solves the issue of
having potentially sensitive data on a disk that can't be accessed
anymore, as it is defective
– Maintenance contracts often require that a failed disk must be sent
back to the vendor after replacing it with a new one. Without disk
encryption, returning disks may not be possible since a failed disk
cannot be erased anymore.
– Full disk encryption makes it harder for an attacker to retrieve data
from the "empty" space on the disks, which often contains traces
of previously stored data.
 Protecting data at rest
• Self-Encrypting Drives (SEDs):
– Use in laptops and desktops
– When an SED is powered up, authentication is required to access
data – the user must type in a password to start the boot
sequence of the computer
– Encryption is built into the disk drive’s hardware
– Encryption keys are stored on the disk
• Cryptographic Disk Erasure (CDE):
– Deletes the encryption key on the disk
– This has the same effect as erasing all disk contents
• Without the key, unencrypted data can no longer be read from the disk
• One of the best ways to fully wipe a disk’s contents
 SAN zoning
• SAN zoning is a method of arranging Fibre
Channel devices into logical groups on a SAN
fabric for security purposes
– SAN zoning is implemented in the SAN switches
– SAN zones are comparable with VLANs in Ethernet
networks
– Fibre Channel devices can only communicate with
each other if they are members of the same zone
 SAN Logical unit number masking
• In a SAN, Logical unit number masking makes a
Logical unit number available to some hosts
and unavailable to other hosts
• LUN masking is implemented primarily at the
HBA level, not in the SAN switches
• It is good practice to use a combination of SAN
zoning and Logical unit number masking