CLOUD COMPUTING UNIT 5

Evolution Of Storage Technology In Cloud Computing

The evolution of storage technology in cloud computing has been marked by significant advancements
aimed at improving scalability, performance, and reliability. Here’s an overview of key developments:

1. Initial Stage: Traditional Storage Solutions (2000s)

 Direct Attached Storage (DAS): Early cloud services often used DAS, where storage was
physically attached to servers. This limited scalability and accessibility.
 Network Attached Storage (NAS): Introduced centralized file storage accessible over a network,
allowing multiple users to share files but still limited in terms of scalability.

2. Advent of Virtualization (Mid-2000s)

 Storage Virtualization: Combined multiple physical storage devices into a single virtual unit,
enabling better resource utilization and management.
 Introduction of SAN (Storage Area Network): Offered high-speed networks of storage devices,
improving performance and reliability for enterprise applications.

3. The Rise of Cloud Storage (Late 2000s)

 Public Cloud Storage Solutions: Providers like Amazon S3, Google Cloud Storage, and Microsoft
Azure emerged, offering scalable and accessible storage solutions.
 Object Storage: Unlike traditional file systems, object storage enabled data to be stored as
discrete units (objects) with metadata, making it more scalable and suitable for unstructured
data.

4. Advanced Technologies and Hybrid Solutions (2010s)

 Hybrid Cloud Storage: Combination of on-premises and cloud storage, allowing businesses to
leverage the scalability of the cloud while retaining control over sensitive data.
 File Storage Services: Solutions like Amazon EFS and Azure Files provided managed file systems
in the cloud, making it easier for applications to access files.

5. Emergence of New Architectures (2020s)

 Serverless Storage: Services like AWS Lambda and Google Cloud Functions promote a serverless
architecture, allowing users to focus on applications without managing infrastructure.
 Multi-cloud Strategies: Organizations began adopting multi-cloud environments, leveraging
multiple providers to optimize costs, performance, and redundancy.

6. Innovations in Data Management and Access (Present)

 AI and Machine Learning: Enhanced data management, optimizing storage utilization, and
predicting usage patterns for better cost management.
  Edge Computing: Storage solutions are now integrated with edge computing, allowing data to
be processed closer to its source, reducing latency and bandwidth use.

7. Future Trends

 Decentralized Storage: Technologies like blockchain are being explored for decentralized
storage solutions, enhancing security and redundancy.
 Sustainable Storage Solutions: Focus on green computing and energy-efficient storage
technologies to reduce the environmental impact of cloud data centers.

Storage Models In Cloud Computing

In cloud computing, various storage models cater to different use cases, performance needs, and scalability
requirements. Here’s an overview of the primary storage models:

1. Object Storage

 Description: Stores data as objects within a flat address space. Each object includes the data itself,
metadata, and a unique identifier.
 Use Cases: Best for unstructured data, such as media files, backups, and large datasets.
 Examples: Amazon S3, Google Cloud Storage, Microsoft Azure Blob Storage.

2. Block Storage

 Description: Divides data into fixed-size blocks, which can be independently managed. This model allows
for high-performance storage that can be attached to servers.
 Use Cases: Ideal for applications requiring low-latency access, such as databases and enterprise
applications.
 Examples: Amazon Elastic Block Store (EBS), Google Persistent Disk, Azure Managed Disks.

3. File Storage

 Description: Organizes data in a hierarchical structure of files and directories. This model mimics
traditional file systems.
 Use Cases: Suitable for applications that require shared file access, such as content management systems
and development environments.
 Examples: Amazon Elastic File System (EFS), Azure Files, Google Filestore.

4. Hybrid Storage

 Description: Combines on-premises storage with cloud storage, allowing data to be stored and accessed
across both environments.
 Use Cases: Useful for businesses needing flexibility, such as those managing sensitive data or needing
backup solutions.
 Examples: VMware Cloud on AWS, Azure Stack.

5. Cold Storage

 Description: A cost-effective solution for storing infrequently accessed data. Data retrieval may take
longer compared to other models.
 Use Cases: Ideal for archival storage, backups, and compliance data.
  Examples: Amazon S3 Glacier, Google Coldline Storage, Azure Blob Storage (Cool and Archive tiers).

6. Distributed File Systems

 Description: A file system that allows access to files across multiple servers, enabling high availability and
fault tolerance.
 Use Cases: Suitable for big data applications and environments requiring redundancy and parallel
processing.
 Examples: Hadoop Distributed File System (HDFS), Ceph, GlusterFS.

7. Database Storage

 Description: Specialized storage for structured data managed by database management systems (DBMS).
 Use Cases: Best for applications requiring complex querying and transaction support, such as web
applications and financial systems.
 Examples: Amazon RDS, Google Cloud SQL, Azure SQL Database.

8. In-Memory Storage

 Description: Stores data in RAM rather than on disk, providing extremely fast access speeds.
 Use Cases: Ideal for applications requiring real-time processing, such as caching and session management.
 Examples: Amazon ElastiCache, Azure Cache for Redis

File Systems And Database In Cloud Computing

In Cloud Computing, File Systems And Databases Serve As Essential Components For Data Storage And
Management. Here’s A Detailed Look At Both:

File Systems In Cloud Computing

1. Object Storage

 Description: Data Is Stored As Objects In A Flat Namespace, Each With A Unique Identifier And
Metadata.
 Examples: Amazon S3, Google Cloud Storage.
 Use Cases: Ideal For Unstructured Data, Such As Images, Videos, And Backups.

2. File Storage

 Description: Organizes Data In A Hierarchical File-And-Directory Structure, Similar To Traditional

File Systems.
 Examples: Amazon Elastic File System (Efs), Azure Files, Google Filestore.
 Use Cases: Suitable For Applications Requiring Shared File Access, Like Content Management
Systems And Collaborative Tools.

3. Distributed File Systems

  Description: Allows Files To Be Stored Across Multiple Servers, Providing Redundancy And High
Availability.
 Examples: Hadoop Distributed File System (Hdfs), Ceph, Glusterfs.
 Use Cases: Useful For Big Data Applications And Environments Requiring Parallel Processing.

4. Network File Systems (Nfs)

 Description: A Protocol Allowing Remote Access To Files As If They Were Stored Locally.
 Examples: Nfs On Aws, Azure Nfs.
 Use Cases: Useful For Legacy Applications That Require Traditional File Access Methods.

Databases In Cloud Computing

1. Relational Databases

 Description: Structured Data Is Stored In Tables With Predefined Schemas, Supporting Sql For
Querying.
 Examples: Amazon Rds, Google Cloud Sql, Azure Sql Database.
 Use Cases: Ideal For Applications Needing Complex Transactions And Relationships, Like
Financial Systems And Inventory Management.

2. Nosql Databases

 Description: Non-Relational Databases Designed To Handle Unstructured Or Semi-Structured

Data. They Can Store Data In Various Formats, Such As Key-Value, Document, Column-Family, Or
Graph.
 Examples: Amazon Dynamodb (Key-Value), Mongodb Atlas (Document), Google Bigtable
(Column-Family).
 Use Cases: Suitable For Big Data Applications, Real-Time Analytics, And Applications With
Varying Data Structures.

3. Data Warehouses

 Description: Specialized Systems For Analytical Processing Of Large Volumes Of Data, Optimized
For Read-Heavy Workloads.
 Examples: Amazon Redshift, Google Bigquery, Snowflake.
 Use Cases: Best For Business Intelligence, Reporting, And Data Analytics.

4. In-Memory Databases

 Description: Databases That Primarily Rely On Memory For Data Storage To Provide Fast Access
Speeds.
 Examples: Amazon Elasticache, Redis, Azure Cache For Redis.
 Use Cases: Ideal For Caching, Real-Time Analytics, And Session Management.

5. Newsql Databases

 Description: Combine The Scalability Of Nosql With The Acid Guarantees Of Traditional Sql
Databases.
  Examples: Cockroachdb, Google Spanner.
 Use Cases: Suitable For Applications Requiring High Scalability Without Sacrificing Consistency.

Key Differences

 Data Structure: File Systems Are Often Used For Unstructured Or Semi-Structured Data, While
Databases Are Structured, Typically Following A Predefined Schema.
 Access Patterns: File Systems Allow For File-Based Access, While Databases Offer Complex
Querying Capabilities.
 Scalability: Databases Can Offer Horizontal Scalability (Especially Nosql), Whereas File Systems
Are Designed To Handle large volumes of data across distributed environments.

Distributed File Systems In Cloud Computing

Distributed file systems (DFS) in cloud computing enable the storage and management of data across
multiple servers or nodes. This approach enhances redundancy, scalability, and fault tolerance. Here’s a
detailed look at distributed file systems in the cloud:

Key Features of Distributed File Systems

1. Scalability
o DFS can easily scale horizontally by adding more nodes, allowing for increased storage
capacity and performance as data volumes grow.
2. Fault Tolerance
o Data is replicated across multiple nodes, ensuring that if one node fails, data remains
accessible from other nodes. This redundancy helps in disaster recovery.
3. Data Consistency
o DFS employs various strategies to maintain data consistency across nodes, ensuring that
users see the same data at all times.
4. High Availability
o By distributing data across multiple locations, DFS can provide continuous access even
during node failures or maintenance.
5. Parallel Processing
o DFS supports concurrent access to data from multiple users or applications, improving
performance for large-scale data processing tasks.

Popular Distributed File Systems in Cloud Computing

1. Hadoop Distributed File System (HDFS)

o Description: Designed for big data applications, HDFS is part of the Apache Hadoop
ecosystem. It stores large files by splitting them into blocks and distributing them across
a cluster.
o Use Cases: Data processing for analytics, machine learning, and big data workloads.
2. Ceph
o Description: A highly scalable, open-source storage system that provides object, block,
and file storage in a unified system. Ceph is known for its high availability and
performance.
 o Use Cases: Cloud infrastructures needing object and block storage, suitable for virtual
machines and cloud-native applications.
3. GlusterFS
o Description: An open-source distributed file system that aggregates storage resources
from multiple servers into a single global namespace.
o Use Cases: Media streaming, backup solutions, and cloud storage environments.
4. Google File System (GFS)
o Description: A proprietary distributed file system designed for Google’s internal
applications, optimized for handling large files and providing fault tolerance.
o Use Cases: Data-intensive applications within Google’s infrastructure.
5. Amazon Elastic File System (EFS)
o Description: A fully managed file storage service that provides scalable file storage for
use with Amazon EC2 instances.
o Use Cases: Content management systems, web applications, and shared file access.
6. Azure Data Lake Storage
o Description: A scalable and secure data lake that allows you to store data of any size,
type, or ingestion speed. It integrates seamlessly with Azure analytics services.
o Use Cases: Big data analytics and machine learning workloads.

Advantages of Distributed File Systems

 Cost-Effectiveness: By utilizing commodity hardware, organizations can build scalable storage

solutions without significant investment in proprietary systems.
 Data Durability: With multiple copies of data stored across nodes, DFS ensures high durability
and reduces the risk of data loss.
 Flexibility: Organizations can configure distributed file systems to meet specific performance
and scalability needs, adapting to changing requirements.

Challenges

 Complexity: Managing and configuring a distributed file system can be complex, requiring
specialized knowledge and skills.
 Performance Overhead: While DFS provides scalability, there may be performance trade-offs
due to data replication and network latency.
 Consistency Models: Ensuring strong consistency in a distributed environment can be
challenging and may require careful design considerations.

General Parallel File Systems IN Cloud Computing

General parallel file systems (GPFS) are designed to handle the simultaneous access and management of
data across multiple nodes, optimizing performance and scalability. In cloud computing, GPFS play a
critical role in managing large datasets, particularly for high-performance computing (HPC) and big data
applications. Here’s an overview of GPFS in the context of cloud computing:
Key Features of General Parallel File Systems

1. Parallel Access
o GPFS allows multiple clients to read from and write to the file system simultaneously,
significantly improving data throughput and reducing access times.
2. Scalability
o These systems can scale out easily by adding more storage nodes or servers,
accommodating increasing data volumes and user demands.
3. High Availability
o GPFS provides data redundancy and replication across nodes, ensuring continuous
access to data even in the event of hardware failures.
4. Data Striping
o Data is split into smaller chunks and distributed across multiple storage devices. This
striping enhances performance by enabling parallel data transfers.
5. Consistency and Data Integrity
o GPFS maintains data consistency and integrity across distributed nodes, often using
advanced locking mechanisms to manage concurrent access.

Examples of General Parallel File Systems

1. IBM Spectrum Scale (formerly GPFS)

o Description: A high-performance, scalable file system that supports parallel access,
suitable for large-scale data processing.
o Use Cases: HPC environments, big data analytics, and enterprise applications requiring
large-scale data management.
2. Lustre
o Description: An open-source parallel file system designed for large-scale cluster
computing. It supports thousands of clients and high-throughput workloads.
o Use Cases: Commonly used in supercomputing centers and research institutions for
scientific applications.
3. BeeGFS
o Description: A parallel file system that is easy to install and manage. It is optimized for
performance and flexibility, suitable for diverse workloads.
o Use Cases: HPC, rendering farms, and media production.
4. PanFS (by Panasas)
o Description: A parallel file system designed for performance and simplicity, providing a
scalable solution for data-intensive applications.
o Use Cases: HPC environments, big data analytics, and machine learning workloads.
5. Ceph
o Description: While primarily known for its object and block storage capabilities, Ceph
can also provide file system functionality through CephFS, supporting parallel access.
o Use Cases: Cloud infrastructures and environments requiring unified storage solutions.

Advantages of General Parallel File Systems

 Enhanced Performance: GPFS can significantly improve read and write speeds for large files due
to simultaneous access across multiple nodes.
 Scalable Architecture: As data needs grow, GPFS can easily scale by adding more storage
resources without major overhauls.
  Flexibility: These systems can adapt to various workloads, making them suitable for diverse
applications in research, enterprise, and cloud environments.

Challenges

 Complexity of Management: Setting up and managing GPFS can be complex and may require
specialized expertise.
 Cost: High-performance parallel file systems can be expensive to implement, especially in terms
of the underlying hardware and software.
 Network Dependency: Performance can be affected by network latency and bandwidth,
particularly when accessing data across geographically distributed locations.

Google file system IN Cloud Computing

The Google File System (GFS) is a proprietary distributed file system developed by Google to handle
large-scale data storage needs. It is designed to provide efficient, reliable, and scalable storage for
Google's extensive data processing requirements. Here’s an overview of GFS in the context of cloud
computing:

Key Features of Google File System

1. Distributed Architecture
o GFS operates across multiple machines, allowing it to store and manage vast amounts of
data. It is designed to handle thousands of files and petabytes of data efficiently.
2. Fault Tolerance
o GFS is built to be fault-tolerant. Data is automatically replicated across multiple servers,
ensuring that even if some nodes fail, data remains accessible.
3. High Throughput
o The system is optimized for high throughput rather than low latency, making it suitable
for applications that process large volumes of data, such as data analytics and machine
learning.
4. Data Integrity
o GFS maintains data integrity through checksums and versioning. It regularly checks data
integrity during reads and writes to ensure consistency.
5. Streamlined for Large Files
o It is optimized for handling large files (typically larger than 64 MB), making it ideal for
data-heavy applications. Small files can lead to inefficiencies, so GFS emphasizes larger
file sizes.
6. Master-Slave Architecture
o GFS uses a master-slave architecture where a single master node manages the
metadata, while chunk servers handle the actual data storage. This design allows for
efficient data management and quick metadata operations.

Components of GFS

1. Master Node
 oThe master node is responsible for managing the metadata, including the namespace,
access control, and the mapping of files to data chunks. It also coordinates data
replication and consistency.
2. Chunk Servers
o Data is divided into fixed-size chunks (usually 64 MB), which are stored on chunk
servers. Each chunk is replicated across multiple servers for redundancy.
3. Clients
o Clients interact with GFS by reading and writing data through the master node, which
directs them to the appropriate chunk servers.

Use Cases

 Data Processing: GFS is widely used in Google’s data processing frameworks, such as
MapReduce, to efficiently manage large datasets.
 Machine Learning: It supports machine learning applications that require large volumes of
training data, enabling rapid access and processing.
 Big Data Analytics: GFS is fundamental for applications that analyze massive datasets, allowing
organizations to derive insights from their data.

Advantages of GFS

 Scalability: GFS can scale horizontally by adding more chunk servers, accommodating increasing
data volumes without significant performance loss.
 Reliability: With its fault-tolerant design, GFS ensures that data remains accessible and secure,
even in the event of hardware failures.
 Performance: The system is optimized for high throughput, making it suitable for data-intensive
applications that require efficient data access.

Challenges

 Complexity: Implementing and managing a GFS-like system can be complex, requiring

specialized knowledge and resources.
 Single Point of Failure: The master node is a single point of failure; however, GFS has
mechanisms for redundancy and recovery to mitigate this risk.
 Geographical Limitations: While GFS is highly effective for Google’s infrastructure, its design is
tailored to Google’s needs and may not be as suitable for diverse cloud environments.