MODULE 1: INTRODUCTION TO

NOSQL

BY
SHASHIKALA H K
OUTLINE
 Why NoSQL?
 The Value of Relational Databases
 Getting at Persistent Data, Concurrency, Integration
 A (Mostly) Standard Model, Impedance Mismatch,
Application and Integration Databases
 Attack of the Clusters
 Emergence of NoSQL
 Aggregate Data Models: Aggregates: Example of
Relations and Aggregates, Consequences of Aggregate
Orientation, Summarizing Aggregate-Oriented Databases.
 More Details on Data Models: Relationships, Schema
less Databases, Materialized Views, Modeling for Data
Access
 Introduction to NOSQL
 Relational databases have been the default choice for data storage,
especially in the world of enterprise applications
 NoSQL, which stands for "Not Only SQL," it is used to describe a broad
category of DBMS that differ from traditional relational databases
(SQL databases) in terms of data model, storage, and processing.

Why NoSQL?
 There are several reasons why organizations might choose NoSQL
databases over traditional SQL databases for certain use cases:
1. Schema flexibility: NoSQL databases are often schema-less or
schema-flexible, allowing you to store and manage data without a
predefined schema.
 This flexibility is particularly beneficial when dealing with
unstructured or semi-structured data, as it allows for easier adaptation
to changing data requirements.
2. Scalability: Many NoSQL databases are designed to scale horizontally,
meaning they can handle a growing amount of data by adding more
servers to a distributed database system.
 This makes NoSQL databases suitable for applications with rapidly
increasing data volumes and traffic.
3. Performance: NoSQL databases are optimized for specific use
cases, providing high performance for certain types of queries and
operations.
 They are often designed to handle large amounts of read and write
operations efficiently, making them suitable for applications that
require low-latency data access.
4. Variety of data models: NoSQL databases support various data
models, such as document-oriented, key-value, column-family, and
graph databases.
 This diversity allows organizations to choose the most appropriate
model for their specific application needs.
5. Agile development and iteration: The flexibility of NoSQL
databases makes them well-suited for agile development
methodologies.
 Developers can quickly iterate on their applications without being
constrained by a rigid schema, making it easier to adapt to changing
requirements.
6. Handling large amounts of unstructured data: NoSQL
databases are often better equipped to handle unstructured or semi-
structured data, which is common in modern applications.
 This is particularly useful in scenarios where data doesn't fit neatly
into tables with predefined relationships.
7. Cost-effectiveness: NoSQL databases can be more cost-effective in
certain scenarios, especially when dealing with large-scale distributed
systems.
 They can leverage commodity hardware and scale horizontally,
potentially reducing infrastructure costs.
WHY RELATIONAL DATABASES BECAME SO
DOMINANT, AND WHY WE NEED NOSQL?

 The Value of Relational Databases

1. Getting at Persistent Data
 Two areas of memory:
 Fast, small, volatile main memory
 Larger, slower, non volatile backing store
 Since main memory is volatile, backing store was
used,
 The backing store can be:
 File system
 Database
 The database is more flexible than file system in storing large
amounts of data that allows an application program to get
information quickly and easily.
2. Concurrency
 Enterprise applications have many people using same data at once
(possibly modifying that data).
 Most of the time they are working on different areas of that data,
but occasionally they operate on the same bit of data.
 As a result , We have to worry about coordinating interactions
between them to avoid problems.
 Since enterprise applications can have lots of users and other
systems all working concurrently, there’s a lot of room for bad things
to happen.
 Relational databases help to handle this by controlling all access to
their data through transactions
 Transactions also play a role in error handling. With transactions,
you can make a change, and if an error occurs during the processing
of the change you can roll back the transaction to clean things up.
3. Integration
 Enterprise requires multiple applications, written by different
teams, to collaborate in order to get things done.
 Applications often need to use the same data and updates made
through one application have to be visible to others.
 A common way to do this is shared database integration where
multiple applications store their data in a single database.
 Using a single database allows all the applications to use each
others’ data easily, while the database’s concurrency control
handles multiple applications in the same way as it handles
multiple users in a single application.
4. A (Mostly) Standard Model
 Relational databases have succeeded because they provide the
core benefits in a (mostly) standard way.
 As a result, developers can learn the basic relational model and
apply it in many projects.
 Although there are differences between different relational
databases, the core mechanisms remain the same.
 Different vendors’ SQL semantics are similar, transactions
operate in mostly the same way.
IMPEDANCE MISMATCH
 For Application developers using relational databases, major
issue is impedance mismatch ( difference between relational
model and in-memory data structures).
 The relational data model organizes data into a structure of
tables.
 Tuple: set of name-value pairs and relation: set of tuples.
 The values in a relational tuple have to be simple—they cannot contain any
structure (nested record or list).
 This limitation isn’t true for in-memory data structures, which can take
much richer structures than relations.
 So if you want to use a richer in-memory data structure, you
have to translate it to a relational representation to store it on
disk.
 Hence the impedance mismatch—two different representations
that require translation.
 RELATIONAL MODEL VS. IN-MEMORY DATA
STRUCTURES:

 Relational Model: relational databases, organize data into tables

with rows and columns.
 They use SQL for querying and manipulating data.
 Relationships between tables are typically established using foreign
keys.

 In-Memory Data Structures: In contrast, when application

developers work with data in their code, they often use in-memory
data structures such as objects, arrays, and graphs.
 These data structures are more aligned with the programming
language's native representations.
CHALLENGES AND FRUSTRATIONS
 Object-Relational Mapping (ORM) Challenges: Developers face
challenges when trying to map between the relational model and in-
memory data structures.
 This mapping process is often manual or requires the use of ORM
tools, and mismatches can lead to complex, error-prone code.

 Performance Concerns: Retrieving data from a relational database

and mapping it to in-memory structures can be computationally
expensive, especially when dealing with complex queries or large
datasets.

 Expressiveness Differences: SQL, as a declarative language, has a

different expressiveness than imperative programming languages.
Bridging the gap between the two can be challenging.
SOLUTIONS AND MITIGATIONS:
 ORM Tools: helps to automate the mapping between the relational
model and in-memory data structures.
 Examples: Hibernate, Entity Framework, and Django ORM.

 In-Memory Databases: Some applications use in-memory databases

to reduce the impedance mismatch.
 These databases store data in a format closer to in-memory data
structures, improving performance.

 Caching Strategies: Caching can be employed to store frequently

accessed data in memory, reducing the need for repeated database
queries.

 Careful Design and Consideration: Developers need to be careful to

reduce differences between the two models during application design.
 Choosing appropriate data structures and designing efficient data
access patterns can mitigate some challenges.
ONGOING EFFORTS:
 New Database Paradigms: NoSQL databases, which often use
more flexible data models, have emerged as alternatives to
traditional relational databases.
 They can sometimes provide a better fit for applications with
specific requirements.

 Language and Database Integration: Some programming

languages and databases are working on improving integration.
 Ex: some databases support JSON data types, making it easier to
work with semi-structured data in an object-oriented manner.
 The impedance mismatch remains a significant challenge for
developers, and addressing it often involves a combination of careful
design, the use of appropriate tools, and ongoing efforts in both the
database and programming language communities.
 This impedance mismatch lead to relational databases being
replaced with databases that replicate the in memory data
structures to disk.
 Due to growth of OO programming languages, object-oriented
databases became popular (dominant environment for software
development in the new millennium).
 Impedance mismatch has made much easier to deal with wide
availability of object relational mapping frameworks ( Hibernate
and iBATIS ) that implement well-known mapping patterns, but
the mapping problem is still an issue.
 Relational databases continued to dominate the enterprise
computing world in the 2000s.
APPLICATION AND INTEGRATION DATABASES
 In relational databases, the database acts as an integration
database —where multiple applications developed by separate
teams storing their data in a common database.
 This improves communication because all the applications are operating
on a consistent set of persistent data.

There are downsides to shared database integration.

1. integrating many applications is more complex than any single
application needs.
 If an application wants to make changes to its data storage, it needs to
coordinate with all the other applications using the database.
2. Different applications have different structural and performance
needs, so index required by one application may cause a problem.
 A different approach is to treat your database as an application
database —which is only accessed by a single application
codebase that’s looked after by a single team.

Advantages
 With an application database, only the team using application
needs to know about the database structure, which makes it
much easier to maintain and evolve the schema.
 Since the application team controls both the database and the
application code, the responsibility for database integrity can
be put in the application code.
ATTACK OF THE CLUSTERS
 Large sets of data in websites appeared as: links, social networks,
activity in logs, mapping data.
 To Cope up with with the increase in data and traffic required more
computing resources.
 To handle this kind of increase, you have two choices: up or out

1. Scaling up implies:
 bigger machines
 more processors
 more disk storage
 more memory

Scaling up disadvantages:
 Bigger machines get more and more expensive.
 There are real limits as size increases.
2. Use lots of small machines in a cluster:
 The alternative is to use lots of small machines in a cluster.
 cluster of small machines uses cheaper hardware.
 more resilient—while individual machine failures are common, the overall
cluster can be built to keep going despite such failures, providing high
reliability.

Cluster disadvantages
 Relational databases are not designed to be run on clusters.
 Relational databases could also be run as separate servers for different sets
of data, effectively sharding the database.
 Even though this separates the load, all the sharding has to be controlled by
the application which has to keep track of which database server to talk to for
each bit of data.
 The mismatch between relational databases and clusters led
some organization to consider an alternative route to data
storage.
 Two companies in particular
1. Google
2. Amazon
 Both were running large clusters
 They were capturing huge amounts of data
 In 2000s, both companies produced brief but highly influential
papers about their efforts: – BigTable from Google – Dynamo
from Amazon
 Amazon and Google operate at scales, so they may not be
relevant to an average organization.
THE EMERGENCE OF NOSQL
 The term “NoSQL” first made its appearance in the late 90s as the
name of an open-source relational database.
 The usage of “NoSQL” that we recognize today traces back to a
meetup on June 11, 2009 in San Francisco.
 It is “open-source, distributed, nonrelational databases.”
 Some of the NOSQL systems are: Voldemort, Cassandra,
Dynomite, HBase, Hypertable, CouchDB, and MongoDB
 Relational databases use ACID transactions to handle consistency
across the whole database.
 NoSQL databases offer a range of options for consistency and
distribution
 Some of the organizations developed their own systems
 Google developed a proprietary NOSQL system known as BigTable,
Apache Hbase and column-based stores
 Amazon developed a NOSQL system called DynamoDB
 Facebook developed a NOSQL system called Cassandra
 CouchDB, which are classified as document-based NOSQL systems
or document stores
 graph-based NOSQL systems, or graph databases; these include
Neo4J and GraphBase
 OrientDB, combine concepts from many of the categories
CATEGORIES OF NOSQL SYSTEMS
 NOSQL systems have been characterized into four major
categories:
1. Document-based NOSQL systems: These systems store data
in the form of documents using JSON.
 Documents are accessible via document id or indexes.
2. NOSQL key-value stores: has simple data model based on fast
access by the key to the value associated with the key
3. Column-based or wide column NOSQL systems: These
systems partition a table by column into column families, where
each column family is stored in its own files.
4. Graph-based NOSQL systems: Data is represented as graphs,
and related nodes can be found by traversing the edges using
path expressions.
APPLICATIONS
 NoSQL databases are often used in applications where there is a high
volume of data that needs to be processed and analyzed in real-time.
 Social media analytics
 E-commerce
 Gaming
 Content management systems
 Document management, and
 Customer relationship management.
 NoSQL databases are used in real-time web applications and big data
and their use are increasing over time.
 However, NoSQL databases may not be suitable for all applications, as
they may not provide the same level of data consistency and
transactional guarantees as traditional relational databases.
 It is important to carefully evaluate the specific needs of an application
when choosing a database management system.
 FEATURES OF NOSQL DATABASES
1. NoSQL databases doesn’t use SQL as a query language. Instead,
database is manipulated through shell scripts that can be combined
into the usual UNIX pipelines.
2. They are generally open-source projects.
3. Most NoSQL databases are driven by the need to run on clusters.
4. Relational databases use ACID transactions to handle consistency
across the whole database. NoSQL databases offer a range of options
for consistency and distribution.
5. Graph databases are one style of NoSQL databases that uses a
distribution model similar to relational databases but offers a different
data model that makes it better at handling data with complex
relationships.
6. NoSQL databases operate without a schema, allows to freely add
fields to database records without having to define any changes in
structure first.
7. NoSQL databases are used in real-time web applications and big
data and their use are increasing over time.
CHARACTERISTICS OF NOSQL SYSTEMS
1. Schema-less or Schema-flexible:
 NoSQL databases typically allow for a flexible schema, i.e each
record in a database can have a different set of fields, and new
fields can be added without requiring a predefined schema.
2. Non-relational:
 NoSQL databases do not use a fixed schema and are not based
on the traditional tabular structure with rows and columns.
3. Horizontal Scalability:
 NoSQL databases are designed to scale horizontally, allowing
them to handle larger amounts of data and increased load by
adding more servers to a distributed database.
4. Distributed Architecture:
 Many NoSQL databases are designed to be distributed across
multiple servers or nodes, providing improved performance and
fault tolerance
5. High Performance:
 NoSQL databases are optimized for specific types of queries and are
capable of providing high-performance read and write operations.
6. Various Data Models:
 NoSQL databases support various data models such as document-
oriented (like MongoDB), key-value stores (like Redis), column-family
stores (like Apache Cassandra), and graph databases (like Neo4j).
7. BASE (Basically Available, Soft state, Eventually consistent):
 NoSQL databases often prioritize availability and partition tolerance
over strict consistency. i.e in the event of network partitioning, the
system may continue to operate but may provide inconsistent results
temporarily.
8. Big Data and Unstructured Data Support:
 NoSQL databases are often used for handling large volumes of
unstructured or semi-structured data, making them suitable for big
data applications.
9. Simple API:
 NoSQL databases typically offer simple APIs for data access and
manipulation, which can be more developer-friendly compared to
SQL-based databases.
10. Open Source:
 Many NoSQL databases are open source, allowing for community
collaboration and customization.
11. Availability and Replication
 NOSQL systems require continuous system availability.
 Two major replication models are used in NOSQL systems are:
 master-slave replication: one copy to be the master copy; all write
operations must be applied to the master copy and then propagated
to the slave copies
 master-master replication: allows reads and writes at any of the
replicas but may not guarantee that reads at nodes that store
different copies see the same values
12. Sharding of Files
 distribute the load of accessing the file records to multiple nodes.
 improves load balancing and data availability

13. Less Powerful Query Languages

 provide a set of functions and operations as a programming API
 Operations are called CRUD or SCRUD operations

14. Versioning
 provide storage of multiple versions of the data items, with the
timestamps of when the data version was created
ADVANTAGES OF NOSQL
1. High scalability: NoSQL databases use sharding for horizontal
scaling. horizontal scaling is easy to implement. Ex: horizontal
scaling databases are MongoDB, Cassandra, etc. NoSQL can handle
a huge amount of data because of scalability.
2. Flexibility: NoSQL databases are designed to handle unstructured
or semi-structured data, which means that they can accommodate
dynamic changes to the data model.
3. High availability: The auto, replication feature in NoSQL
databases makes it highly available
4. Scalability: NoSQL databases are highly scalable, i.e. they can
handle large amounts of data and traffic with ease. This makes
them a good fit for applications that need to handle large amounts of
data or traffic
5. Performance: NoSQL databases are designed to handle large
amounts of data and traffic, so they offers improved performance
compared to traditional relational databases.
6. Cost-effectiveness: NoSQL databases are often more cost-effective
than traditional relational databases, as they are typically less complex
and do not require expensive hardware or software.
7. Agility: Ideal for agile development.
DISADVANTAGES OF NOSQL
1. Lack of standardization: There are many different types of NoSQL
databases, each with its own unique strengths and weaknesses. This
lack of standardization can make it difficult to choose the right
database for a specific application
2. Lack of ACID compliance: NoSQL databases are not fully ACID-
compliant, i.e. they do not guarantee the consistency, integrity, and
durability of data. This can be a drawback for applications that require
strong data consistency guarantees.
3. Narrow focus: NoSQL databases have a very narrow focus as it is
mainly designed for storage but it provides very little functionality.
Relational databases are a better choice in the field of Transaction
Management than NoSQL.
4. Open-source: NoSQL is an open-source database. There is no reliable
standard for NoSQL yet. In other words, two database systems are
likely to be unequal.
5. Lack of support for complex queries: NoSQL databases are not
designed to handle complex queries, which means that they are not a
good fit for applications that require complex data analysis or
reporting.
6. Lack of maturity: NoSQL databases are relatively new and lack the
maturity of traditional relational databases. This can make them less
reliable and less secure than traditional databases.
7. Management challenge: Data management in NoSQL is much
more complex than in a relational database. NoSQL, in particular, has a
reputation for being challenging to install and even more hectic to
manage on a daily basis.
8. GUI is not available: GUI mode tools to access the database are not
flexibly available in the market.
9. Backup: Backup is a great weak point for some NoSQL databases
like MongoDB.
10. Large document size: Some database systems like MongoDB and
CouchDB store data in JSON format. This means that documents are quite
large.
WHEN SHOULD NOSQL BE USED
 When huge amount of data need to be stored and retrieved.
 The relationship between data you store is not that important.
 The data changing over time and is not structured.
 Support of constraint and joins is not required at database level.
 The data is growing continuously and you need to scale the database
regular to handle the data.
 Flexible Schema Requirements
 Scalability and High Throughput
 Rapid Development and Agile Iterations
 Big Data Applications
 High Availability and Fault Tolerance
 Real-time Applications(IOT)
IMPORTANT FEATURES OF NOSQL DATABASES
1. Not using the relational model
2. Running well on clusters
3. Open-source
4. Built for the 21st century web applications
5. Schemaless
AGGREGATE DATA MODELS
 Data Model: Model through which we identify and manipulate
our data. It describes how we interact with the data in the
database.
 Storage model: Model which describes how the database
stores and manipulates the data internally.
 In NoSQL “data model” refer to the model by which the
database organizes data more formally called a metamodel.
 The dominant data model is relational data model which uses
set of tables:
 Each table has rows
 Each row representing entity
 Column describe entity
 Column may refer to relationship
 Each NoSQL solution has a different model that it uses:
1. Key-value
2. Document
3. Column-family
4. Graph

 Out of this first three share a common characteristic of their

data models which is called as aggregate orientation.
 Aggregate orientation: helps to operate on data in units that
have a more complex structure than a set of tuples.
 Such as complex record that allows lists and other record
structures to be nested inside it.
 An aggregate is a collection of related objects that we wish to
treat as a unit.
Why aggregates matter?
 In relational databases, all these details would be split across
multiple tables.
 In NoSQL, aggregate-oriented databases store related data together,
making it easier to access and update.

Example of aggregates vs. relations:

 Relational model: A customer’s orders are in one table, shipping
details in another, and payment info in yet another table.
 Relational databases break data into separate tables for
normalization and flexibility but require multiple joins to retrieve
complex data.
 Aggregate model: Everything about a single order (items, shipping,
and payment) is stored together.
 NoSQL databases store related data together in a single structure,
which simplifies and speeds up data retrieval for specific use cases.
EXAMPLE OF RELATIONS AND AGGREGATES
 Let’s assume we have to build an e-commerce website; we are
going to be selling items directly to customers over the web, and
we will have to store information about users, our product
catalog, orders, shipping addresses, billing addresses, and
payment data.
 Data model for a relational database:
AN AGGREGATE DATA MODEL
 We’ve used the black-
diamond composition marker
in UML to show how data
fits into the aggregation
structure.
 The customer contains a list
of billing addresses;
 The order contains a list of
order items, a shipping
address, and payments.
 The payment itself contains
a billing address for that
payment
 black-diamond composition marker in UML to show how data fits
into the aggregation structure.
 The customer aggregate contains a list of billing addresses.
 The order aggregate contains a list of order items, a shipping
address, and payments.
 The payment itself contains a billing address for that payment.
 Here single logical address record appears three times but instead
of using IDs it’s treated as a value and copied each time.
 The link between the customer and the order isn’t within either
aggregate—it’s a relationship between aggregates.
 To draw aggregate boundary you have to think about accessing
that data—and make that part of your thinking when developing
the application data model.
 Indeed we could draw our aggregate boundaries differently, putting
all the orders for a customer into the customer aggregate Embed all
the objects for customer and the customer’s orders
CONSEQUENCES OF AGGREGATE
ORIENTATION
 Relational databases have no concept of aggregate within
their data model.
 In the NoSQL world, graph databases are also aggregate-
ignorant.
 It’s often difficult to draw aggregate boundaries well,
particularly if the same data is used in many different
contexts.
 Relational databases allow you to manipulate any
combination of rows from any tables in a single transaction.
Such transactions are called ACID transactions.
 In general, aggregate-oriented databases don’t have ACID
transactions that span multiple aggregates.
 Instead, they support atomic manipulation of a single
aggregate at a time.
MORE DETAILS ON DATA MODELS
Relationships
 Aggregates are useful because they put together data that is
commonly accessed together.
 But there are still lots of cases where data that’s related is
accessed differently.
 Ex: Consider the relationship between a customer and all of his
orders. Some applications will want to access the order history
whenever they access the customer; this fits in well with
combining the customer with his order history into a single
aggregate.
 Other applications, however, want to process orders
individually and thus model orders as independent aggregates.
 Aggregate oriented databases treat the aggregate as the unit of
data retrieval.
 Consequently, atomicity is only supported within the contents of
a single aggregate.
 If you update multiple aggregates at once, you have to deal
yourself with a failure partway through.
 Relational databases help you with this by allowing you to
modify multiple records in a single transaction, providing ACID
guarantees while altering many rows.
WITH NOSQL DATABASES, WAY OF
STORING DATA
 A key-value store allows you to store any data you like under a
key.
 A document database effectively does the same thing, since it
makes no restrictions on the structure of the documents you
store.
 Column-family databases allow you to store any data under any
column you like.
 Graph databases allow you to freely add new edges and freely
add properties to nodes and edges as you wish.
SCHEMALESS DATABASES
 A common theme across all the forms of NoSQL databases is that they are
schemaless.
 When you want to store data in a relational database, you first have to
define a schema—a defined structure for the database which says what
tables exist, which columns exist, and what data types each column can
hold.
 Before you store some data, you have to have the schema defined for it in
relational database.
With a schema:
 You have to figure out in advance what you need to store, but that can be
hard to do.

Without a schema:
 You can easily store whatever you need.
 You can easily add new things as you discover them.
 A schemaless store makes it easier to deal with nonuniform data: data
where each record has a different set of fields. It allows each record to
contain just what it needs—no more, no less.
GRAPH DATABASES
 Graph-based data models
store data in nodes that
are connected by edges.
 Aggregate Data Models
in NoSQL are widely
used for storing the huge
volumes of complex
aggregates and
multidimensional data
having many
interconnections between
them.
USE CASES:
 Graph-based Data Models are used in social networking
sites to store interconnections.
 It is used in fraud detection systems.
 This Data Model is also widely used in Networks and IT
operations.
MATERIALIZED VIEWS
 materialized views are views that are computed in advance and cached
on disk.
 Materialized views are effective for data that is read heavily.
 Materialized views are like pre-made summaries of data stored in a
database to save time when repeatedly accessing or analyzing the same
information.
 NoSQL databases don’t have views, they may have precomputed and
cached queries
 Example: Imagine a grocery store tracking daily sales:
 Without materialized views: Every time a manager asks for "total sales
for the day," the system calculates the total by going through all sales
records again.
 With materialized views: The system pre-calculates and stores "daily
total sales" at the end of the day. The next time the manager asks, it
retrieves the stored value instantly, saving time.
MODELLING FOR DATA ACCESS
 when modeling data
aggregates we need to consider
how the data is going to be
read as well as what are the
side effects on data related to
those aggregates.
 In this scenario, the
application can read
customer’s information and all
the related data by using the
key.
 If the requirements are to read
orders or products sold in each
order, the whole object has to
be read and then parsed on the
client side to build the results.