MongoDB Architecture

Core Processes:
The core components in the MongoDB package are
• mongod , which is the core database process
• mongos , which is the controller and query router for sharded clusters
• mongo , which is the interactive MongoDB shell
Mongod:

mongod is the primary daemon process for the MongoDB system. It handles data requests,
manages data access, and performs background management operations.This document provides
a complete overview of all command line options for mongod. These command line options are
primarily useful for testing: In common operation, use the configuration file options to control
the behavior of your database.

When a mongod is run without any arguments, it connects to the default data directory, which is
C:\data\db or /data/db , and default port 27017, where it listens for socket connections. It’s
important to ensure that the data directory exists and you have write permissions to the directory
before the mongod process is started. If the directory doesn’t exist or you don’t have write
permissions on the directory, the start of this process will fail. If the default port 27017 is not
available, the server will fail to start. mongod also has a HTTP server which listens on a port
1000 higher than the default port, so if you started the mongod with the default port 27017, in
this case the HTTP server will be on port 28017 and will be accessible using the URL
http://localhost:28017 . This basic HTTP server provides administrative information about the
database.

Mongo:

mongo provides an interactive JavaScript interface for the developer to test queries and
operations directly on the database and for the system administrators to manage the database.
This is all done via the command line. When the mongo shell is started, it will connect to the
default database called test . This database connection value is assigned to global variable db. As
a developer or administrator you need to change the database from test to your database post the
first connection is made. You can do this by using dbname.

Mongos:

For a sharded cluster, the mongos instances provide the interface between the client applications
and the sharded cluster. The mongos instances route queries and write operations to the
shards.mongos is used in MongoDBsharding. It acts as a routing service that processes queries
from the application layer and determines where in the sharded cluster the requested data is
located.
MongoDB Tools

mongodump

Creates a binary export of the contents of a mongod database.

mongorestore

Restores data from a mongodump database dump into a mongod or mongos.The binary database
dump created by the mongodump utility is imported to a new or an existing database using the
mongorestore utility

bsondump

Converts BSON dump files into JSON.: This utility converts the BSON files into human-
readable formats such as JSON and CSV. For example, this utility can be used to read the output
file generated by mongodump.

mongoimport ,mongoexport :mongoimport provides a method for taking data in JSON, CSV,
or TSV formats and importing it into a mongod instance. mongoexport provides a method to
export data from a mongod instance into JSON, CSV, or TSV formats.

mongoimport

Imports content from an Extended JSON, CSV, or TSV export file.

mongoexport

Produces a JSON or CSV export of data stored in a mongod instance.

mongostat ,mongotop , mongosniff: These utilities provide diagnostic information related to

the current operation of a mongod instance

mongostat

Provides a quick overview of the status of a currently running mongod or mongos instance.

mongotop

Provides an overview of the time a mongod instance spends reading and writing data.

Standalone Deployment:

 What It Is: Imagine you have a single notebook where you write all your important notes.
This notebook is your "database." You can access it and make changes anytime, but if
something happens to it—like if it gets lost or damaged—you lose all your notes.
  Why It's Used: This setup, called a "standalone deployment," is often used for practice or
development because it's simple and easy to manage. But because there's no backup, it's
risky to use in real-world (production) situations.
 Key Components: A standalone MongoDB deployment involves just one mongod
(MongoDB server) instance and a client that connects to it.

2. Replication:

 What It Is: Replication is like having multiple notebooks (copies) of your important
notes. If one notebook gets lost, you still have others, so you don’t lose any information.
 Why It's Important: In a real-world setting, you don’t want to lose data, especially if the
primary system fails. Replication helps by automatically making copies of your data
across different servers (nodes). If one server fails, the others can take over, ensuring your
data and applications remain available.
 How It Helps: Apart from protecting your data, replication also allows you to manage
tasks better. For example, you can take backups from one of the copies without affecting
the main database's performance.

Types of Replication in MongoDB:

A. Master/Slave Replication:

 What It Is: Imagine one "master" notebook where you write all your notes. You then
have several "slave" notebooks that automatically copy everything from the master
notebook. However, if the master notebook is lost, the copying stops.
 How It Works:
o The master server (mongod instance) records all the changes (like writing in the
master notebook).
o The slave servers replicate these changes (copy them into their own notebooks).
 Limitations:
o The master has to manage all the work, which can be overwhelming if there are too
many slaves.
 o If the master fails, replication stops, and you might need to manually fix the
situation.

B. Replica Set:

 What It Is: A replica set is a more advanced and reliable form of replication. Think of it
as a team where everyone has a copy of the same notebook. One person is elected as the
leader (primary) to make changes, and the others (secondaries) automatically update their
notebooks based on the leader’s notes.
 How It Works:
o If the leader (primary) is unable to continue, the team votes to choose a new leader
from the secondaries. This way, the work continues without interruption.
o The leader writes to the primary database, and the secondary nodes replicate the
data.
o If a secondary falls behind, it catches up by copying the latest notes (using an
"oplog," a special log of operations).
 Benefits:
Automatic Failover: If the primary node fails, a new primary is automatically chosen without
manual intervention.
High Availability: Your data is always available because multiple copies are maintained.
Scalability: You can spread the reading tasks across different nodes, making the system more
efficient.
The replica set is a sophisticated form of the traditional master-slave replication and is a
recommended method in MongoDB deployments. Replica sets are basically a type of master-
slave replication but they provide automatic failover. A replica set has one master, which is
termed as primary, and multiple slaves, which are termed as secondary in the replica set context;
however, unlike master-slave replication, there’s no one node that is fixed to be primary in the
replica set.f a master goes down in replica set, automatically one of the slave nodes is promoted
to the master. The clients start connecting to the new master, and both data and application will
remain available. In a replica set, this failover happens in an automated fashion. We will explain
the details of how this process happens later. The primary node is selected through an election
mechanism. If the primary goes down, the selected node will be chosen as the primary node.
The points to be noted are
• A replica set is a mongod’s cluster, which replicates among one another and ensuresautomatic
failover.
• In the replica set, one mongod will be the primary member and the others will besecondary
members.
• The primary member is elected by the members of the replica set. All writes aredirected to the
primary member whereas the secondary members replicate from the primary.
• The secondary’s data sets reflect the primary data sets, enabling them to bepromoted to primary
in case of unavailability of the current primary.

Example to Illustrate:

 Standalone Deployment: You have one diary. If it’s lost, everything is gone.
 Master/Slave Replication: You have one main diary and several backup diaries. If the
main one is lost, the backups are safe, but no new notes will be added until you get a new
main diary.
 Replica Set: You have a team of diary keepers. If the main keeper can't keep the diary,
the team immediately appoints a new keeper, so the notes continue without any loss.

Primary and Secondary Members in a Replica Set

The two main types of members in a replica set are Primary Members and Secondary
Members. Let’s explore these concepts in the simplest terms possible.

Primary Member

 What is it?
Think of the primary member as the "boss" of the replica set. It is the only member where
all the writing (inserting, updating, deleting) happens. Any changes made to the database
(like adding a new record) are first done on the primary member.
 How does it get chosen?
The primary member is elected by other members of the replica set through a voting
process. Only one primary can exist at any time.
 Why is it important?
Since all the changes (writes) happen here, it ensures that there is one true version of the
data at any moment. After the primary member receives the changes, it shares (replicates)
these changes with the secondary members.

Example: Imagine a library where the librarian (primary member) writes down all the book
loans. The librarian is the only person who can officially record this information, and then
other librarians (secondary members) copy this information into their records.

Secondary Member

 What is it?
A secondary member is like a backup to the primary member. It holds a copy of all the
data from the primary. If something happens to the primary (like a failure), one of the
secondary members can step up and become the new primary.
 Can it be a Primary?
Yes! If the primary member goes down, a secondary member can be promoted to become
the new primary. This keeps the system running smoothly without interruption.

Example: In our library analogy, if the main librarian is out sick, another librarian
(secondary member) can take over the main duties of recording the book loans.

Types of Secondary Members

Besides the normal secondary members, there are specialized secondary members that play
different roles:

1. Priority 0 Members

 What is it?
A Priority 0 member is a secondary member that holds a copy of the data but can never
become the primary. It is set up specifically to not take on the role of the primary.
 Why is it useful?
 o Cold Standby: It can act as a backup in case something goes wrong but will not
take over as primary.
o Different Hardware: If you have servers with different capabilities, you might not
want a weaker server to become the primary.
o Geographical Distribution: In setups spread across multiple locations, you can
ensure only servers in the main data center become primary, making failovers faster
and more reliable.

Example: Imagine a reserve librarian who has a record of all the loans but never takes on the
full responsibility of recording new loans.

2. Hidden Members

 What is it?
A Hidden Member is a type of Priority 0 member that is invisible to client applications. It
still has all the data, can vote in elections, but doesn’t serve any read requests or traffic
except for keeping up-to-date with replication.
 Why is it useful?
Hidden members are perfect for tasks like generating reports or creating backups without
affecting the primary operations.

Example: Think of a librarian who works behind the scenes, updating records and preparing
reports, but never deals with the public or checks out books.

3. Delayed Members

 What is it?
A Delayed Member is a secondary member that copies the data from the primary, but with
a time delay. This delay helps in cases where you might want to undo a mistake that was
made recently, like accidentally deleting important data.
 Why is it useful?
If an error happens on the primary, like an accidental deletion, you can recover the lost
data from the delayed member, as it hasn’t yet received the deletion operation.

Example: Imagine a librarian who updates their records at the end of each day instead of
instantly. If a mistake is made during the day, you can go to this librarian's record to find the
correct information.

4.Arbiter

Arbiters are mongod instances that are part of a replica set but do not hold data (i.e. do not
provide data redundancy). They can, however, participate in elections. Arbiters have minimal
resource requirements and do not require dedicated hardware.

Election Process in MongoDB Replica Sets

What is an Election in MongoDB?

In a MongoDB replica set, an election is the process where the system decides which server
(node) should act as the primary node. The primary node is responsible for handling all write
operations, while the secondary nodes copy the data from the primary.

Why Do Elections Happen?

Elections are necessary when the current primary node becomes unavailable (for example, if it
crashes). In such cases, the system needs to quickly choose a new primary to ensure that the
database continues to operate smoothly without losing any data.

How the Election Process Works:

1. Majority Voting:
o Suppose you have a replica set with X servers, each with 1 vote.
o For a server to become the primary, it needs more than half of the votes. This means
it needs at least [(X/2) + 1] votes.
o Example: If you have 5 servers, a server needs at least 3 votes (since 5/2 = 2.5, and
2.5 + 1 = 3.5, rounded to 3) to become the primary.
2. Two-Server Problem:
o If you have only two servers (one primary and one secondary), a problem can arise.
o Both servers have 1 vote each, so if the primary server fails, the secondary server
only has 1 vote, which is not enough to become the new primary. This situation
leads to both servers remaining as secondaries, and no primary exists to handle
writes.
o To avoid this, an arbiter is often added. The arbiter doesn't store data but only
helps in voting, giving the system a third vote.
3. Using an Arbiter:
o Let's add an arbiter to the two-server setup, so now you have 3 votes (1 for each
server and 1 for the arbiter).
o If the primary server and arbiter are in one data center and the secondary server is in
another, and the primary fails, the secondary can become the primary because it can
now get the arbiter’s vote, reaching the required majority.

Example - Step-by-Step Election Process:

Let’s break it down with an example of a three-member replica set: A1 (Primary), B1, and C1
(Secondaries).

1. Heartbeat Check:
 o Every few seconds, each server sends a "heartbeat" to the others to check if they are
online and what their status is (primary or secondary).
o A1, as the primary, sends heartbeats to B1 and C1, and they respond with their
status and eligibility to be promoted to primary if needed.
2. Primary Failure:
o If A1 (the current primary) goes down, B1 and C1 will notice that they are not
getting a response from A1.
o B1 and C1 will then consider starting an election to promote one of them to the new
primary.
3. Running for Election:
o B1 might decide to run for primary and will start by checking a few things:
 Is A1 already the primary? (No, it’s down).
 Is B1 eligible to become primary? (Yes, it is).
4. Requesting Votes:
o B1 sends a message to C1, saying, "I want to become primary. Please vote for me."
o C1 will run some checks, like:
 Does B1 have the most recent data?
 Is there any other server that could be primary?
o If everything checks out, C1 will send back a “go-ahead” vote.
5. Final Election Phase:
o If B1 gets the "go-ahead" from C1 and no one vetoes (blocks) the process, B1 will
declare itself the primary.
o B1 then locks the election for 30 seconds to ensure no other election processes start
during that time.
o Finally, B1 becomes the new primary, and C1 becomes the secondary.

Key Points to Remember:

 Majority is crucial: For a server to become primary, it must get the majority of votes.
 Arbiters help with voting: In cases where there are only two servers, an arbiter can help
break ties and avoid deadlock.
 Automatic and Safe: The election process is automatic, and MongoDB ensures that your
data remains safe and consistent even during elections.
Implementing Advanced Clustering with Replica Sets:
Replica sets in MongoDB are a method of ensuring data availability and redundancy by
duplicating the data across multiple servers. This guide will walk you through the basic steps of
setting up, managing, and inspecting a replica set. Each term and action will be explained as
simply as possible.
1. Setting Up a Replica Set
Objective: Get multiple MongoDB servers (nodes) to work together as a team (replica set) to
ensure data is always available, even if one server fails.
Steps:
1. Start the first server:
o Create a folder to store data for this server: C:\db1\active1\data.
o Start MongoDB using the command:
mongod --dbpath C:\db1\active1\data --port 27021 --replSettestset/ANOC9:27021 --rest
o Explanation:
 --dbpath: Location where MongoDB will store its data.
 --port 27021: Port number where this MongoDB instance will listen for
requests.
 --replSettestset/ANOC9:27021: This MongoDB instance is part of a team
named "testset."
 --rest: Enables a web interface for easier monitoring.
2. Start the second server:
o Create a folder for this second server: C:\db1\active2\data.
o Start MongoDB using a similar command as before but on a different port:
mongod --dbpath C:\db1\active2\data --port 27022 --replSettestset/ANOC9:27021 --rest
3. Start the third server (passive):
o Create a folder for the passive server: C:\db1\passive1\data.
o Start MongoDB with:
mongod --dbpath C:\db1\passive1\data --port 27023 --replSettestset/ANOC9:27021 --rest
o Explanation:
 A passive server (low priority) is less likely to become the primary server,
which takes the lead role.
4. Initialize the Replica Set:
o Connect to the first server using the MongoDB shell:
mongo ANOC9 --port 27021
Switch to the admin database:
use admin
 o Create a configuration for the replica set:
cfg = {
_id: 'testset',
members: [
{_id: 0, host: 'ANOC9:27021'},
{_id: 1, host: 'ANOC9:27022'},
{_id: 2, host: 'ANOC9:27023', priority: 0}
]
}
o Explanation:
 _id: Unique ID for each server in the set.
 host: The server's address.
 priority: 0: This server is passive.
o Initialize the set:
rs.initiate(cfg)
5. Check the status:
o Ensure everything is working correctly:
rs.status()
2. Removing a Server
Objective: Safely remove a server from the replica set.
Steps:
1. Connect to the server you want to remove:
mongo ANOC9 --port 27022
2. Shut down the server:

db.shutdownServer()
3. Connect to the primary server and remove the server from the set:
rs.remove("ANOC9:27022")
4. Verify the removal with:
rs.status()
3. Adding a Server
Objective: Add a new server to the replica set.
Steps:
1. Create a data folder for the new server: C:\db1\active3\data.
2. Start the MongoDB instance:
mongod --dbpath C:\db1\active3\data --port 27024 --replSettestset/ANOC9:27021 --rest
3. Connect to the primary server and add the new server:
rs.add("ANOC9:27024")
4. Verify the addition with:
rs.status()
4. Adding an Arbiter
Objective: Add an arbiter, a special server that only helps with elections but doesn't store data.
Steps:
1. Create a data folder for the arbiter: C:\db1\arbiter\data.
2. Start the MongoDB instance:
mongod --dbpath c:\db1\arbiter\data --port 30000 --replSettestset/ANOC9:27021 --rest
3. Connect to the primary server and add the arbiter:
rs.addArb("ANOC9:30000")
4. Verify with:
rs.status()
5. Inspecting the Status
Objective: Check the health and roles of each server in the replica set.
Steps:
 Simply use:
rs.status()
 Explanation: This command shows details like which server is primary, which is
secondary, and if everything is functioning well.
6. Forcing a New Election of a Primary
Objective: Force a new leader to be chosen among the servers.
Steps:
 On the primary server, run:
rs.stepDown()
  Explanation: This is useful for testing or maintenance when you want another server to
temporarily take over.
7. Using the Web Interface to Inspect the Status
Objective: View the status of the replica set through a web browser.
Steps:
 Access the web interface by opening:
http://localhost:28021
 Explanation: This address is based on the primary server's port number (27021 + 1000 =
28021).
Sharding in MongoDB
Sharding is a powerful feature in MongoDB that helps manage large amounts of data efficiently
by spreading it across multiple servers. This process not only ensures that data can be stored and
retrieved quickly, but it also helps in handling the increased workload as the data grows. Let’s
break down the concept of sharding into simple terms and examples to make it easier to
understand.

Imagine you have a huge library of books, but instead of keeping all the books in one giant
bookshelf, you split them into different shelves based on some criteria, like the author's name or
the genre. Each shelf holds a part of the library's collection, making it easier to find and manage
the books.In MongoDB, sharding is like that. Instead of storing all the data on one server, the
data is split across multiple servers (called "shards"). This helps MongoDB handle larger
datasets and more traffic.

Why Sharding is Needed

Imagine you have a single computer storing all the data for your application. As your data
grows, the computer starts to struggle—its memory is full, the disk is nearly out of space, and
reading data becomes slower. Just like how you might need more bookshelves when your book
collection grows, your database needs more servers to handle more data. This is where sharding
comes in.
In MongoDB, sharding means splitting your data across multiple servers (called shards) so that
each server only has to manage a portion of the data. This process is known as horizontal
scaling.

Components of Sharding
When you set up sharding in MongoDB, three key components come into play:
1. Shards:
These are the individual servers where data is stored. Each shard holds a part of the entire
dataset. For example, if you have 1TB of data and 4 shards, each shard might hold about
250GB of data.
 2. Mongos:
This is the “traffic controller” of the system. It routes client requests (for reading or
writing data) to the correct shard. Clients don’t need to know where the data is stored;
they just interact with mongos, and it handles everything.
3. Config Servers:
These servers store metadata, which is the information about where data is located in the
shards. They help mongos route the requests properly. For a stable system, it's
recommended to have at least three config servers.
How Sharding Works
Let’s take an example to see how sharding actually works.
Example:
Imagine you’re running a blogging platform with millions of posts. You decide to split (shard)
the data based on the date each post was published. Here’s how it might look:
 Shard 1: Stores posts from the beginning of time to July 2009.
 Shard 2: Stores posts from August 2009 to December 2009.
 Shard 3: Stores posts from January 2010 to the present.
Now, if a user wants to read blog posts from 2010 onwards:
1. The Client Queries Mongos: The user sends a request to view posts from 2010.
2. Mongos Knows Where to Look: Mongos knows that all posts from 2010 are on Shard 3,
so it sends the request there.
3. Shard 3 Retrieves the Data: Shard 3 finds the relevant posts and sends them back to
mongos.
4. Mongos Combines and Returns Data: Mongos takes the data from Shard 3, combines it
(if needed), and sends it back to the client.
Types of Sharding
MongoDB uses two main ways to decide how to split your data:
1. Range-Based Sharding:
Data is split based on a range of values. For example, in the blog posts scenario, the data
is split by date. So, all posts from a certain time period are stored together on one shard.
2. Hash-Based Sharding:
Instead of using ranges, data is split based on the hashed value of the shard key. This
ensures that the data is evenly distributed across shards.
When to Use Sharding
Sharding isn’t necessary for every database, but it becomes essential when:
 The Dataset is Huge: Your data is too big to fit on one server.
  Memory Limits are Reached: MongoDB uses memory for quick data access. When the
active data (working set) is too large to fit in memory, sharding helps by distributing the
load.
 Write-Intensive Applications: If your application is constantly writing new data,
sharding can spread these writes across multiple servers, preventing any single server
from getting overwhelmed.
Challenges with Sharding
While sharding can improve performance and scalability, it also adds complexity to your
deployment. For example, you need to manage multiple servers, and there's a need for careful
planning on how data is distributed (using shard keys).
Data Distribution Process
Shard Key
The shard key is like the rule that decides which shelf (or shard) a book (or document) goes on.
You choose a specific field in your data (like a date or an ID) to be the shard key. MongoDB
then uses this key to determine how to distribute the data across the shards.
How Data is Distributed
Data in MongoDB is distributed using one of two methods:
1. Range-Based Partitioning:
o Think of this as arranging books in order of publication dates. If you pick a date as
the shard key, MongoDB will divide your data into ranges based on this date.
o For example, if your data ranges from 2000 to 2024, one shard might store data
from 2000 to 2009, another from 2010 to 2014, and so on.
o The advantage: It's easy to find data within a certain range (like all data from 2012).
o The downside: If most of the data is clustered around certain dates (like 2023), one
shard might become overloaded, while others remain underused.
2. Hash-Based Partitioning:
o Instead of arranging books by date, imagine scattering them randomly across the
shelves. MongoDB uses a hash function to do this, spreading the data more evenly
across shards.
o The advantage: Even distribution, so no single shard gets overloaded.
o The downside: It's harder to find data within a specific range.
Chunks
MongoDB further divides the data into smaller pieces called "chunks," which are about 64MB in
size. These chunks can move between shards to keep everything balanced.
Example: Blog Posts
Let's say you have a collection of blog posts with a date field, and you decide to shard the data
based on the date.
 Shard 1: Holds data from the beginning of time up to July 2009.
 Shard 2: Holds data from August 2009 to December 2009.
 Shard 3: Holds data from January 2010 to the present.
When you query for blog posts from 2023, MongoDB knows to only look in Shard 3, speeding
up the process.
The Role of Config Servers
Config servers are like the brains behind sharding. They keep track of where all the data is stored
and help move data around when needed.
For example, if Shard 2 becomes too full, the config servers will split the data and move some of
it to another shard. They also tell the system about new shards and where to store data.
Data Balancing Process
In a sharded cluster, MongoDB needs to maintain a balanced distribution of data across all the
shards to ensure optimal performance. However, as new data is added or existing data is
modified, or even when servers are added or removed, this balance can be disrupted. MongoDB
uses specific background processes to maintain this balance:
1. Chunk Splitting
Chunk splitting is essential for maintaining the specified chunk size within each shard. When
documents are added or updated, and a chunk exceeds the default size of 64MB (configurable),
MongoDB automatically splits this chunk into smaller chunks. This ensures that no single chunk
becomes too large, which could lead to performance issues.
Process Overview:
 Shard keys are used to determine how documents are distributed across shards.
 Documents are grouped into chunks (default size is 64MB).
 If a chunk exceeds its size limit due to data insertion or updates, it is split into smaller
chunks.
 This splitting does not involve moving data between shards but may lead to an imbalance
where one shard has more chunks than another.
2. Balancer
The balancer is a critical background process that ensures an even distribution of chunks across
all shards in the cluster. When chunks become unevenly distributed due to operations like chunk
splitting, document insertion, or shard addition/removal, the balancer steps in to redistribute the
chunks.
Key Functions:
 Balancer Activation: Initiated by any mongos within the cluster, the balancer acquires a
lock on the config database to manage chunk migrations.
  Migration Process:
1. Initiation: A moveChunk command is sent to the source shard.
2. Data Copying: The source shard begins copying the chunk's documents and queues
them for transfer.
3. Receiving: The destination shard starts receiving the copied data.
4. Synchronization: Ensures all changes during migration are updated on the
destination shard.
5. Metadata Update: The destination shard updates the chunk's location in the config
database.
6. Cleanup: The source shard deletes the chunk's data after successful migration.
Impact and Configuration:
 Balancer activities can affect database performance, so they can be configured to:
1. Start Migration Based on Threshold: Migration begins when the difference in the
number of chunks between the most and least loaded shards reaches a certain
threshold
2. Scheduled Operations: The balancer can be scheduled to run during low-traffic
periods to minimize impact.
 Error Handling: If an error occurs during migration, the balancer aborts the process,
leaving the chunk on the original shard.
Operations in a Sharded Cluster
When performing read and write operations in a sharded cluster, MongoDB uses the config
servers to maintain metadata, which is crucial for routing requests.
 Cached Metadata: mongos instances cache metadata to efficiently route operations
without burdening the config servers.
 Shard Key Operations:
o If a query or operation uses the shard key, mongos can target specific shards,
optimizing the operation.
o Without a shard key, the operation is broadcast to all shards.
 Post-Processing: For certain operations like sorting or aggregation, mongos may need to
combine and process data from multiple shards before returning the final result to the
client.
Implementing Sharding
To implement sharding in MongoDB, follow these steps:
1. Set Up Configuration Server:
o Create a data directory for the config server and start mongod with the --configsvr
option.
 2. Start mongos Controller:
o Use the mongos command with the --configdb option to point to the config server
and set a chunk size.
3. Set Up Shard Servers:
o Start mongod instances for each shard with the --shardsvr option.
4. Add Shards to the Cluster:
o Use the mongo shell to connect to mongos and add shard servers with the
db.runCommand({addshard: "localhost:PORT"}) command.
5. Enable Sharding for a Database and Collection:
o Enable sharding on a database and specify the shard key for a collection using
db.runCommand.
6. Import Data:
o Use the mongoimport command to load data into the sharded collection.
7. Check Data Distribution:
o Use the count() command in each shard's console to verify how data is distributed.
Example Sharding Cluster Setup
 Configuration:
o Component: mongos, mongod (Config Server, Shard0, Shard1)
o Ports: 27021 (mongos), 27022 (Config Server), 27023 (Shard0), 27024 (Shard1)
 Process:
o Create data directories and start the mongod instances.
o Add shards using the mongo shell.
o Enable sharding on the testdb database and testcollection collection.
o Import data and verify distribution.
Controlling Collection Distribution with Tag-Based Sharding
In MongoDB, sharding is a method of distributing data across multiple servers. This ensures
scalability and high availability. However, you may want to have more control over where
specific data is stored within a sharded cluster. This is where Tag-Based Sharding comes into
play. This feature, introduced in version 2.2.0, allows you to dictate which collections go to
which shard, providing you with fine-grained control over data distribution.
Setting Up a Sharded Cluster for Tag-Based Sharding
Before diving into tag-based sharding, ensure that your sharded cluster is up and running. You'll
need a config server, a mongos instance, and three shard servers. If you've been following along,
you already have a cluster set up with two shards. For this example, we'll add Shard2 back into
the cluster to have three shards.
Prerequisite: Starting the Cluster
Let's ensure that the necessary components are running:
1. Start the Config Server:

C:\> mkdir C:\db1\config\data

C:\> cd c:\practicalmongodb\bin
C:\practicalmongodb\bin> mongod --port 27022 --dbpath C:\db\config\data --configsvr
Start the Mongos:

C:\> cd c:\practicalmongodb\bin
c:\practicalmongodb\bin> mongos --configdb localhost:27022 --port 27021
Start the Shard Servers:

 Shard0:

C:\> cd c:\practicalmongodb\bin
c:\practicalmongodb\bin> mongod --port 27023 --dbpath c:\db1\shard0\data --shardsvr
Shard1:

C:\> cd c:\practicalmongodb\bin
C:\practicalmongodb\bin> mongod --port 27024 --dbpath c:\db1\shard1\data --shardsvr
Shard2:

C:\> cd c:\practicalmongodb\bin
c:\practicalmongodb\bin> mongod --port 27025 --dbpath c:\db1\shard2\data --shardsvr

Since Shard2 was removed earlier, you'll need to add it back to the cluster:

Adding Shard2 to the Cluster

1. Connect to Mongos:

C:\> cd c:\practicalmongodb\bin
c:\practicalmongodb\bin> mongo localhost:27021
Once connected, you should see a prompt similar to this:

MongoDB shell version: 3.0.4

connecting to: localhost:27021/test
Drop the Existing Database:
mongos> use testdb

switched to db testdb

mongos> db.dropDatabase()
{ "dropped" : "testdb", "ok" : 1 }
Add Shard2:

mongos> use admin

switched to db admin

mongos> db.runCommand({addshard: "localhost:27025", allowlocal: true})

{ "shardAdded" : "shard0002", "ok" : 1 }
Verify the Shards:
mongos> db.runCommand({listshards:1})
You should see something like this:

"shards" : [

{ "_id" : "shard0000", "host" : "localhost:27023" },

{ "_id" : "shard0001", "host" : "localhost:27024" },

{ "_id" : "shard0002", "host" : "localhost:27025" }

],

"ok" : 1
}
Enabling Sharding and Tagging Collections
Now that your sharded cluster is up, let's enable sharding for specific collections and assign them
to different shards based on tags.
1. Connect to Mongos at Port 30999:

C:\ > cd c:\practicalmongodb\bin

c:\practicalmongodb\bin> mongos --port 30999 --configdb localhost:27022

production.
Connect to Mongos and Enable Sharding:
C:\ > cd c:\practicalmongodb\bin

c:\practicalmongodb\bin> mongo localhost:27021

mongos> conn = new Mongo("localhost:30999")

mongos> db = conn.getDB("movies")

mongos> sh.enableSharding("movies")
{ "ok" : 1 }
Shard the Collections:
 movies.drama by originality:

mongos> sh.shardCollection("movies.drama", {originality:1})

{ "collectionsharded" : "movies.drama", "ok" : 1 }
movies.action by distribution:

mongos> sh.shardCollection("movies.action", {distribution:1})

{ "collectionsharded" : "movies.action", "ok" : 1 }
movies.comedy by collections:

mongos> sh.shardCollection("movies.comedy", {collections:1})

{ "collectionsharded" : "movies.comedy", "ok" : 1 }
Insert Data into Collections:

mongos> for(var i=0; i<100000; i++) { db.drama.insert({originality:Math.random(), count:i,

time:new Date()}); }

mongos> for(var i=0; i<100000; i++) { db.action.insert({distribution:Math.random(), count:i,

time:new Date()}); }
mongos> for(var i=0; i<100000; i++) { db.comedy.insert({collections:Math.random(), count:i,
time:new Date()}); }
Applying Tags to Control Collection Distribution
To ensure that each collection is stored on a specific shard, you can apply tags:
1. Assign Tags to Shards:

mongos> sh.addShardTag("shard0000", "dramas")

mongos> sh.addShardTag("shard0001", "actions")

mongos> sh.addShardTag("shard0002", "comedies")
Create Tagging Rules for Collections:

 Rule 1: Tag all chunks of movies.drama as "dramas":

mongos>sh.addTagRange("movies.drama", {originality:MinKey}, {originality:MaxKey},

"dramas")

Rule 2: Tag all chunks of movies.action as "actions":

mongos>sh.addTagRange("movies.action",{distribution:MinKey},{distribution:MaxKey}
, "actions")

Rule 3: Tag all chunks of movies.comedy as "comedies":

mongos>sh.addTagRange("movies.comedy",{collections:MinKey},{collections:MaxKey
}, "comedies")
Verifying Data Distribution

After applying the tags and rules, allow the cluster some time to rebalance the chunks
automatically based on your configuration.

To verify the distribution:

mongos> use config
switched to db config
mongos> db.chunks.find({ns:"movies.drama"}, {shard:1, _id:0}).sort({shard:1})

This command will display how the chunks are distributed across the shards.
Scaling with Tagging

Sometimes, you might need to reallocate data across shards for scaling purposes. For example, if
movies.action grows rapidly, you might want to distribute its chunks across two shards.

1. Move movies.comedy chunks to Shard0:

mongos> sh.addShardTag("shard0000","comedies")
mongos> sh.removeShardTag("shard0002","comedies")

2. Distribute movies.action across Shard1 and Shard2:

3. mongos> sh.addShardTag("shard0002","actions")

Finally, you can re-issue the find command after some time to see the updated chunk
distribution.
With tag-based sharding, you can effectively control how data is distributed across your sharded
cluster, ensuring optimal performance and storage management.

Production Cluster Architecture Overview

Think of a MongoDB production cluster like a global social network—Facebook, for
example. You want to connect users from different parts of the world who can share posts,
photos, and comments. To ensure everything runs smoothly and data is always available,
even if something goes wrong, you need to set up the system smartly. So the application’s
production cluster should have the following components :
1. At least two mongos instance, but you can have more as per need.
2. Three config servers, each on a separate system.
3. Two or more replica sets serving as shards. The replica sets are distributed across
geographies with read concern set to nearest.

Key Components of the Architecture:

1. Mongos instances: These act like traffic directors. They guide the data to where it needs
to go in the cluster.
2. Config Servers: These are like the maps or blueprints of your cluster. They keep track of
how your data is spread out.
3. Replica Sets serving as Shards: These are the actual data storage units. The data is
spread across them, and they are located in different geographical regions to ensure
availability.

Failure Scenarios & Their Impact

Now, let’s go through each failure scenario with a simple example.

Scenario 1: Mongos Becomes Unavailable.

Analogy: Imagine you're at a big airport, and there’s a central information desk (Mongos) that
tells you which gate to go to for your flight. The desk doesn’t actually have your luggage (data);
it just gives you directions.

What Happens: If the information desk temporarily closes (Mongos goes down), no one loses
their luggage because it’s still safely stored at the gates (shards). Once the desk reopens (Mongos
restarts), it syncs up with the airport’s systems (config servers) and can once again provide
directions to the passengers.

Impact: Your app might briefly lose its ability to navigate the data, but once Mongos is back
online, everything resumes normally, with no loss of data.

The application server where mongos has gone down will not be able to communicate with the
cluster but it will not lead to any data loss since the mongos don’t maintain any data of its own.
The mongos can restart, and while restarting, it can sync up with the config servers to cache
the cluster metadata, and the application can normally start its operations.
Scenario 2: One of the Replica Set Members in a Shard Becomes Unavailable

 Analogy: Think of a team of three backup dancers (replica set members) in a dance
performance. If one dancer can’t perform (one replica member is down), the others will
still continue the show (primary node switches if necessary).
 What Happens: If the main dancer (primary node) is unavailable, the team will quickly
assign another dancer to take the lead (another node becomes primary). If it's just one of
the backup dancers (secondary node), the show still goes on, but with one less backup.
 Impact: No data loss occurs because another node steps in. However, your redundancy is
reduced until the failed dancer (node) recovers or is replaced.

Since you used replica sets to provide high availability, there is no data loss. If a primary
node is down, a new primary is chosen, whereas if it’s a secondary node, then it is
disconnected and the functioning continues normally.

Scenario 3: One of the Shards Becomes Unavailable

 Analogy: Picture a library where books are stored in multiple rooms (shards). If one room
is temporarily locked (shard down), the readers can still access books from the other
rooms.
 What Happens: The data in the unavailable shard can’t be accessed, but users can still
interact with the data in other shards. If your query requires data from the locked room
(down shard), you might get incomplete results.
 Impact: The application remains operational, but you may experience missing or partial
data. You need to bring the shard back online as quickly as possible.

In this scenario, the data on the shard will be unavailable, but the other shards will be
available, so it won’t stop the application. The application can continue with its read/ write
operations; however, the partial results must be dealt with within the application. In
parallel, the shard should attempt to recover as soon as possible.

Scenario 4: Only One Config Server is Available

 Analogy: Imagine a blueprint for a large building is kept in three vaults (config servers). If
only one vault remains accessible, you can still view the building’s current structure (read
operations), but you can’t make changes to it (write operations) until all vaults are
available.
 What Happens: The cluster becomes read-only. You can still access data but can't
perform actions that would change the structure, such as moving data between shards
or splitting chunks.
 Impact: This is a risky situation because if all three config servers fail, the cluster
becomes completely unusable. It's crucial to replace the failed config servers as soon as
possible.
 In this scenario, although the cluster will become read only, it will not serve any operations
that might lead to changes in the cluster structure, thereby leading to a change of metadata
such as chunk migration or chunk splitting. The config servers should be replaced ASAP
because if all config servers become unavailable, this will lead to an inoperable cluster.

Putting it All Together

In a production MongoDB cluster, the system is designed with high availability in mind. Even
if one component fails, the rest of the system continues to function, allowing your application to
keep running. However, you need to quickly recover from these failures to maintain redundancy
and ensure the cluster remains fully operational.