Sharding in MongoDB

Sharding is a method for distributing data across multiple machines. MongoDB uses sharding to
support deployments with very large data sets and high throughput operations.
Database systems with large data sets or high throughput applications can challenge the capacity
of a single server. For example, high query rates can exhaust the CPU capacity of the server.
Working set sizes larger than the system's RAM stress the I/O capacity of disk drives.
There are two methods for addressing system growth: vertical and horizontal scaling.
Vertical Scaling involves increasing the capacity of a single server, such as using a more
powerful CPU, adding more RAM, or increasing the amount of storage space. Limitations in
available technology may restrict a single machine from being sufficiently powerful for a given
workload. Additionally, Cloud-based providers have hard ceilings based on available hardware
configurations. As a result, there is a practical maximum for vertical scaling.
Horizontal Scaling involves dividing the system dataset and load over multiple servers, adding
additional servers to increase capacity as required. While the overall speed or capacity of a single
machine may not be high, each machine handles a subset of the overall workload, potentially
providing better efficiency than a single high-speed high-capacity server. Expanding the capacity
of the deployment only requires adding additional servers as needed, which can be a lower
overall cost than high-end hardware for a single machine. The trade off is increased complexity
in infrastructure and maintenance for the deployment.
MongoDB supports horizontal scaling through sharding.

Sharded Cluster
A MongoDB sharded cluster consists of the following components:
 shard: Each shard contains a subset of the sharded data. Each shard can be deployed as
a replica set.
 mongos: The mongos acts as a query router, providing an interface between client
applications and the sharded cluster. Starting in MongoDB 4.4, mongos can
support hedged reads to minimize latencies.
 config servers: Config servers store metadata and configuration settings for the cluster.
The following graphic describes the interaction of components within a sharded cluster:

MongoDB shards data at the collection level, distributing the collection data across the shards in
the cluster.

Shard Keys
MongoDB uses the shard key to distribute the collection's documents across shards. The shard
key consists of a field or multiple fields in the documents.
 Starting in version 4.4, documents in sharded collections can be missing the shard key
fields. Missing shard key fields are treated as having null values when distributing the
documents across shards but not when routing queries. For more information,
see Missing Shard Key Fields.
 In version 4.2 and earlier, shard key fields must exist in every document for a sharded
collection.
You select the shard key when sharding a collection.
 Starting in MongoDB 5.0, you can reshard a collection by changing a collection's shard
key.
 Starting in MongoDB 4.4, you can refine a shard key by adding a suffix field or fields to
the existing shard key.
 In MongoDB 4.2 and earlier, the choice of shard key cannot be changed after sharding.
A document's shard key value determines its distribution across the shards.
 Starting in MongoDB 4.2, you can update a document's shard key value unless your
shard key field is the immutable _id field. See Change a Document's Shard Key
Value for more information.
 In MongoDB 4.0 and earlier, a document's shard key field value is immutable.

Shard Key Index

To shard a populated collection, the collection must have an index that starts with the shard key.
When sharding an empty collection, MongoDB creates the supporting index if the collection
does not already have an appropriate index for the specified shard key. See Shard Key Indexes.

Shard Key Strategy

The choice of shard key affects the performance, efficiency, and scalability of a sharded cluster.
A cluster with the best possible hardware and infrastructure can be bottlenecked by the choice of
shard key. The choice of shard key and its backing index can also affect the sharding
strategy that your cluster can use.
---------------------------------------------------------

MongoDB: Chunks
MongoDB partitions sharded data into chunks. Each chunk has an inclusive lower and exclusive
upper range based on the shard key.

Chunk Splits
Splitting is a process that keeps chunks from growing too large. When a chunk grows beyond
a specified chunk size, or if the number of documents in the chunk exceeds Maximum Number
of Documents Per Chunk to Migrate, MongoDB splits the chunk based on the shard key values
the chunk represent. A chunk may be split into multiple chunks where necessary. Inserts and
updates may trigger splits. Splits are an efficient meta-data change. To create splits, MongoDB
does not migrate any data or affect the shards.

Splits may lead to an uneven distribution of the chunks for a collection across the shards. In such
cases, the balancer redistributes chunks across shards. See Cluster Balancer for more details on
balancing chunks across shards.
Chunk Migration
MongoDB migrates chunks in a sharded cluster to distribute the chunks of a sharded collection
evenly among shards. Migrations may be either:
 Manual. Only use manual migration in limited cases, such as to distribute data during
bulk inserts. See Migrating Chunks Manually for more details.
 Automatic. The balancer process automatically migrates chunks when there is an uneven
distribution of a sharded collection's chunks across the shards. See Migration
Thresholds for more details.

Balancing
The balancer is a background process that manages chunk migrations. If the difference in
number of chunks between the largest and smallest shard exceed the migration thresholds, the
balancer begins migrating chunks across the cluster to ensure an even distribution of data.
You can manage certain aspects of the balancer. The balancer also respects any zones created as
a part of configuring zones in a sharded cluster.

Balancer and Even Chunk Distribution

In an attempt to achieve an even distribution of chunks across all shards in the cluster,
a balancer runs in the background to migrate chunks across the shards .