Architecture

#Documents
MongoDB stores data records as BSON documents. BSON is a binary representation of JSON documents,
though it contains more data types than JSON. For the BSON spec, see bsonspec.org. See also BSON Types.
A MongoDB document.

#Databases
In MongoDB, databases hold one or more collections of documents. To select a database to use, in
mongosh, issue the use <db> statement, as in the following example:
use myDB
Create a Database
If a database does not exist, MongoDB creates the database when you first store data for that database. As
such, you can switch to a non-existent database and perform the following operation in mongosh:
use myNewDB
db.myNewCollection1.insertOne( { x: 1 } )
The insertOne() operation creates both the database myNewDB and the collection myNewCollection1 if
they do not already exist. Be sure that both the database and collection names follow MongoDB Naming
Restrictions.
#Collections
MongoDB stores documents in collections. Collections are analogous to tables in relational databases.

Create a Collection
If a collection does not exist, MongoDB creates the collection when you first store data for that collection.
db. myNewCollection2.insertOne({x: 1})
db. myNewCollection3.createIndex({y: 1})
Both the insertOne() and the createIndex() operations create their respective collection if they do not already
exist. Be sure that the collection name follows MongoDB Naming Restrictions.
#What is JSON?
JSON, or JavaScript Object Notation, is a human-readable data interchange format, specified in the early
2000s. Even though JSON is based on a subset of the JavaScript programming language standard, it’s
completely language-independent.
JSON objects are associative containers, wherein a string key is mapped to a value (which can be a number,
string, boolean, array, an empty value — null, or even another object). Almost any programming language
has an implementation for this abstract data structure — objects in JavaScript, dictionaries in Python, hash
tables in Java and C#, associative arrays in C++, and so on. JSON objects are easy for humans to understand
and for machines to parse and generate:

#What is BSON?
BSON stands for “Binary JSON,” and that’s exactly what it was invented to be. BSON’s binary structure
encodes type and length information, which allows it to be traversed much more quickly compared to JSON.
BSON adds some non-JSON-native data types, like dates and binary data, without which MongoDB would
have been missing some valuable support.
The following are some example JSON objects and their corresponding BSON representations.
#Storage Engines
In MongoDB, storage engines are responsible for managing how data is stored, accessed, and managed on
disk. MongoDB has evolved over the years with different storage engines, but currently, the primary and
default storage engine is WiredTiger. Here’s an overview:
WiredTiger
 Concurrency Control: Supports fine-grained concurrency control at the document-level. This
means multiple operations can access and modify different documents concurrently.
 Compression: Includes built-in support for data compression, which reduces storage requirements
and can improve read and write performance.
 Logging: Uses a modern logging mechanism for crash recovery and to ensure data durability in case
of system failures.
 Suitability: WiredTiger is the default storage engine in MongoDB since version 3.2. It is designed to
handle a wide range of workloads efficiently, including large-scale deployments and applications
with high concurrency requirements.
MMAPv1 (Memory-Mapped Architecture Version 1)
 Concurrency Control: Uses a global reader-writer lock at the database level. This means that while
multiple read operations can occur simultaneously, write operations may block other operations.
 Architecture: Relies on memory-mapped files where data files are directly mapped into virtual
memory. This architecture can be less efficient for large datasets or write-heavy workloads.
 Suitability: MMAPv1 was the default storage engine in MongoDB prior to version 3.0. It was
deprecated in MongoDB 4.0 and is no longer recommended for new deployments. It was suitable for
simpler applications with less demanding concurrency requirements.
#Journaling
Journaling in MongoDB refers to the process by which changes to the database are recorded in a journal or
log before they are applied to the actual data files. This mechanism ensures durability and consistency in
case of unexpected shutdowns or failures.
Here’s how journaling works in MongoDB:
1. Purpose of Journaling: MongoDB uses journaling to ensure that write operations are durable even in the
event of a crash. When a write operation is performed, it is first written to the journal files on disk. Once the
write operation is confirmed in the journal, it is then applied to the database data files.
2. Journal Files: MongoDB maintains a set of journal files (named `j._journal` by default) in its data
directory. These files store write operations in a format that allows MongoDB to quickly recover any
uncommitted changes in case of a crash.
3. Journaling Process:
- Write Operation: When a write operation (insert, update, delete) is performed, MongoDB writes the
operation to the journal file on disk.
- Confirmation: Once the write is confirmed in the journal file, MongoDB applies the operation to the data
files.
- Commit: Once the operation is applied to the data files, MongoDB marks the write operation as
committed in the journal.
4. Recovery: In case of a crash or unexpected shutdown, MongoDB can replay the journal files to recover
any write operations that were not yet applied to the data files. This ensures that the database can recover to
a consistent state without losing data.
5. Configuration: Journaling is enabled by default in MongoDB, starting from version 2.0. You can
configure the journaling options in the MongoDB configuration file (`mongod.conf`). The main
configuration options related to journaling include `journal`, which enables or disables journaling, and
`journalCommitInterval`, which specifies how often MongoDB syncs journal files to disk.
6. Performance Considerations: While journaling ensures data durability, it does have a slight
performance overhead due to the additional disk writes. MongoDB provides options to tune journaling
performance based on your application's requirements.

#Write Path
In MongoDB, the write path refers to the sequence of steps and processes involved when a write operation
(insert, update, delete) is performed on the database. Understanding the write path is crucial for optimizing
MongoDB performance and ensuring data consistency and durability. Here's a detailed overview of the write
path in MongoDB:
Write Path Steps:
1. Client Application:
- The write operation originates from a client application that interacts with the MongoDB server using
one of the MongoDB drivers or through the MongoDB shell.
2. Query Parsing and Optimization:
- When MongoDB receives a write operation, it parses the query to understand the intent of the operation
and optimizes it if possible. This involves checking indexes and determining the most efficient way to
perform the operation.
3. Document Validation:
- If document validation rules are defined (e.g., via schema validation or validation rules in MongoDB
3.2+), MongoDB validates the incoming document against these rules to ensure data integrity and
consistency.
4. Write Operation Execution:
- Once validated, MongoDB executes the write operation. Depending on the type of operation (insert,
update, delete), different actions are taken:
- Insert: If inserting a new document, MongoDB checks if the document already exists and then writes it
to the collection.
- Update: For updates, MongoDB may update existing documents or upsert (update or insert if not exists)
based on the query criteria.
- Delete: Deletes documents matching the specified query criteria.
5. Journaling (if enabled):
- If journaling is enabled (which is the default in MongoDB), the write operation is recorded in the journal
files before it is applied to the data files on disk. This ensures durability and recoverability in case of a crash.
6. Data File Write (on Disk):
- After journaling (if enabled), MongoDB writes the updated data to the data files (e.g., WiredTiger
storage engine files) on disk. This step ensures that the data is persistent and can be retrieved even after
server restarts.
7. Memory (RAM) Updates:
- MongoDB also updates its in-memory data structures (e.g., indexes, working set) to reflect the changes
made to the data files. This improves read performance by maintaining frequently accessed data in memory.
8. Response to Client:
- Once the write operation is successfully completed and committed to disk (and optionally to journal),
MongoDB sends an acknowledgment (ACK) back to the client application, confirming the operation's
success.
Optimization and Considerations:
- Write Concern: MongoDB provides options to specify the level of acknowledgment required from the
server for write operations (e.g., acknowledgments from a single server, a majority of servers, or all servers).
- Indexes: Properly designed indexes can significantly improve the performance of write operations by
reducing the time it takes to locate and modify documents.

- Journaling Configuration: Tuning journaling settings (e.g., `journalCommitInterval`, `commitIntervalMs`)

can impact write performance and durability trade-offs based on application requirements.
- Concurrency Control: MongoDB uses concurrency control mechanisms (e.g., multi-version concurrency
control - MVCC) to handle concurrent write operations safely and maintain data consistency.
Capped Collections
Capped collections are fixed-size collections that insert and retrieve documents based on insertion order.
Capped collections work similarly to circular buffers: once a collection fills its allocated space, it makes
room for new documents by overwriting the oldest documents in the collection.
Restrictions
Capped collections cannot be sharded.
You cannot create capped collections on serverless instances.
Capped collections are not supported in Stable API V1.
You cannot write to capped collections in transactions.
The $out aggregation pipeline stage cannot write results to a capped collection.
You cannot use read concern "snapshot" when reading from a capped collection.
Command Syntax
The following example creates a capped collection called log with a maximum size in bytes.
db.createCollection( "log", { capped: true, size: number, max: number } )
#Oplog collection
In MongoDB, an oplog (short for operations log) is a special capped collection that keeps a rolling record of
all write operations that modify data within a MongoDB replica set. It is crucial for replication and allows
secondary nodes to replicate changes from the primary node.
Key Points about the Oplog Collection:
1. Purpose: The primary use of the oplog is to provide a mechanism for replicating data across MongoDB
replica sets. It records all write operations (inserts, updates, deletes) in the order they occur on the primary
node.
2. Structure:
- The oplog is implemented as a capped collection. Capped collections in MongoDB are fixed-size
collections that automatically remove the oldest documents to make room for new ones when they reach
their maximum size.
- By default, the oplog is named `oplog.rs` and resides in the `local` database of each MongoDB node.
3. Size and Configuration:
- The size of the oplog is configurable during the setup of a replica set with the `replSet` configuration
option `oplogSizeMB`. This determines how much data the oplog can hold before it starts overwriting the
oldest entries.
- The default size of the oplog is 5% of free disk space, with a minimum of 990MB and a maximum of
50GB.
4. Usage:
- MongoDB's replication process relies on the oplog to ensure that all write operations on the primary are
replicated to the secondary nodes.
- When a secondary node starts up or falls behind, it uses the oplog to catch up to the current state of the
primary by applying all the operations from the oplog that it has not yet processed.
5. Access and Monitoring:
- Direct access to the oplog collection is not recommended for typical application use because its internal
structure and format may change between MongoDB versions.
- MongoDB provides commands (`rs.printReplicationInfo()`, `rs.status()`) and utilities (`mongodump`,
`mongorestore`) to monitor and manage replication, which indirectly utilize the oplog.
6. Backup and Maintenance:
- Backing up the oplog is not typically necessary because it's designed for replication rather than data
recovery.
- Routine maintenance of the oplog involves ensuring sufficient space (`oplogSizeMB` configuration) and
monitoring its usage to prevent overruns.
#TTL Indexes
TTL indexes are special single-field indexes that MongoDB can use to automatically remove documents
from a collection after a certain amount of time or at a specific clock time. Data expiration is useful for
certain types of information like machine generated event data, logs, and session information that only need
to persist in a database for a finite amount of time.
Create a TTL Index
To create a TTL index, use createIndex(). Specify an index field that is either a date type or an array that
contains date type values. Use the expireAfterSeconds option to specify a TTL value in seconds.
The TTL index expireAfterSeconds value must be within 0 and 2147483647 inclusive.
For example, to create a TTL index on the lastModifiedDate field of the eventlog collection with a TTL
value of 3600 seconds, use the following operation in mongosh:
db.eventlog.createIndex( { "lastModifiedDate": 1 }, {
expireAfterSeconds: 3600 } )
Starting in MongoDB 6.3, you can create partial TTL indexes on time series collections. These indexes use
the collection timeField as the key field, and require a partial filter expression on the metaField.
Time series collections include an optional expireAfterSeconds field. If you do not set it, a TTL index with a
partialFilterExpression lets you set an expiration period for documents that match the filter. If you do set
expireAfterSeconds, a partial TTL index lets you set a different, shorter expiration period for matching
documents. You can only create a partialFilterExpression on the metaField.
If a time series collection contains documents with timeField timestamps before 1970-01-01T00:00:00.000Z
or after 2038-01-19T03:14:07.000Z, no documents are deleted from the collection by the TTL "time to live"
feature.
This weather data time series collection deletes documents after 24 hours:
db.createCollection(
"weather24h",
{
timeseries: {
timeField: "timestamp",
metaField: "sensor",
granularity: "hours"
},
expireAfterSeconds: 86400})
This TTL index deletes documents from the MongoDB NYC headquarters weather sensor after 1 hour,
instead of 24 hours:
db.eventlog.createIndex(
{ "timestamp": 1 },
 { partialFilterExpression: { "sensor": { $eq: "40.761873, -73.984287" } } },
{ expireAfterSeconds: 3600 } )

Convert a non-TTL single-field Index into a TTL Index

Starting in MongoDB 5.1, you can add the expireAfterSeconds option to an existing single-field index. To
change a non-TTL single-field index to a TTL index, use the collMod database command:
db.runCommand({
"collMod": <collName>,
"index": {
"keyPattern": <keyPattern>,
"expireAfterSeconds": <number>
}
})
The following example converts a non-TTL single-field index with the pattern { "lastModifiedDate": 1 }
into a TTL index:
db.runCommand({
"collMod": "tickets",
"index": {
"keyPattern": { "lastModifiedDate": 1 },
"expireAfterSeconds": 100
}
})
Change the expireAfterSeconds value for a TTL Index
To change the expireAfterSeconds value for a TTL Index, use the collMod database command:
db.runCommand({
"collMod": <collName>,
"index": {
"keyPattern": <keyPattern>,
"expireAfterSeconds": <number>
}
})
The following example changes the expireAfterSeconds value for an index with the pattern
{ "lastModifiedDate": 1 } on the tickets collection:
db.runCommand({
"collMod": "tickets",
"index": {
"keyPattern": { "lastModifiedDate": 1 },
"expireAfterSeconds": 100
}
})
Behavior
Expiration of Data
TTL indexes expire documents after the specified number of seconds has passed since the indexed field
value. The expiration threshold is the indexed field value plus the specified number of seconds.
If the field is an array, and there are multiple date values in the index, MongoDB uses lowest (earliest) date
value in the array to calculate the expiration threshold.
For time series collections, TTL indexes also remove a bucket of data when all documents inside it expire.
This is equal to the upper timestamp limit of the bucket, plus the expireAfterSeconds value. For example, if
a bucket covers data up until 2023-03-27T18:29:59Z and expireAfterSeconds is 300, the TTL index expires
the bucket after 2023-03-27T18:34:59Z.
If the indexed field in a document is not a date or an array that holds one or more date values, the document
will not expire.
If a document does not contain the indexed field, the document will not expire.
Delete Operations
A background thread in mongod reads the values in the index and removes expired documents from the
collection.
When the TTL thread is active, you will see delete operations in the output of db.currentOp() or in the data
collected by the database profiler.
Starting in MongoDB 6.1:
To improve efficiency, MongoDB may batch multiple document deletions together.
The explain command results contain a new BATCHED_DELETE stage for batched document deletions.
If a time series collection contains documents with timeField timestamps before 1970-01-01T00:00:00.000Z
or after 2038-01-19T03:14:07.000Z, no documents are deleted from the collection by the TTL "time to live"
feature.
Timing of the Delete Operation
MongoDB begins removing expired documents or time series buckets as soon as the index finishes building
on the primary. For more information on the index build process, see Index Builds on Populated Collections.
The TTL index does not guarantee that expired data is deleted immediately upon expiration. There may be a
delay between the time that a document expires and the time that MongoDB removes the document from the
database.
The background task that removes expired documents runs every 60 seconds. As a result, documents may
remain in a collection during the period between the expiration of the document and the running of the
background task. MongoDB starts deleting documents 0 to 60 seconds after the index completes.
Because the duration of the removal operation depends on the workload of your mongod instance, expired
data may exist for some time beyond the 60 second period between runs of the background task.
The delete operations initiated by the TTL task run in the foreground, like other deletes.
Replica Sets
On replica set members, the TTL background thread only deletes documents when a member is in state
primary. The TTL background thread is idle when a member is in state secondary. Secondary members
replicate deletion operations from the primary.
Support for Queries
A TTL index supports queries in the same way non-TTL indexes do.
Restrictions
TTL indexes are single-field indexes. Compound indexes do not support TTL and ignore the
expireAfterSeconds option.
The _id field does not support TTL indexes.
You cannot create a TTL index on a capped collection.
You can only create TTL indexes for a time series collection on the collection timeField.
You cannot use createIndex() to change the value of expireAfterSeconds of an existing index. Instead use
the collMod database command. See Change the expireAfterSeconds value for a TTL Index.
If a non-TTL single-field index already exists for a field, you cannot create a TTL index on the same field
since you cannot create indexes that have the same key specification and differ only by the options. To
change a non-TTL single-field index to a TTL index, use the collMod database command.
#GridFS
GridFS is a specification for storing and retrieving files that exceed the BSON-document size limit of 16
MB.
Instead of storing a file in a single document, GridFS divides the file into parts, or chunks [1], and stores
each chunk as a separate document. By default, GridFS uses a default chunk size of 255 kB; that is, GridFS
divides a file into chunks of 255 kB with the exception of the last chunk. The last chunk is only as large as
necessary. Similarly, files that are no larger than the chunk size only have a final chunk, using only as much
space as needed plus some additional metadata.
GridFS uses two collections to store files. One collection stores the file chunks, and the other stores file
metadata. The section GridFS Collections describes each collection in detail.
When you query GridFS for a file, the driver will reassemble the chunks as needed. You can perform range
queries on files stored through GridFS. You can also access information from arbitrary sections of files, such
as to "skip" to the middle of a video or audio file.
GridFS is useful not only for storing files that exceed 16 MB but also for storing any files for which you
want access without having to load the entire file into memory. See also When to Use GridFS.
When to Use GridFS
In MongoDB, use GridFS for storing files larger than 16 MB.
In some situations, storing large files may be more efficient in a MongoDB database than on a system-level
filesystem.
If your filesystem limits the number of files in a directory, you can use GridFS to store as many files as
needed.
When you want to access information from portions of large files without having to load whole files into
memory, you can use GridFS to recall sections of files without reading the entire file into memory.
.

When you want to keep your files and metadata automatically synced and deployed across a number of
systems and facilities, you can use GridFS. When using geographically distributed replica sets, MongoDB
can distribute files and their metadata automatically to a number of mongod instances and facilities.
Do not use GridFS if you need to update the content of the entire file atomically. As an alternative you can
store multiple versions of each file and specify the current version of the file in the metadata. You can
update the metadata field that indicates "latest" status in an atomic update after uploading the new version of
the file, and later remove previous versions if needed.
Furthermore, if your files are all smaller than the 16 MB BSON Document Size limit, consider storing each
file in a single document instead of using GridFS. You may use the BinData data type to store the binary
data. See your drivers documentation for details on using BinData.
Use GridFS
To store and retrieve files using GridFS, use either of the following:
A MongoDB driver. See the drivers documentation for information on using GridFS with your driver.
The mongofiles command-line tool. See the mongofiles reference for documentation.
GridFS Collections
GridFS stores files in two collections:
chunks stores the binary chunks. For details, see The chunks Collection.
files stores the file's metadata. For details, see The files Collection.
GridFS places the collections in a common bucket by prefixing each with the bucket name. By default,
GridFS uses two collections with a bucket named fs:
fs.files
fs.chunks

You can choose a different bucket name, as well as create multiple buckets in a single database. The full
collection name, which includes the bucket name, is subject to the namespace length limit.
The chunks Collection
Each document in the chunks [1] collection represents a distinct chunk of a file as represented in GridFS.
Documents in this collection have the following form:
{
"_id" : <ObjectId>,
"files_id" : <ObjectId>,
"n" : <num>,
"data" : <binary>
}

A document from the chunks collection contains the following fields:

chunks._id
The unique ObjectId of the chunk.
chunks.files_id
The _id of the "parent" document, as specified in the files collection.
chunks.n
The sequence number of the chunk. GridFS numbers all chunks, starting with 0.
chunks.data
The chunk's payload as a BSON Binary type.
The files Collection
Each document in the files collection represents a file in GridFS.
{
"_id" : <ObjectId>,
"length" : <num>,
"chunkSize" : <num>,
"uploadDate" : <timestamp>,
"md5" : <hash>,
"filename" : <string>,
"contentType" : <string>,
"aliases" : <string array>,
"metadata" : <any>,
}
Documents in the files collection contain some or all of the following fields:
files._id
The unique identifier for this document. The _id is of the data type you chose for the original document. The
default type for MongoDB documents is BSON ObjectId.
files.length
The size of the document in bytes.
files.chunkSize
The size of each chunk in bytes. GridFS divides the document into chunks of size chunkSize, except for the
last, which is only as large as needed. The default size is 255 kilobytes (kB).
files.uploadDate
The date the document was first stored by GridFS. This value has the Date type.
files.md5
Deprecated
The MD5 algorithm is prohibited by FIPS 140-2. MongoDB drivers deprecate MD5 support and will
remove MD5 generation in future releases. Applications that require a file digest should implement it
outside of GridFS and store in files.metadata.
An MD5 hash of the complete file returned by the filemd5 command. This value has the String type.
files.filename
Optional. A human-readable name for the GridFS file.
files.contentType
Deprecated
Optional. A valid MIME type for the GridFS file. For application use only.
Use files.metadata for storing information related to the MIME type of the GridFS file.
files.aliases
Deprecated
Optional. An array of alias strings. For application use only.
Use files.metadata for storing information related to the MIME type of the GridFS file.
files.metadata
Optional. The metadata field may be of any data type and can hold any additional information you want to
store. If you wish to add additional arbitrary fields to documents in the files collection, add them to an object
in the metadata field.
GridFS Indexes
GridFS uses indexes on each of the chunks and files collections for efficiency. Drivers that conform to the
GridFS specification automatically create these indexes for convenience. You can also create any additional
indexes as desired to suit your application's needs.
The chunks Index
GridFS uses a unique, compound index on the chunks collection using the files_id and n fields. This allows
for efficient retrieval of chunks, as demonstrated in the following example:
db.fs.chunks.find( { files_id: myFileID } ).sort( { n: 1 } )
Drivers that conform to the GridFS specification will automatically ensure that this index exists before read
and write operations. See the relevant driver documentation for the specific behavior of your GridFS
application.

If this index does not exist, you can issue the following operation to create it using mongosh:
db.fs.chunks.createIndex( { files_id: 1, n: 1 }, { unique: true } );
The files Index
GridFS uses an index on the files collection using the filename and uploadDate fields. This index allows for
efficient retrieval of files, as shown in this example:
db.fs.files.find( { filename: myFileName } ).sort( { uploadDate: 1 } )
Drivers that conform to the GridFS specification will automatically ensure that this index exists before read
and write operations. See the relevant driver documentation for the specific behavior of your GridFS
application.
If this index does not exist, you can issue the following operation to create it using mongosh:
db.fs.files.createIndex( { filename: 1, uploadDate: 1 } );
[1] (1, 2) The use of the term chunks in the context of GridFS is not related to the use of the term chunks
in the context of sharding.
Sharding GridFS
There are two collections to consider with GridFS - files and chunks.
chunks Collection
To shard the chunks collection, use either { files_id : 1, n : 1 } or { files_id : 1 } as the shard key index.
files_id is an ObjectId and changes monotonically.
For MongoDB drivers that do not run filemd5 to verify successful upload (for example, MongoDB drivers
that support MongoDB 4.0 or greater), you can use Hashed Sharding for the chunks collection.
If the MongoDB driver runs filemd5, you cannot use Hashed Sharding. For details, see SERVER-9888.
files Collection
The files collection is small and only contains metadata. None of the required keys for GridFS lend
themselves to an even distribution in a sharded environment. Leaving files unsharded allows all the file
metadata documents to live on the primary shard.

If you must shard the files collection, use the _id field, possibly in combination with an application field.