How to Paginate with Mongoose in Node.js?

When working with large datasets in Node.js applications using Mongoose, fetching and displaying all records at once can lead to performance issues. Pagination is a technique that allows us to break down data into smaller, more manageable manner. In this article, we'll explore how to implement pagination with Mongoose in Node.js.

How to Paginate with Mongoose in Node.js?

When dealing with large datasets, fetching and displaying all records at once can be inefficient and lead to performance issues. Pagination allows us to break down the data into smaller, manageable manner. Below  are the traditional ways to paginate with Mongoose that help us to perform pagination with Mongoose in Node.js are as follows:

1. Offset-based pagination
   
2. Cursor-based pagination

1. Offset Based Pagination

Offset-based pagination involves specifying a fixed number of items to skip (offset) and a limit on the number of items to retrieve per page.

For example, if we want to display 10 items per page, we would skip the first "(page number - 1) * 10" items and fetch the next 10. This method is simple and easy to implement but can become inefficient as the offset increases, especially with large datasets. The main drawback is that skipping a large number of items on each request can impact performance.

Syntax:

const data = await Model.find().skip(offset).limit(limit);

Example: The below code uses offset-based pagination to paginate the data with Mongoose in Node.js.

// Define the limit and page number for pagination
const limit = 5;
const page = 1;

// Calculate the offset based on the page number and limit
const offset = (page - 1) * limit;

// Retrieve notes from the database with pagination
const notes = await Note.find()
    .skip(offset)
    .limit(limit);

// Prepare the response object with pagination metadata and notes data
const response = {
    status: 200,
    totalResults: notes.length,    // Total number of notes in the current page
    notes: notes.map(note => ({    // Map each note to a simplified object
        _id: note._id,
        user: note.user,
        title: note.title,
        desc: note.desc,
        category: note.category,
        isCompleted: note.isCompleted,
        createdAt: note.createdAt,
        updatedAt: note.updatedAt,
        _v: note._v
    }))
};

// Output the response
console.log(response);

Output:

Explanation:

• This Node.js code defines a function "getNotes" for fetching and paginating notes using the Mongoose model Notes.
  
• It starts by importing the necessary requirements, including the Mongoose model for tasks. The function initializes parameters for pagination, such as page, limit, and offset.
  
• It then performs pagination using the Mongoose find method with the skip and limit functions, retrieving a subset of notes based on the specified page and limit.
  
• The paginated data, along with metadata such as the total number of results, is returned in a JSON response with a status code of 200.

2. Cursor-based pagination

• Cursor-based pagination is a more efficient approach, especially for large datasets.
  
• Instead of depend on an offset, it uses a cursor, which is a unique identifier pointing to a specific record.
  
• The server sends the cursor along with each page of results, and the client uses it to request the next set of records. Cursors can be based on a sortable field (like a timestamp or unique ID) and are more resilient to changes in the dataset.
  
• This method is well-suited for real-time applications and overcome the drawback of offset-based pagination as it scales.

Syntax:

let query = {};  // write any query 
query._id = { $gt: cursor };

Example: The below code uses cursor-based pagination to paginate the data with mongoose in Node.js.

// importing all requirements
const Notes = require('../models/Task');


// to fetch the notes and paginate the data
const getNotes = async (req, res) => {

    const { cursor, limit = 10 } = req.query;
    let query = {};

    // If a cursor is provided, add it to the query
    if (cursor) {
      query._id = { $gt: cursor };
    }

    // Fetch notes using the cursor-based query
    const notes = await Notes.find(query).limit(Number(limit));

    // Extract the next and previous cursor from the result
    const prevCursor = cursor && notes.length > 0 ? notes[0]._id : null;
    const nextCursor = notes.length > 0 ? notes[notes.length - 1]._id : null;

    // Return the paginated data
    return res.status(200).json({
      status: 200,
      nextCursor,
      prevCursor,
      totalResults: notes.length,
      notes,
    });
}

// exporting notess functions
module.exports = { getNotes };

Output:

Explanation:

• status: The HTTP status code indicating the success of the request (200 for success).
  
• nextCursor: The cursor value to be used for fetching the next page of results.
  
• The notes array contains objects representing individual notes, each with _id, title, and content fields. This structure allows the client to paginate through the notes data efficiently, with the nextCursor and prevCursor values helping to navigate between pages.

• prevCursor: The cursor value to be used for fetching the previous page of results.
  
• totalResults: The total number of notes returned in the current page.
  
• notes: An array containing the notes fetched based on the provided cursor and limit.

Conclusion

Overall, In this article, we've covered how to paginate with Mongoose in Node.js using both offset-based and cursor-based pagination. Offset-based pagination involves skipping a fixed number of items and limiting the number of items per page.

Cursor-based pagination, on the other hand, uses a cursor to navigate through the dataset, making it more efficient for large datasets. By implementing pagination, we can improve the performance of our Node.js applications when dealing with large amounts of data.