Scribd
Upload
24 views8 pages

5 (Migration MongoDB) MongoDB Cluster Migration With Zero Downtime

This document outlines a process for migrating a MongoDB cluster to a new region with zero downtime. The process involves adding new replica instances in the destination region to the existing replica set, updating DNS entries to point to the new instances, reconfiguring the replica set to elect a primary from the destination region, removing instances from the source region one by one until the migration is complete. By maintaining synchronization between the old and new clusters and performing a leadership handoff, the migration can be done without any downtime.

Uploaded by

congdonglinux123
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
24 views8 pages

5 (Migration MongoDB) MongoDB Cluster Migration With Zero Downtime

This document outlines a process for migrating a MongoDB cluster to a new region with zero downtime. The process involves adding new replica instances in the destination region to the existing replica set, updating DNS entries to point to the new instances, reconfiguring the replica set to elect a primary from the destination region, removing instances from the source region one by one until the migration is complete. By maintaining synchronization between the old and new clusters and performing a leadership handoff, the migration can be done without any downtime.

Uploaded by

congdonglinux123
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 8

[Migration MongoDB ] MongoDB

Cluster Migration With Zero


Downtime
Problem Statement
The standard way to migrate data with MongoDB was to use the traditional
dump and restore method which would have required a few minutes to hours
of downtime depending on the data size
Since we are processing critical customers and order data continuously from
mongoDb we cannot afford a long downtime. Hence, we came up with a
solution where we are maintaining continuous sync between old and new
cluster data
Below are the snapshots of business critical data rendering from mongo
cluster

Prerequisite
Mongo should be running with replica-set
Solution
Add new instances of destination region into existing cluster replica-set as
new replicas
Update DNS entries
ReConfigure replica-set to ensure new primary get's elected from destination
region
Remove old replica from replica-set
Initiate leader election to automatically migrate primary node from source
region to destination region
Step by step migration
Add new replicas (destination region)
Create AMI for any mongo instance (replica) and copy it to the destination
region
Launch n (same number of instances as source region) instances using AMI
Update bindIp of new launch instances to use private Ip of instance and
restart new mongo instances
--bindIp: It is the IP address that MongoDB binds to listen for connections from
different applications
Note: This Step is optional if you are using bindIpAll : true config or you are setting
mongo instance manually instead of AWS AMI.
Login to the primary instance and add all new region instances as a replica
rs.add( { host: "mongodbd4.example.net:27017", priority: 0, votes:0} )
Note: As per mongo documentation, replica-set can trigger a leader election in
response to add replica events hence, we should configure the new members to be
excluded from replica-set leader election during adding as well as synchronisation
stage by setting vote and priority equal to Zerofor all new instances

2
Wait until all new region replicas get sync with the primary or in other words
wait until stateStr becomes SECONDARY state for all new region instances
rs.status()

After adding new instance as replica

3
Update DNS records
Update route53 host to point to new instance (n instances of destination
region). To be on a safer side, redeploy all client applications using mongoDb
to refresh DNS entries in case of any caching.
Note: We are using private Ip’s (all instances belonging to the same vpc or vpc
peering is setUp) for internal communication and DNS hostname will be used by
clients for connecting with mongo clusters.

Reconfiguring replica set


In case of primary stepDown(), we need to reconfigure priority and vote for
replica-set to ensure primary get’s elected from destination cluster
Note: In order to ensure leader get's elected from destination region, we will be
setting new region replicas priority higher than old region and give voting rights to
new replicas

4
Remove existing region replicas
Remove replicasof the source region one by one until only primary left.
rs.remove(hostname)

After removing replica instance from existing region

5
Initiate leader election
Stepping Down current primary to elect new primary in destination region
rs.stepDown(stepDownSecs, secondaryCatchUpPeriodSecs)
Above command instructs the primary of the replica set to become a secondary.
After the primary steps down, eligible secondaries will hold an election for primary.
This will ensure that existing primary will become secondary and the primary will
be elected in the new region.
-- stepDownSecs: The number of seconds to step down the primary, during which
time the step down member is ineligible for becoming primary. If you specify a
non-numeric value, the command uses 60 seconds.
--secondaryCatchUpPeriodSecs: The number of seconds that mongod will wait
for an electable secondary to catch up to the primary. Default is 10 sec.

6
After leader election

Remove the remaining replica in the source region


Final stage

Take-Away!

7
MongoDb clusters can indeed be migrated to another region without
downtime. However the replica set cannot process write operations until the
election completes successfully. The median time before a cluster elects a
new primary should not typically exceed 12 seconds. Your application
connection logic should include tolerance for automatic fail overs and the
subsequent leader elections
If you are using MongoDb > 4.2, you will get retryable writes by default. Refer
Docfor more info
The replica set can continue to serve read queries during leader election if
such queries are configured to run on secondaries

https://eng.blackbuck.com/mongodb-cluster-migration-with-zero-downtime/

You might also like