MongoDB replicaset with PS or PSS setup could not have arbiter added due to crashloop for goal state #1615
Description
@nammn Hello, I am not able to add arbiter to 2 member replicaset. (Same for 3 member replicaset). After adding, the pod is created however operator throws error about not reaching the goal state. After some time replicaset members go down as readiness probe is failed.
What did you do to encounter the bug?
Steps to reproduce the behavior:
- Changed my CR MongoDB cluster yaml manifest database.yaml spec.arbiters: 0 to spec.arbiters: 1
- Use kubectl apply -f database.yaml
- Checked MongoDB community operator logs. Observed the debug log stating that none of pods reached goal state.
What did you expect?
Arbiter to be added to 2 member replicaset. Both 2 members and 1 arbiter to reach goal state.
What happened instead?
Neither members or arbiter could reach goal state. Replicaset stuck in crashloop. Working members become failed after not reaching goal state for some time, too.
Operator Information
- 0.10.0
- docker.io/mongo:6.0.17
Kubernetes Cluster Information
- AWS EKS
- 1.28
If possible, please include:
- The operator logs
2024-09-09T11:35:35.491Z DEBUG scram/scram.go:102 Credentials have not changed, using credentials stored in: secret/dms-user-scram-scram-credentials
2024-09-09T11:35:35.492Z DEBUG agent/agent_readiness.go:111 The Agent in the Pod 'mongodb-0' hasn't reached the goal state yet (goal: 30, agent: 29) {"ReplicaSet": "mongodb-surplus/mongodb"}
2024-09-09T11:35:35.492Z DEBUG agent/agent_readiness.go:111 The Agent in the Pod 'mongodb-1' hasn't reached the goal state yet (goal: 30, agent: 29) {"ReplicaSet": "mongodb-surplus/mongodb"}
2024-09-09T11:35:35.492Z DEBUG agent/agent_readiness.go:111 The Agent in the Pod 'mongodb-2' hasn't reached the goal state yet (goal: 30, agent: 29) {"ReplicaSet": "mongodb-surplus/mongodb"}
2024-09-09T11:35:35.492Z DEBUG agent/agent_readiness.go:111 The Agent in the Pod 'mongodb-arb-0' hasn't reached the goal state yet (goal: 30, agent: -1) {"ReplicaSet": "mongodb-surplus/mongodb"}
2024-09-09T11:35:35.492Z DEBUG agent/replica_set_port_manager.go:122 No port change required {"ReplicaSet": "mongodb-surplus/mongodb"}
2024-09-09T11:35:35.492Z DEBUG agent/replica_set_port_manager.go:40 Calculated process port map: map[mongodb-0:27017 mongodb-1:27017 mongodb-2:27017 mongodb-arb-0:27017] {"ReplicaSet": "mongodb-surplus/mongodb"}
2024-09-09T11:35:35.492Z DEBUG controllers/replica_set_controller.go:505 AutomationConfigMembersThisReconciliation {"mdb.AutomationConfigMembersThisReconciliation()": 3}
2024-09-09T11:35:35.492Z DEBUG controllers/replica_set_controller.go:358 Waiting for agents to reach version 30 {"ReplicaSet": "mongodb-surplus/mongodb"}
2024-09-09T11:35:35.492Z DEBUG agent/agent_readiness.go:111 The Agent in the Pod 'mongodb-0' hasn't reached the goal state yet (goal: 30, agent: 29) {"ReplicaSet": "mongodb-surplus/mongodb"}
2024-09-09T11:35:35.492Z DEBUG agent/agent_readiness.go:111 The Agent in the Pod 'mongodb-1' hasn't reached the goal state yet (goal: 30, agent: 29) {"ReplicaSet": "mongodb-surplus/mongodb"}
2024-09-09T11:35:35.492Z DEBUG agent/agent_readiness.go:111 The Agent in the Pod 'mongodb-2' hasn't reached the goal state yet (goal: 30, agent: 29) {"ReplicaSet": "mongodb-surplus/mongodb"}
2024-09-09T11:35:35.492Z INFO controllers/mongodb_status_options.go:110 ReplicaSet is not yet ready, retrying in 10 seconds
- Below we assume that your replicaset database pods are named
mongo-<>. For instance:
❯ k get pods
NAME READY STATUS RESTARTS AGE
mongo-0 2/2 Running 0 19h
mongo-1 2/2 Running 0 19h
❯ k get mdbc
NAME PHASE VERSION
mongo Running 4.4.0
- yaml definitions of your MongoDB Deployment(s):
kubectl get mdbc -oyaml
apiVersion: v1
items:
- apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
annotations:
mongodb.com/v1.lastAppliedMongoDBVersion: 6.0.17
creationTimestamp: "2024-01-03T07:47:03Z"
generation: 48
labels:
k8slens-edit-resource-version: v1
name: mongodb
namespace: mongodb-<SENSITIVE>
resourceVersion: "391080428"
uid: 8dbc92a1-061b-4ebb-a2be-d1b5dd6d696b
spec:
additionalMongodConfig:
storage.wiredTiger.engineConfig.journalCompressor: zlib
arbiters: 1
members: 3
security:
authentication:
ignoreUnknownUsers: true
modes:
- SCRAM
statefulSet:
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: NodeGroup
operator: In
values:
- <SENSITIVE>
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mongodb-<SENSITIVE>
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- name: mongod
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 500m
memory: 1Gi
volumeClaimTemplates:
- metadata:
name: data-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 70G
storageClassName: ebs-sc
- metadata:
name: logs-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10G
storageClassName: ebs-sc
type: ReplicaSet
version: 6.0.17
status:
currentMongoDBMembers: 3
currentStatefulSetReplicas: 3
message: ReplicaSet is not yet ready, retrying in 10 seconds
mongoUri: mongodb://mongodb-0.mongodb-svc.mongodb-surplus.svc.cluster.local:27017,mongodb-1.mongodb-svc.mongodb-surplus.svc.cluster.local:27017,mongodb-2.mongodb-svc.mongodb-surplus.svc.cluster.local:27017/?replicaSet=mongodb
phase: Pending
version: 6.0.17
kind: List
metadata:
resourceVersion: ""
- yaml definitions of your kubernetes objects like the statefulset(s), pods (we need to see the state of the containers):
kubectl get sts -oyaml
apiVersion: v1
items:
- apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: "2024-09-06T13:47:18Z"
generation: 4
name: mongodb
namespace: mongodb-XXX
ownerReferences:
- apiVersion: mongodbcommunity.mongodb.com/v1
blockOwnerDeletion: true
controller: true
kind: MongoDBCommunity
name: mongodb
uid: 8dbc92a1-061b-4ebb-a2be-d1b5dd6d696b
resourceVersion: "391084063"
uid: 25fa6a25-5016-4a16-af39-8c6907338a49
spec:
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: Retain
podManagementPolicy: OrderedReady
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app: mongodb-XXX
serviceName: mongodb-XXX
template:
metadata:
annotations:
kubectl.kubernetes.io/restartedAt: "2024-09-09T07:49:13Z"
creationTimestamp: null
labels:
app: mongodb-XXX
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: NodeGroup
operator: In
values:
- XXX
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mongodb-XXX
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- args:
- ""
command:
- /bin/sh
- -c
- "\nif [ -e \"/hooks/version-upgrade\" ]; then\n\t#run post-start hook
to handle version changes (if exists)\n /hooks/version-upgrade\nfi\n\n#
wait for config and keyfile to be created by the agent\nwhile ! [ -f /data/automation-mongod.conf
-a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep
3 ; done ; sleep 2 ;\n\n# start mongod with this configuration\nexec mongod
-f /data/automation-mongod.conf;\n\n"
env:
- name: AGENT_STATUS_FILEPATH
value: /healthstatus/agent-health-status.json
image: docker.io/mongo:6.0.17
imagePullPolicy: IfNotPresent
name: mongod
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 500m
memory: 1Gi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: data-volume
- mountPath: /healthstatus
name: healthstatus
- mountPath: /hooks
name: hooks
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
- command:
- /bin/bash
- -c
- |-
current_uid=$(id -u)
declare -r current_uid
if ! grep -q "${current_uid}" /etc/passwd ; then
sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
export LD_PRELOAD=libnss_wrapper.so
export NSS_WRAPPER_GROUP=/etc/group
fi
agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -logLevel INFO -maxLogFileDurationHrs 24
env:
- name: AGENT_STATUS_FILEPATH
value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
- name: AUTOMATION_CONFIG_MAP
value: mongodb-config
- name: HEADLESS_AGENT
value: "true"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/mongodb/mongodb-agent:12.0.15.7646-1
imagePullPolicy: Always
name: mongodb-agent
readinessProbe:
exec:
command:
- /opt/scripts/readinessprobe
failureThreshold: 40
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/lib/automation/config
name: automation-config
readOnly: true
- mountPath: /data
name: data-volume
- mountPath: /var/log/mongodb-mms-automation/healthstatus
name: healthstatus
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
dnsPolicy: ClusterFirst
initContainers:
- command:
- cp
- version-upgrade-hook
- /hooks/version-upgrade
image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.6
imagePullPolicy: Always
name: mongod-posthook
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /hooks
name: hooks
- command:
- cp
- /probes/readinessprobe
- /opt/scripts/readinessprobe
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.12
imagePullPolicy: Always
name: mongodb-agent-readinessprobe
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccount: mongodb-database
serviceAccountName: mongodb-database
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: agent-scripts
- name: automation-config
secret:
defaultMode: 416
secretName: mongodb-config
- emptyDir: {}
name: healthstatus
- emptyDir: {}
name: hooks
- emptyDir: {}
name: mongodb-keyfile
- emptyDir: {}
name: tmp
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: data-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 70G
storageClassName: ebs-sc
volumeMode: Filesystem
status:
phase: Pending
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: logs-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10G
storageClassName: ebs-sc
volumeMode: Filesystem
status:
phase: Pending
status:
availableReplicas: 0
collisionCount: 0
currentReplicas: 3
currentRevision: mongodb-6847cc6f7
observedGeneration: 4
replicas: 3
updateRevision: mongodb-6847cc6f7
updatedReplicas: 3
- apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: "2024-09-06T13:47:10Z"
generation: 8
name: mongodb-arb
namespace: mongodb-XXX
ownerReferences:
- apiVersion: mongodbcommunity.mongodb.com/v1
blockOwnerDeletion: true
controller: true
kind: MongoDBCommunity
name: mongodb
uid: 8dbc92a1-061b-4ebb-a2be-d1b5dd6d696b
resourceVersion: "391081887"
uid: 1267641d-8cfa-4d13-ae06-a79c5255facc
spec:
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: Retain
podManagementPolicy: OrderedReady
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: mongodb-XXX
serviceName: mongodb-XXX
template:
metadata:
annotations:
kubectl.kubernetes.io/restartedAt: "2024-09-09T07:45:41Z"
creationTimestamp: null
labels:
app: mongodb-XXX
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: NodeGroup
operator: In
values:
- XXX
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mongodb-XXX
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- args:
- ""
command:
- /bin/sh
- -c
- "\nif [ -e \"/hooks/version-upgrade\" ]; then\n\t#run post-start hook
to handle version changes (if exists)\n /hooks/version-upgrade\nfi\n\n#
wait for config and keyfile to be created by the agent\nwhile ! [ -f /data/automation-mongod.conf
-a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep
3 ; done ; sleep 2 ;\n\n# start mongod with this configuration\nexec mongod
-f /data/automation-mongod.conf;\n\n"
env:
- name: AGENT_STATUS_FILEPATH
value: /healthstatus/agent-health-status.json
image: docker.io/mongo:6.0.17
imagePullPolicy: IfNotPresent
name: mongod
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 500m
memory: 1Gi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: data-volume
- mountPath: /healthstatus
name: healthstatus
- mountPath: /hooks
name: hooks
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
- command:
- /bin/bash
- -c
- |-
current_uid=$(id -u)
declare -r current_uid
if ! grep -q "${current_uid}" /etc/passwd ; then
sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
export LD_PRELOAD=libnss_wrapper.so
export NSS_WRAPPER_GROUP=/etc/group
fi
agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -logLevel INFO -maxLogFileDurationHrs 24
env:
- name: AGENT_STATUS_FILEPATH
value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
- name: AUTOMATION_CONFIG_MAP
value: mongodb-config
- name: HEADLESS_AGENT
value: "true"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/mongodb/mongodb-agent:12.0.15.7646-1
imagePullPolicy: Always
name: mongodb-agent
readinessProbe:
exec:
command:
- /opt/scripts/readinessprobe
failureThreshold: 40
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/lib/automation/config
name: automation-config
readOnly: true
- mountPath: /data
name: data-volume
- mountPath: /var/log/mongodb-mms-automation/healthstatus
name: healthstatus
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
dnsPolicy: ClusterFirst
initContainers:
- command:
- cp
- version-upgrade-hook
- /hooks/version-upgrade
image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.6
imagePullPolicy: Always
name: mongod-posthook
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /hooks
name: hooks
- command:
- cp
- /probes/readinessprobe
- /opt/scripts/readinessprobe
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.12
imagePullPolicy: Always
name: mongodb-agent-readinessprobe
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccount: XXX
serviceAccountName: XXX
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: agent-scripts
- name: automation-config
secret:
defaultMode: 416
secretName: mongodb-config
- emptyDir: {}
name: healthstatus
- emptyDir: {}
name: hooks
- emptyDir: {}
name: mongodb-keyfile
- emptyDir: {}
name: tmp
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: data-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 70G
storageClassName: ebs-sc
volumeMode: Filesystem
status:
phase: Pending
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: logs-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10G
storageClassName: ebs-sc
volumeMode: Filesystem
status:
phase: Pending
status:
availableReplicas: 1
collisionCount: 0
currentReplicas: 1
currentRevision: mongodb-arb-5f6bc75bb8
observedGeneration: 8
readyReplicas: 1
replicas: 1
updateRevision: mongodb-arb-5f6bc75bb8
updatedReplicas: 1
kind: List
metadata:
resourceVersion: ""
- The agent clusterconfig of the faulty members:
kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/lib/automation/config/cluster-config.json
{"version":32,"processes":[{"name":"mongodb-0","disabled":false,"hostname":"mongodb-0.mongodb-svc.mongodb-xxx.svc.cluster.local","args2_6":{"net":{"port":27017},"repl
ication":{"replSetName":"mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibilityVersion":"6.0","processTy
pe":"mongod","version":"6.0.17","authSchemaVersion":5},{"name":"mongodb-1","disabled":false,"hostname":"mongodb-1.mongodb-svc.mongodb-xxx.svc.cluster.local","args2_6"
:{"net":{"port":27017},"replication":{"replSetName":"mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibil
ityVersion":"6.0","processType":"mongod","version":"6.0.17","authSchemaVersion":5},{"name":"mongodb-2","disabled":false,"hostname":"mongodb-2.mongodb-svc.mongodb-xxx.
svc.cluster.local","args2_6":{"net":{"port":27017},"replication":{"replSetName":"mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"
zlib"}}}},"featureCompatibilityVersion":"6.0","processType":"mongod","version":"6.0.17","authSchemaVersion":5},{"name":"mongodb-arb-0","disabled":false,"hostname":"mongod
b-arb-0.mongodb-svc.mongodb-xxx.svc.cluster.local","args2_6":{"net":{"port":27017},"replication":{"replSetName":"mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"
engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibilityVersion":"6.0","processType":"mongod","version":"6.0.17","authSchemaVersion":5}],"replicaSets":[{"_id":
"mongodb","members":[{"_id":0,"host":"mongodb-0","arbiterOnly":false,"votes":1,"priority":1},{"_id":1,"host":"mongodb-1","arbiterOnly":false,"votes":1,"priority":1},{"_id
":2,"host":"mongodb-2","arbiterOnly":false,"votes":1,"priority":1},{"_id":100,"host":"mongodb-arb-0","arbiterOnly":true,"votes":1,"priority":1}],"protocolVersion":"1","nu
mberArbiters":1}],"auth":{"usersWanted":[{"mechanisms":[],"roles":[{"role":"clusterAdmin","db":"admin"},{"role":"userAdminAnyDatabase","db":"admin"}],"user":"admin-user",
"db":"admin","authenticationRestrictions":[],"scramSha256Creds":{"iterationCount":15000,"salt":"WioeMJQXT8w9Coif5Fq3gqV1OfeDi64Bvq/maw==","serverKey":"iK8SJLwCzmk95+mUePC
3wrpGw29Tfx9vN+ZCKSMPKMM=","storedKey":"Z33GU9ix2W++nlnkFBIbYP7kEATZ/6sDVQqdhEd+tT0="},"scramSha1Creds":{"iterationCount":10000,"salt":"Q9mmbNXpyLDRtYmoln1xgA==","serverK
ey":"AmaNP+YmbrNf23l8URaZAZKKOz0=","storedKey":"0d8SscAfTMph+2aW416TXB1/UZw="}},{"mechanisms":[],"roles":[{"role":"readWrite","db":"xxx"}],"user":"xxx-prod-user","db":"ad
min","authenticationRestrictions":[],"scramSha256Creds":{"iterationCount":15000,"salt":"GOkTsWgdrct5KSSQtTHC20myEJM76v5OMEGXOA==","serverKey":"ksqF9YIWnI50+bQJhjl0/zA1a0H
0UpcNnzxnFEjciV4=","storedKey":"GyxjpwCp9hTHsK5CX2ObkIs73NP2zL1VrwbCQTLDvGE="},"scramSha1Creds":{"iterationCount":10000,"salt":"4RAcRyAxnRCQQhcHWRDA2w==","serverKey":"59K
Q8PQV/rS4zxSuVca/tQbDNWw=","storedKey":"v68O4bx8u7/RNIks1WBvmXIJ+H8="}},{"mechanisms":[],"roles":[{"role":"readWrite","db":"xxx"}],"user":"xxx-prod-user","db":"admin","au
thenticationRestrictions":[],"scramSha256Creds":{"iterationCount":15000,"salt":"vYX0jOTF0NvPPmpQm1oz/b7v1/sAnFOMWdm5Pg==","serverKey":"MWuSeUedUk33YD57g/pVw+kV89vQK8OmTib
RLl2hR0U=","storedKey":"ffXuxQ5HTf0FcH2FdcNvKigWSO/TgPdF0elXk9iYX3E="},"scramSha1Creds":{"iterationCount":10000,"salt":"zV90H0Z2XJ8sCiupCSK3PQ==","serverKey":"IwdxN4BVrGS
qyLPXDhrbZKFsbtc=","storedKey":"e5XxJTwdueUyUyJd3ioqQFuEKbc="}},{"mechanisms":[],"roles":[{"role":"readWrite","db":"xxx"},{"role":"readWrite","db":"xxx"}],"user":"xxx-
user","db":"admin","authenticationRestrictions":[],"scramSha256Creds":{"iterationCount":15000,"salt":"0HwrZJa9FIMCy5r4erCq7o2gb/RSaHofCV1XMw==","serverKey":"4FTg/fstci+8W
6BZE4jfyXpLJr9/f4zsuDiKrLnBcgg=","storedKey":"lThVu1E2tv14Q7H58DYYNK1jlqXaIZDCp/Omp44wR1A="},"scramSha1Creds":{"iterationCount":10000,"salt":"gDxOqOLC16/e/WvhWSGDdA==","s
erverKey":"q6SKd30cOY+PFQnqRFMpdgmNTFA=","storedKey":"9swpmpNkjLofRRRprZWGrCBoolk="}},{"mechanisms":[],"roles":[{"role":"dbAdmin","db":"admin"},{"role":"userAdminAnyDatab
ase","db":"admin"},{"role":"readWrite","db":"xxx"},{"role":"readWrite","db":"xxx"},{"role":"readWrite","db":"local"}],"user":"xxx-user","db":"admin","authenticationRe
strictions":[],"scramSha256Creds":{"iterationCount":15000,"salt":"m40BqXb1jbVT4NIN9NjAdbTdQp84O6KtEbRjgA==","serverKey":"hztnbCDXJs0zBcwtaJsLquRtEgCHDykKj04SaQ3eLn8=","st
oredKey":"cftguOpTPNr4QYvL2XtrRlybPzi96CgzGoZ91EVZg2g="},"scramSha1Creds":{"iterationCount":10000,"salt":"SofiWm+P4s3RwvvIxflOOQ==","serverKey":"73knk0VrQPm6PWSYM5PFYwUK1
lA=","storedKey":"wnYGbRIv2qPtcpv3j4r+lUX8x/4="}},{"mechanisms":[],"roles":[{"role":"read","db":"cps"},{"role":"read","db":"xxx"},{"role":"changestreamrole","db":"admin
"}],"user":"xxx-user","db":"admin","authenticationRestrictions":[],"scramSha256Creds":{"iterationCount":15000,"salt":"wzzV+4JFqwIGw1mAzRucb1oiIcYYR/gdcwZyJw==","serverKey
":"Ty105G/oxXhrv9UwgIqqXHO7ZxM5LYW9T/mta7uiQYo=","storedKey":"63E/7kQy2g4O/MUd0a62q8pQNBtITkJ74dUagrRESO4="},"scramSha1Creds":{"iterationCount":10000,"salt":"R/PWGnO94tyg
xdNjIavdkQ==","serverKey":"Gr8vtcLPyb/pR/h8GRivOrY6/sE=","storedKey":"kwWqu/dnDzNTDOEu03TcgeDWxPY="}}],"disabled":false,"authoritativeSet":false,"autoAuthMechanisms":["SC
RAM-SHA-256"],"autoAuthMechanism":"SCRAM-SHA-256","deploymentAuthMechanisms":["SCRAM-SHA-256"],"autoUser":"mms-automation","key":"8tQDoV1eZdKJvpc7cA8rtu939Glj1IgsL9CNE1nf
7SuZMFw8Te47PmhA9Z1NPi27cRw5+bs16kenEAPP82V7v51Xcv5Q9xPZKUxltKlc3t9cfq2Q7Il42DJsrjhQUhne5lKNghWLRHPSFVb8IHbuImgPcvu7mPz6VsYClu6Lno5ewW3ziIvilIW/2xvpxqG0qz4jvz5/cmtTWeNn7V
JzNOYwYurWdFfvdUDL+Z+kQqcbsa95SSYA8217h6aKE2guOwlVpK0VZBYCPACg+ID1dARawAHG7xCA92lFttymLfgu8kbUXeW6RxBsgwz5iuOXjiIrm8XpWhpWHLJNplf5YaGsqBMIbRlH3tAXGv6auqLaiGup3+kQXDJNwC7J
uaa5F0FGXg+PdQPMOH4xv2SZy0zGHh988CaEtXhBVWiQ06FhnNWyxziLCl8BGJpCbD2bsGiiUBcUHvxkCybARhguLYdnS60+tlJcMIr3rpt7MTgRuHhwki0gX1KcVmEe+tPeg57RdqcVcEEqqHYwc4Ghkk/PF/10BlsO0NiUZm
JxZqow7ffSRZHtZ/VKW2og6CZp2V3BaYZmzYwHn5XFFRCDNUu8mbwvtySQVSlVVY4GbKRkgepYsWrYGc20yPH7Hzni9b8N0zCmX8HPy5icn8+jf4z7BRw=","keyfile":"/var/lib/mongodb-mms-automation/authent
ication/keyfile","keyfileWindows":"%SystemDrive%\\MMSAutomation\\versions\\keyfile","autoPwd":"eX-rXNR2PB_ytwagyylk"},"tls":{"CAFilePath":"","clientCertificateMode":"OPTI
ONAL"},"mongoDbVersions":[{"name":"6.0.17","builds":[{"platform":"linux","url":"","gitVersion":"","architecture":"amd64","flavor":"rhel","minOsVersion":"","maxOsVersion":
"","modules":[]},{"platform":"linux","url":"","gitVersion":"","architecture":"amd64","flavor":"ubuntu","minOsVersion":"","maxOsVersion":"","modules":[]},{"platform":"linu
x","url":"","gitVersion":"","architecture":"aarch64","flavor":"ubuntu","minOsVersion":"","maxOsVersion":"","modules":[]},{"platform":"linux","url":"","gitVersion":"","arc
hitecture":"aarch64","flavor":"rhel","minOsVersion":"","maxOsVersion":"","modules":[]}]}],"backupVersions":[],"monitoringVersions":[],"options":{"downloadBase":"/var/lib/
mongodb-mms-automation"}}
- The agent health status of the faulty members:
kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
{"statuses":{"mongodb-arb-0":{"IsInGoalState":false,"LastMongoUpTime":1725903583,"ExpectedToBeUp":true,"ReplicationStatus":-1}},"mmsStatus":{"mongodb-arb-0":{"name":"mong
odb-arb-0","lastGoalVersionAchieved":-1,"plans":[{"automationConfigVersion":32,"started":"2024-09-09T17:37:58.413084367Z","completed":null,"moves":[{"move":"Start","moveD
oc":"Start the process","steps":[{"step":"StartFresh","stepDoc":"Start a mongo instance (start fresh)","isWaitStep":false,"started":"2024-09-09T17:37:58.413103457Z","com
pleted":"2024-09-09T17:38:02.701441838Z","result":"success"}]},{"move":"WaitRsInit","moveDoc":"Wait for the replica set to be initialized by another member","steps":[{"st
ep":"WaitRsInit","stepDoc":"Wait for the replica set to be initialized by another member","isWaitStep":true,"started":"2024-09-09T17:38:02.701493829Z","completed":null,"r
esult":"wait"}]},{"move":"WaitFeatureCompatibilityVersionCorrect","moveDoc":"Wait for featureCompatibilityVersion to be right","steps":[{"step":"WaitFeatureCompatibilityV
ersionCorrect","stepDoc":"Wait for featureCompatibilityVersion to be right","isWaitStep":true,"started":null,"completed":null,"result":""}]}]}],"errorCode":0,"errorString
":""}}}
- The verbose agent logs of the faulty members:
kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent-verbose.log
[2024-09-09T17:40:52.041+0000] [.info] [src/director/director.go:tracef:806] <mongodb-arb-0> [17:40:52.041] because
[All the following are true:
['currentState.Up' = true]
['currentState.CanRsInit' = false]
['desiredState.ReplSetConf' != <nil> ('desiredState.ReplSetConf' = ReplSetConfig{id=mongodb,version=0,commitmentStatus=false,configsvr=false,protocolVersion=1,forcePr
otocolVersion=false,writeConcernMajorityJournalDefault=,members={id:0,HostPort:mongodb-0.mongodb-svc.mongodb-xxx.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,H
idden:false,SecondaryDelaySecs:0,Votes:1,Tags:map[]},{id:1,HostPort:mongodb-1.mongodb-svc.mongodb-xxx.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false
,SecondaryDelaySecs:0,Votes:1,Tags:map[]},{id:2,HostPort:mongodb-2.mongodb-svc.mongodb-xxx.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false,SecondaryD
elaySecs:0,Votes:1,Tags:map[]},{id:100,HostPort:mongodb-arb-0.mongodb-svc.mongodb-xxx.svc.cluster.local:27017,ArbiterOnly:truePriority:1,Hidden:false,SecondaryDelaySe
cs:0,Votes:1,Tags:map[]},settings=map[]})]
['currentState.ReplSetConf' = <nil>]
]
[2024-09-09T17:40:52.041+0000] [.info] [src/director/director.go:planAndExecute:575] <mongodb-arb-0> [17:40:52.041] Step=WaitRsInit as part of Move=WaitRsInit in plan fai
led : <mongodb-arb-0> [17:40:52.041] Postcondition not yet met for step WaitRsInit because
['currentState.ReplSetConf' = <nil>].
Recomputing a plan...
[2024-09-09T17:40:52.362+0000] [.warn] [metrics/collector/util.go:getPingStatus:84] <hardwareMetricsCollector> [17:40:52.362] Failed to fetch replStatus for mongodb-arb-0
: <hardwareMetricsCollector> [17:40:52.362] Error executing WithClientFor() for cp=mongodb-arb-0.mongodb-svc.mongodb-surplus.svc.cluster.local:27017 (local=false) connec
tMode=SingleConnect : <hardwareMetricsCollector> [17:40:52.362] Error running command for runCommandWithTimeout(dbName=admin, cmd=[{replSetGetStatus 1}]) : result={} iden
tityUsed=__system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][668] : (NotYetInitialized) no replset config has been received
[2024-09-09T17:40:52.678+0000] [.info] [src/config/config.go:ReadClusterConfig:433] [17:40:52.678] Retrieving cluster config from /var/lib/automation/config/cluster-confi
g.json...
[2024-09-09T17:40:52.678+0000] [.info] [main/components/agent.go:LoadClusterConfig:277] [17:40:52.678] clusterConfig unchanged
[2024-09-09T17:40:53.072+0000] [.info] [src/mongoclientservice/mongoclientservice.go:func1:1619] [17:40:53.072] Testing auth with username __system db=local to mongodb-ar
b-0.mongodb-svc.mongodb-xxx.svc.cluster.local:27017 (local=false) connectMode=SingleConnect ipversion=0 tls=false
[2024-09-09T17:40:53.081+0000] [.info] [src/mongoctl/processctl.go:GetKeyHashes:2080] <mongodb-arb-0> [17:40:53.081] Able to successfully auth to mongodb-arb-0.mongodb-sv
c.mongodb-xxx.svc.cluster.local:27017 (local=false) using desired auth key
[2024-09-09T17:40:53.108+0000] [.info] [src/mongoctl/processctl.go:Update:3555] <mongodb-arb-0> [17:40:53.108] <DB_WRITE> Updated with query map[] and update [{$set [{age
ntFeatures [StateCache]} {nextVersion 32}]}] and upsert=true on local.clustermanager
[2024-09-09T17:40:53.125+0000] [.info] [src/director/director.go:computePlan:278] <mongodb-arb-0> [17:40:53.125] ... process has a plan : WaitRsInit,WaitFeatureCompatibil
ityVersionCorrect
[2024-09-09T17:40:53.125+0000] [.info] [src/director/director.go:tracef:806] <mongodb-arb-0> [17:40:53.125] Running step: 'WaitRsInit' of move 'WaitRsInit'
[2024-09-09T17:40:53.125+0000] [.info] [src/director/director.go:tracef:806] <mongodb-arb-0> [17:40:53.125] because
[All the following are true:
['currentState.Up' = true]
['currentState.CanRsInit' = false]
['desiredState.ReplSetConf' != <nil> ('desiredState.ReplSetConf' = ReplSetConfig{id=mongodb,version=0,commitmentStatus=false,configsvr=false,protocolVersion=1,forcePr
otocolVersion=false,writeConcernMajorityJournalDefault=,members={id:0,HostPort:mongodb-0.mongodb-svc.mongodb-xxx.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,H
idden:false,SecondaryDelaySecs:0,Votes:1,Tags:map[]},{id:1,HostPort:mongodb-1.mongodb-svc.mongodb-xxx.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false
,SecondaryDelaySecs:0,Votes:1,Tags:map[]},{id:2,HostPort:mongodb-2.mongodb-svc.mongodb-xxx.svc.cluster.local:27017,ArbiterOnly:falsePriority:1,Hidden:false,SecondaryD
elaySecs:0,Votes:1,Tags:map[]},{id:100,HostPort:mongodb-arb-0.mongodb-svc.mongodb-xxx.svc.cluster.local:27017,ArbiterOnly:truePriority:1,Hidden:false,SecondaryDelaySe
cs:0,Votes:1,Tags:map[]},settings=map[]})]
['currentState.ReplSetConf' = <nil>]
]
[2024-09-09T17:40:53.125+0000] [.info] [src/director/director.go:planAndExecute:575] <mongodb-arb-0> [17:40:53.125] Step=WaitRsInit as part of Move=WaitRsInit in plan fai
led : <mongodb-arb-0> [17:40:53.125] Postcondition not yet met for step WaitRsInit because
['currentState.ReplSetConf' = <nil>].
Recomputing a plan...
[2024-09-09T17:40:53.364+0000] [.warn] [metrics/collector/util.go:getPingStatus:84] <hardwareMetricsCollector> [17:40:53.364] Failed to fetch replStatus for mongodb-arb-0
: <hardwareMetricsCollector> [17:40:53.364] Error executing WithClientFor() for cp=mongodb-arb-0.mongodb-svc.mongodb-xxx.svc.cluster.local:27017 (local=false) connec
tMode=SingleConnect : <hardwareMetricsCollector> [17:40:53.364] Error running command for runCommandWithTimeout(dbName=admin, cmd=[{replSetGetStatus 1}]) : result={} iden
tityUsed=__system@local[[MONGODB-CR/SCRAM-SHA-1 SCRAM-SHA-256]][668] : (NotYetInitialized) no replset config has been received
[2024-09-09T17:40:53.718+0000] [.info] [src/config/config.go:ReadClusterConfig:433] [17:40:53.718] Retrieving cluster config from /var/lib/automation/config/cluster-confi
g.json...
[2024-09-09T17:40:53.718+0000] [.info] [main/components/agent.go:LoadClusterConfig:277] [17:40:53.718] clusterConfig unchanged
[2024-09-09T17:40:54.157+0000] [.info] [src/mongoclientservice/mongoclientservice.go:func1:1619] [17:40:54.157] Testing auth with username __system db=local to mongodb-ar
b-0.mongodb-svc.mongodb-xxx.svc.cluster.local:27017 (local=false) connectMode=SingleConnect ipversion=0 tls=false
[2024-09-09T17:40:54.166+0000] [.info] [src/mongoctl/processctl.go:GetKeyHashes:2080] <mongodb-arb-0> [17:40:54.166] Able to successfully auth to mongodb-arb-0.mongodb-sv
c.mongodb-xxx.svc.cluster.local:27017 (local=false) using desired auth key
[2024-09-09T17:40:54.191+0000] [.info] [src/mongoctl/processctl.go:Update:3555] <mongodb-arb-0> [17:40:54.191] <DB_WRITE> Updated with query map[] and update [{$set [{age
ntFeatures [StateCache]} {nextVersion 32}]}] and upsert=true on local.clustermanager
[2024-09-09T17:40:54.203+0000] [.info] [src/director/director.go:computePlan:278] <mongodb-arb-0> [17:40:54.203] ... process has a plan : WaitRsInit,WaitFeatureCompatibil
ityVersionCorrect
- You might not have the verbose ones, in that case the non-verbose agent logs:
kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent.log
[2024-09-09T17:37:58.248+0000] [header.info] [::0] GitCommitId = 25bb5320d7087c7aa24eb6118df217a028238723
[2024-09-09T17:37:58.248+0000] [header.info] [::0] AutomationVersion = 12.0.15.7646
[2024-09-09T17:37:58.248+0000] [header.info] [::0] localhost = mongodb-arb-0.mongodb-svc.mongodb-xxx.svc.cluster.local
[2024-09-09T17:37:58.249+0000] [header.info] [::0] ErrorStateSleepTime = 10s
[2024-09-09T17:37:58.249+0000] [header.info] [::0] GoalStateSleepTime = 10s
[2024-09-09T17:37:58.249+0000] [header.info] [::0] NotGoalStateSleepTime = 1s
[2024-09-09T17:37:58.249+0000] [header.info] [::0] PlanCutoffTime = 300000
[2024-09-09T17:37:58.249+0000] [header.info] [::0] TracePlanner = false
[2024-09-09T17:37:58.249+0000] [header.info] [::0] User = mongodb
[2024-09-09T17:37:58.249+0000] [header.info] [::0] Go version = go1.18.5
[2024-09-09T17:37:58.249+0000] [header.info] [::0] MmsBaseURL =
[2024-09-09T17:37:58.249+0000] [header.info] [::0] MmsGroupId =
[2024-09-09T17:37:58.249+0000] [header.info] [::0] HttpProxy =
[2024-09-09T17:37:58.249+0000] [header.info] [::0] DisableHttpKeepAlive = false
[2024-09-09T17:37:58.249+0000] [header.info] [::0] HttpsCAFile =
[2024-09-09T17:37:58.249+0000] [header.info] [::0] TlsRequireValidMMSServerCertificates = true
[2024-09-09T17:37:58.249+0000] [header.info] [::0] TlsMMSServerClientCertificate =
[2024-09-09T17:37:58.249+0000] [header.info] [::0] KMIPProxyCertificateDir = /tmp
[2024-09-09T17:37:58.249+0000] [header.info] [::0] EnableLocalConfigurationServer = false
[2024-09-09T17:37:58.249+0000] [header.info] [::0] DialTimeoutSeconds = 40
[2024-09-09T17:37:58.249+0000] [header.info] [::0] KeepUnusedMongodbVersions = false
[2024-09-09T17:37:58.249+0000] [header.info] [::0] DisallowDowngrades = false
[2024-09-09T17:37:59.378+0000] [.error] [src/mongoctl/processctl.go:RunCommand:1120] <hardwareMetricsCollector> [17:37:59.378] Server at mongodb-arb-0.mongodb-svc.mongodb
-xxx.svc.cluster.local:27017 (local=false) is down
[2024-09-09T17:37:59.479+0000] [.error] [src/mongoctl/processctl.go:RunCommand:1120] <hardwareMetricsCollector> [17:37:59.479] Server at mongodb-arb-0.mongodb-svc.mongodb
-xxx.svc.cluster.local:27017 (local=false) is down
[2024-09-09T17:38:00.430+0000] [.error] [src/mongoctl/processctl.go:RunCommand:1120] <hardwareMetricsCollector> [17:38:00.430] Server at mongodb-arb-0.mongodb-svc.mongodb
-xxx.svc.cluster.local:27017 (local=false) is down
[2024-09-09T17:38:00.531+0000] [.error] [src/mongoctl/processctl.go:RunCommand:1120] <hardwareMetricsCollector> [17:38:00.531] Server at mongodb-arb-0.mongodb-svc.mongodb
-xxx.svc.cluster.local:27017 (local=false) is down
[2024-09-09T17:38:01.461+0000] [.error] [src/mongoctl/processctl.go:RunCommand:1120] <hardwareMetricsCollector> [17:38:01.461] Server at mongodb-arb-0.mongodb-svc.mongodb
-xxx.svc.cluster.local:27017 (local=false) is down
[2024-09-09T17:38:01.569+0000] [.error] [src/mongoctl/processctl.go:RunCommand:1120] <hardwareMetricsCollector> [17:38:01.569] Server at mongodb-arb-0.mongodb-svc.mongodb
-xxx.svc.cluster.local:27017 (local=false) is down
[2024-09-09T17:38:02.385+0000] [.error] [src/mongoctl/processctl.go:RunCommand:1120] <hardwareMetricsCollector> [17:38:02.385] Server at mongodb-arb-0.mongodb-svc.mongodb
-xxx.svc.cluster.local:27017 (local=false) is down
[2024-09-09T17:38:02.487+0000] [.error] [src/mongoctl/processctl.go:RunCommand:1120] <hardwareMetricsCollector> [17:38:02.487] Server at mongodb-arb-0.mongodb-svc.mongodb
-xxx.svc.cluster.local:27017 (local=false) is down