Skip to content

Mongodb Secondary Node did not become available #6748

Description

@ppathard

Mongodb Version: 4.4.4
Kubernetes Version: 1.15.9
Replicas: 3 (Primary, Secondary, Secondary)
Using Statefulset for mongodb replicaset.

After adding the clusterDomain(customdomain) primary node has come up and running.
Secondary Node prefix-mongodb-1 tries to connect to primary but fails and reboots.

Before reboot logs

Advertised Hostname: prefix-mongodb-1.prefix-mongodb-headless.namespace.svc.customdomain
Pod name doesn't match initial primary pod name, configuring node as a secondary
mongodb 11:56:26.77
mongodb 11:56:26.78 Welcome to the Bitnami mongodb container
mongodb 11:56:26.78 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
mongodb 11:56:26.78 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
mongodb 11:56:26.79
mongodb 11:56:26.79 INFO  ==> ** Starting MongoDB setup **
mongodb 11:56:26.85 INFO  ==> Validating settings in MONGODB_* env vars...
mongodb 11:56:26.94 INFO  ==> Initializing MongoDB...
mongodb 11:56:26.97 INFO  ==> Deploying MongoDB from scratch...
mongodb 11:56:26.99 DEBUG ==> Starting MongoDB in background...
about to fork child process, waiting until server is ready for connections.
forked process: 42
child process started successfully, parent exiting
mongodb 11:56:29.74 INFO  ==> Creating users...
mongodb 11:56:29.75 INFO  ==> Users created
mongodb 11:56:29.75 INFO  ==> Writing keyfile for replica set authentication...
mongodb 11:56:29.95 INFO  ==> Configuring MongoDB replica set...
mongodb 11:56:29.96 INFO  ==> Stopping MongoDB...
mongodb 11:56:31.05 DEBUG ==> Starting MongoDB in background...
about to fork child process, waiting until server is ready for connections.
forked process: 124
child process started successfully, parent exiting
mongodb 11:56:35.37 DEBUG ==> Waiting for primary node...
mongodb 11:56:35.37 DEBUG ==> Waiting for primary node...
mongodb 11:56:35.38 INFO  ==> Trying to connect to MongoDB server prefix-mongodb-0.prefix-mongodb-headless.namespace.svc.customdomain...
mongodb 11:56:35.39 INFO  ==> Found MongoDB server listening at prefix-mongodb-0.prefix-mongodb-headless.namespace.svc.customdomain:27017 !

mongodb 11:59:36.64 ERROR ==> Node prefix-mongodb-0.prefix-mongodb-headless.namespace.svc.customdomain did not become available
mongodb 11:59:36.65 INFO  ==> Stopping MongoDB...

after reboot logs

{"t":{"$date":"2021-06-24T11:59:42.856+00:00"},"s":"I",  "c":"REPL",     "id":21311,   "ctx":"initandlisten","msg":"Did not find local initialized voted for document at startup"}
{"t":{"$date":"2021-06-24T11:59:42.856+00:00"},"s":"I",  "c":"REPL",     "id":21529,   "ctx":"initandlisten","msg":"Initializing rollback ID","attr":{"rbid":1}}
{"t":{"$date":"2021-06-24T11:59:42.856+00:00"},"s":"I",  "c":"REPL",     "id":21313,   "ctx":"initandlisten","msg":"Did not find local replica set configuration document at startup","attr":{"error":{"code":47,"codeName":"NoMatchingDocument","errmsg":"Did not find replica set configuration document in local.system.replset"}}}
{"t":{"$date":"2021-06-24T11:59:42.859+00:00"},"s":"I",  "c":"CONTROL",  "id":20714,   "ctx":"LogicalSessionCacheRefresh","msg":"Failed to refresh session cache, will try again at the next refresh interval","attr":{"error":"NotYetInitialized: Replication has not yet been configured"}}
{"t":{"$date":"2021-06-24T11:59:42.864+00:00"},"s":"I",  "c":"CONTROL",  "id":20711,   "ctx":"LogicalSessionCacheReap","msg":"Failed to reap transaction table","attr":{"error":"NotYetInitialized: Replication has not yet been configured"}}
{"t":{"$date":"2021-06-24T11:59:42.865+00:00"},"s":"I",  "c":"REPL",     "id":40440,   "ctx":"initandlisten","msg":"Starting the TopologyVersionObserver"}
{"t":{"$date":"2021-06-24T11:59:42.869+00:00"},"s":"I",  "c":"REPL",     "id":40445,   "ctx":"TopologyVersionObserver","msg":"Started TopologyVersionObserver"}


mongodb:
  extraFlags:
    - "--tlsAllowConnectionsWithoutCertificates --tlsAllowInvalidCertificates --tlsAllowInvalidHostnames"
  metrics:
    livenessProbe:
      enabled: false
    readinessProbe:
      enabled: false
    enabled: true
    extraUri: "?ssl=true"
    service:
      annotations:
        prometheus.io/targets: "test"
        prometheus.io/scheme: https
  name: mongodb-headless
  dbHostPrefix: mongodb
  replicaSetName: ""
  auth:
    enabled: true
    username: customadmin
    database: customdb
    existingSecret: mongo-creds-secret
 clusterDomain: customdomain
 image:
    registry: registryurl
    repository: mongodb
    tag: "4.4.4-debian-10-r41"
    debug: true
  persistence:
    size: 10Gi
  resources:
    limits:
      cpu: 550m
      memory: 500Mi
    requests:
      cpu: 100m
  tls:
    ## Enable or disable MongoDB TLS Support
    enabled: true
    existingSecret: mongo-tls-cert

Can some one provide suggestions on how to fix this ?

Metadata

Metadata

Assignees

No one assigned

    Labels

    stale15 days without activity

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions