CloudCenter 4.8 has reached End of Life (EOL) as of November 14, 2018. See End of Support Notices for additional context.

MongoDB Troubleshooting                                       

Recover MongoDB with Data

This procedure assumes that you have performed the procedure required to Recover and Rejoin a Cluster after a CCO Server Failure as provided in CCO Troubleshooting.

By default, Mongo DB only listens on the Loopback IP and does not try to rejoin a cluster.

netstat -ln | grep 27017

To recover MongoDB and ensure that it rejoins the Cluster, follow this procedure.

  1.   Modify the MongoDB configuration file to listen on all Interfaces.

    vi /etc/mongod.conf
    #Modify the line "bind_ip=127.0.0.1" and set it to "bind_ip=0.0.0.0"
    #Uncomment the line "#replSet=setname"
  2. Compare the file version on CCO3 to the ones on either CCO1 or CCO (assuming CCO3 is the newly-recovered VM.

    cat /etc/mongod.conf | grep -v ^$  | grep -v ^#

  3. Restart MongoDB.

    systemctl restart mongod
    systemctl status mongod
  4. Verify it is now Listening on 0.0.0.0 instead of 127.0.0.1.

    netstat -ln | grep 27017

  5. Using the Mongo DB CLI client, check the status of the Cluster.

    mongo
    rs.status()
    exit

    In one of the three Replication Sets, you should see one that has "self" set to true. 

    The IP in this set should match the IP of the CCO node you replaced.

    You should also see it's "stateStr" as "SECONDARY".


Verify if the MongoDB Service is Running on the CCO

To verify if the MongoDB service is running on the CCO, follow this procedure:

  1. Login to the CCO VM as the platform administrator (root).

  2. Change to the system directory.


    [root@cco482 ~]# cd /etc/systemd/system
  3. Verify the status of the MongoDB service.

    [root@cco482 ~]# systemctl status mongod
    mongod.service - High-performance, schema-free document-oriented database
       Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
       Active: active (running) since Thu 2017-11-30 06:56:47 UTC; 17h ago
         Docs: https://docs.mongodb.org/manual
      Process: 1199 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
      Process: 1193 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
      Process: 1179 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)
     Main PID: 2373 (mongod)
       Memory: 7.5M
       CGroup: /system.slice/mongod.service
               └─2373 /usr/bin/mongod -f /etc/mongod.conf
    
    Nov 30 06:56:47 cc0482 systemd[1]: Starting High-performance, schema-free document-oriented database...
    Nov 30 06:56:47 cc0482 systemd[1]: Started High-performance, schema-free document-oriented database.
    Nov 30 06:56:49 cc0482 mongod[1257]: about to fork child process, waiting until server is ready for connections.
    Nov 30 06:56:49 cc0482 mongod[1257]: forked process: 2373
    Nov 30 06:56:49 cc0482 mongod[1257]: child process started successfully, parent exiting
  4. Next, verify the status of the CCO service.

    [root@cc0482 ~]# systemctl status cco
    
    ● cco.service - CloudCenter Orchestrator
       Loaded: loaded (/etc/systemd/system/cco.service; enabled; vendor preset: disabled)
       Active: active (exited) since Thu 2017-11-30 06:56:41 UTC; 17h ago
      Process: 668 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
     Main PID: 668 (code=exited, status=0/SUCCESS)
       Memory: 0B
       CGroup: /system.slice/cco.service
    
    Nov 30 06:56:41 cc0482 systemd[1]: Starting CloudCenter Orchestrator...
    Nov 30 06:56:41 cc0482 systemd[1]: Started CloudCenter Orchestrator.
  • No labels
© 2017-2019 Cisco Systems, Inc. All rights reserved