Backup and Recovery in HA Mode

Recommendations

To effectively manage your CloudCenter deployment, backup your deployment on a daily basis – you can setup a cronjob to automatically perform this backup.

If you are upgrading the CloudCenter deployment, the process differs. See Upgrade/Migrate for additional context.

Use this procedure to backup and recover data for the following releases on a per-component basis:

  • CloudCenter 4.6.x
  • CloudCenter 4.7.x

The backup and recovery procedure is performed on a per component basis and the procedure for backup is the same for all components. Instead of repeating this procedure for each role, the procedure calls out the applicable roles for each component in the HA and standalone modes.

Database (PostgreSQL)

Use this procedure for the following roles (see Virtual Appliance Overview > Modes and Roles for additional context).

  • MGMTPOSTGRES_MASTER
  • MGMTPOSTGRES_SLAVE

Backup

Backup your database and application (the following example uses /mnt, you can change this directory as applicable).

Backup from 4.6.x
pg_dump -U cliqr -d cliqrdb > cliqrdb.sql
# At the prompt, enter the password, cliqr

tar czvf cliqrdb.tar.gz cliqrdb.sql
rm cliqrdb.sql  

Recover

This procedure assumes that the MGMTPOSTGRES_MASTER is terminated in the HA set up.

  1. On the existing MGMTPOSTGRES_SLAVE run below command.

    rm -rf /usr/local/osmosix/etc/.HAINSTALLED 
    
    sudo sed -i".bak" "/dbslave/d" /etc/hosts
    sudo sed -i".bak" "/dbmaster/d" /etc/hosts
    sudo sed -i".bak" "/dbslave/d" /root/.ssh/known_hosts
    
    su - postgres
    psql -d cliqrdb -c "select pg_drop_replication_slot('cliqr_rep_slot1');"
    psql -d cliqrdb -c "select pg_drop_replication_slot('cliqr_rep_slot');"
    exit
    
    pcs cluster stop
    pcs cluster destroy
    service pcsd stop
    service pcsd start
  2. Launch the MGMTPOSTGRES_(new)SLAVE VM.
    1. Phase 1: Prepare Infrastructure > MGMTPOSTGRES_MASTER/SLAVE
    2. Phase 2: Configure Firewall Rules > MGMTPOSTGRES_MASTER/SLAVE
    3. Phase 3: Run the Prerequisite Checker > MGMTPOSTGRES_MASTER/SLAVE
    4. Phase 4: Configure Components > MGMTPOSTGRES_MASTER/SLAVE – Use one of the processes (Installer Process or Appliance Process provided after this section) for explicit instructions.

Installer Process

On the new MGMTPOSTGRES VM, follow this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Download the following files from software.cisco.com to the /tmp folder. See Installation Overview > Installation Download Details for additional context.

    • core_installer.bin
    • ccm-installer.jar
    • ccm-response.xml
  3. Run Core installer to setup core system components using the following commands.

    sudo –i
    cd /tmp
    chmod 755 core_installer.bin
    
    
    #Set the following only if a local package store is setup export
    CUSTOM_REPO=<http://local_package_store ip>
    
    
    ./core_installer.bin <ostype><cloudtype> mgmtpostgres
    

    For example:

    ./core_installer.bin centos7 amazon mgmtpostgres

    Syntax:

    <ostype> = centos6, centos7, rhel6, rhel7

    <cloudtype> = amazon, openstack, vmware

  4. Remove the core_installer.bin file.

    rm core_installer.bin
  5. REQUIRED: At this point, you must continue with the Appliance Process to configure the wizard properties and set up the VM.

Appliance Process

Launch an appliance using the MGMTPOSTGRES appliance file. On the new MGMTPOSTGRES_MASTER, follow this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Exchange SSH keys between the new VM and existing MGMTPOSTGRES servers.
    1. Copy the files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from the same location on the existing VM to the same location on new VM.

      If  the .ssh directory doesn’t exist on the New VM, first create it (using the following commands) before copying the files. 

      sudo –i
      mkdir –p ~/.ssh
      chmod 600 ~/.ssh
    2. On new VM, run the following commands.

      sudo –i
      chmod 400 ~/.ssh/id_rsa*
      cat id_rsa.pub >> authorized_keys
      
    3. Verify mutual SSH access between the existing and new VM by running the following command on each server.

      sudo –i
      ssh root@<NEW VM/OLD VM>
      

CCM

Use this procedure for the following roles (see Virtual Appliance Overview > Modes and Roles for additional context).

  • CCM_SA_PRIMARY
  • CCM_SA_SECONDARY

Backup

This procedure assumes that the CCM_SA_PRIMARY is terminated in a HA setup.

  1. To backup your database and application, issue the following commands.

    The following example uses /mnt, you can change this directory as applicable.

    Backup from 4.6.x
    NOW=$(date +"%Y%m%d")
    bakdir="/mnt/bak/$NOW"
    mkdir -p $bakdir
    cd $bakdir
     
    cp -r /usr/local/tomcat/webapps/* . 

Recover

This procedure assumes that the CCM_SA_PRIMARY is terminated in the HA set up.

On the existing CCM_ SA_SECONDARY, perform this procedure.

  1. Navigate to /home/cliqruser.

    cd /home/cliqruser
  2. Remove the .unison folder:

    rm –rf .unison
  3. Verify and delete any running cron jobs containing the name unison.
  4. Navigate to osmosix/etc folder and delete the harole file.

    rm osmosix/etc/harole
  5. To launch the CCM_(new)SA_PRIMARY VM, follow this procedure.

    1.  Phase 1: Prepare Infrastructure > CCM_SA_PRIMARY

    2. Phase 2: Configure Firewall Rules > CCM_SA_PRIMARY

    3. Phase 3: Run the Prerequisite Checker > CCM_SA_PRIMARY

    4.  Phase 4: Configure ComponentsCCM_SA_PRIMARY – Use one of the processes (Installer Process or Appliance Process provided after this section) for explicit instructions.

Installer Process

On the new CCM_SA_PRIMARY VM, perform this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Download the CCM installer files to the /tmp folder:

    1. core_installer.bin
    2. ccm-installer.jar
    3. ccm-response.xml
  3. Run Core installer to setup core system components using the following commands.

    sudo –i
    cd /tmp
    chmod 755 core_installer.bin
    
    
    #Set the following only if a local package store is setup export
    CUSTOM_REPO=<http://local_package_store ip>
    
    
    ./core_installer.bin <ostype><cloudtype> ccm_sa

    For example:

    ./core_installer.bin centos7 amazon ccm_sa

    Syntax:

    <ostype> = centos6, centos7, rhel6, rhel7

    <cloudtype> = amazon, openstack, vmware

  4. Remove the core_installer.bin file.

    rm core_installer.bin
  5. Log off and log back in as the root user to ensure JAVA Home is set
  6. Modify the ccm-response.xml file as follows:

    Response file option

    Value

    Notes

    <entry key="db_host" value="localhost"/> 

    Replace localhost with VIP (MGMTPOSTGRES_VIP_IP)

    Required for the CCMs  to connect to the master Postgres database

  7. Run the appliance installer to setup CCM.

    java -jar ccm-installer.jar ccm-response.xml
  8. Reboot the CCM VM.
  9. REQUIRED: At this point, you must continue with the Appliance Process to configure the wizard properties and set up the VM.

Appliance Process

Launch a VM using the CCM_SA appliance image. To configure the CCM wizard properties, follow this procedure.

  1. On the CCM_SA_PRIMARY server, if the .ssh directory does not exist for the cliqruser, create it using the following commands before copying the files. 

    su - cliqruser
    mkdir –p ~/.ssh
    chmod 600 ~/.ssh
  2. Copy the files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from the CCM_SA_SECONDARY to the same location on CCM_SA_PRIMARY server. 

  3. On the new CCM_SA_PRIMARY server, execute the following commands.

    su - cliqruser
    chmod 400 ~/.ssh/id_rsa*
    cat id_rsa.pub >> authorized_keys
  4. Verify mutual SSH access between the CCM_SA_PRIMARY and CCM_SA_PRIMARY servers by running below command on each server.

    su – cliqruser
    ssh cliqruser@<CCM_SA_PRIMARY_IP or the CCM_SA_ SECONDARY_IP>
  5. On the CCM_SA_ SECONDARY, launch the CCM wizard and configure the properties as specified in CCM_SA_SECONDARY - Run Appliance Install.

  6. Restart both servers.

  7. Update the CCM_LB with the new IP address of the new CCM_SA_PRIMARY server.
    1. SSH into the VM instance using the key pair that you used to launch the VM.

      systemctl stop haproxy
    2. Modify the HAProxy config file as follows to replace the old IP with the new IP address.

      vi /etc/haproxy/haproxy.cfg        
                                                            
      # configuration to listen on 443 with SSL certs and loadbalance
      frontend https-in
          mode http
          log global
          bind *:443 ssl crt /etc/haproxy/mgmtserver.pem ca-file /etc/haproxy/ca.pem
          default_backend ccms
      
      # configuration to listen on 8443 with SSL certs and loadbalance
      frontend httpsalt-in
          mode tcp
          bind *:8443
          default_backend nodes
      
      backend ccms
          balance roundrobin
          mode    http
          log global
          option httplog
          cookie SVR insert preserve nocache
          server  ccm1 <CCM_SA_PRIMARY_IP>:443 check cookie ccm1 ssl verify none
          server  ccm2 <CCM_SA_SECONDARY_IP>:443 check cookie ccm2 ssl verify none
      
      backend nodes
          mode tcp
          balance roundrobin
          option ssl-hello-chk
          server  ccm1 <CCM_SA_PRIMARY_IP>:8443 check
          server  ccm2 <CCM_SA_SECONDARY_IP>:8443 check
      


    3. Start the HAProxy service and check the status to ensure that it is active.

       

      systemctl start haproxy
      systemctl status haproxy 
      

AMQP

Use this procedure for the following roles (see Virtual Appliance Overview Modes and Roles for additional context).

  • AMQP_PRIMARY
  • AMQP_SECONDARY

Backup

Backup the webapp folder containing the exploded war files to a backup folder (the following example uses /mnt, you can change this directory as applicable). This backup only applies to the Guacamole server, not the AMQP server.

NOW=$(date +"%Y%m%d")
bakdir="/mnt/bak/$NOW"
mkdir -p $bakdir
cd $bakdir

cp -r /usr/local/tomcatgua/webapps/* .

Recover

This procedure assumes that the AMQP_PRIMARY is terminated in a HA setup.

Launch the CCM_(new)SA_PRIMARY VM.

  1. Phase 1: Prepare InfrastructureAMQP_PRIMARY
  2. Phase 2: Configure Firewall Rules > AMQP_PRIMARY
  3. Phase 3: Run the Prerequisite Checker > AMQP_PRIMARY
  4. Phase 4: Configure ComponentsAMQP_PRIMARY – Use one of the processes (Installer Process or Appliance Process provided after this section) for explicit instructions.

Installer Process

On the existing AMQP_SECONDARY, perform this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Download the AMQP installer files to the /tmp folder:

    1. core_installer.bin
    2. cco-installer.jar
    3. conn_broker-response.xml
  3. Run Core installer to setup core system components using the following commands.

    sudo –i
    cd /tmp
    chmod 755 core_installer.bin
    
    
    #Set the following only if a local package store is setup export CUSTOM_REPO=<http://local_package_store ip>
    
    
    ./core_installer.bin <ostype><cloudtype> rabbit

    For example:

    ./core_installer.bin centos7 amazon rabbit

    Syntax:

    <ostype> = centos6, centos7, rhel6, rhel7

    <cloudtype> = amazon, azure, azurerm, azurepack, google, openstack, softlayer, vmware, vcd

  4. Remove the core_installer.bin file.

    rm core_installer.bin 
  5. Log off and log back in as the root user to ensure JAVA Home is set.

    exit
    sudo -i
  6. Run the appliance installer to setup AMQP.

    java -jar cco-installer.jar conn_broker-response.xml
  7. Reboot the AMQP VM.
  8. REQUIRED: At this point, you must continue with the Appliance Process to configure the wizard properties and set up the VM.

Appliance Process

Launch a VM using the AMQP appliance image. To configure the GUA wizard properties, follow this procedure.

  1. Copy the cookie from AMQP_SECONDARY to AMQP_PRIMARY node of location /var/lib/rabbitm.  

    chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
    chmod 400 /var/lib/rabb­itmq/.erlang.cookie
    systemctl enable rabbitmq-server.service
    sudo service rabbitmq-server start
  2. Detach AMQP servers using the following command.

    rabbitmq-server –detached
    
  3. On the new AMQP_PRIMARY server, execute the following commands.

    rabbitmqctl stop_app       
    rabbitmqctl join_cluster rabbit@<AMQP_PRIMARY_HOSTNAME>
    rabbitmqctl start_app
  4. On the new AMQP_SECONDARY server, execute the following commands.

    rabbitmqctl stop_app       
    rabbitmqctl join_cluster rabbit@<AMQP_SECONDARY_HOSTNAME>
    rabbitmqctl start_app
  5. Check AQMP cluster status on both the servers.

    rabbitmqctl cluster_status
    
  6. Set the mirroring policy by issuing below command in one of the server.

    rabbitmqctl
    set_policy ha-all "^cliqr" '{"ha-mode":"all"}' -p /cliqr
    
  7. Verify the policy change using the following command.

    rabbitmqctl list_policies –p /cliqr
    
  8. Modify the rabbitmq config file on both servers by adding the following line to rabbit array.

    {cluster_partition_handling,autoheal}
    

    For example:

    [{rabbit,
       [{cluster_partition_handling, autoheal},
          {ssl_listeners, [5671]},
          {handshake_timeout, 300000},
          {ssl_handshake_timeout, 300000},
          {ssl_options,
             [{cacertfile, "/etc/rabbitmq/certs/cacert.pem"},
              {certfile, "/etc/rabbitmq/certs/cert.pem"},
              {keyfile, "/etc/rabbitmq/certs/key.pem"},
              {verify, verify_peer},
              {fail_if_no_peer_cert, false}]}]}]
  9. Restart rabbitmq server on the both the servers.

    /etc/init.d/rabbitmq-server restart
    
  10. Update the AMQP_LB with the new IP address of the new AMQP_PRIMARY server.
    1. SSH into the VM instance using the key pair that you used to launch the VM.

      systemctl stop haproxy
    2. Modify the HAProxy config file as follows to replace the old IP with new

      vi /etc/haproxy/haproxy.cfg
      
      # configuration to listen on 443 and loadbalance 
      
      frontend quacs-in
      	mode tcp
      	log global	
      	bind *:443
      	default_backend amqps
      
      backend quacs
          mode tcp
          balance roundrobin
          option ssl-hello-chk   
          server guac1 <GUAC_PRIMARY>:443 check
          server guac2 <GUAC_SECONDARY>:443 check
      


    3. Start the HAProxy service and check the status, it should be active

       

      systemctl start haproxy
      systemctl status haproxy 
      
  11. Launch the GUA wizard and configure the properties as specified in  GUAC - Configure CCM/CCO Properties for Guacamole. 
  12. Restart the server.

CCO

Use this procedure for the following roles (see Virtual Appliance Overview > Modes and Roles for additional context).

  • CCO
  • CCO_PRIMARY
  • CCO_SECONDARY
  • CCO_TERTIARY
  • CCO_LB

Backup

Backup the exploded war files to a backup folder (the following example uses /mnt, you can change this directory as applicable).

NOW=$(date +"%Y%m%d")
bakdir="/mnt/bak/$NOW"
mkdir -p $bakdir
cd $bakdir
 
cp -r /usr/local/tomcat/webapps/ROOT .

Recover

This procedure assumes that the:

  • CCO_TERTIARY is terminated in a HA setup.
  • Node1, Node2, Node 3 are part of CCO HA.

  • Node3 is down.

  • Node4 is the new Node

To launch the CCM_(new)SA_PRIMARY VM, follow this procedure.

  1. Phase 1: Prepare Infrastructure > CCO_PRIMARY/SECONDARY/TERTIARY
  2. Phase 2: Configure Firewall Rules > CCO_PRIMARY/SECONDARY/TERTIARY
  3. Phase 3: Run the Prerequisite Checker > CCO_PRIMARY/SECONDARY/TERTIARY
  4. Phase 4: Configure ComponentsCCO_PRIMARY/SECONDARY/TERTIARY – Use one of the processes (Installer Process or Appliance Process provided after this section) for explicit instructions.

Installer Process

On the existing CCO_SECONDARY VM, perform this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Download the CCO installer files to the /tmp folder:

    1. core_installer.bin
    2. cco-installer.jar
    3. cco-response.xml
  3. Run Core installer to setup core system components using the following commands.

    sudo –i
    cd /tmp
    chmod 755 core_installer.bin
    
    
    #Set the following only if a local package store is setup export
    CUSTOM_REPO=<http://local_package_store ip>
    
    
    ./core_installer.bin <ostype><cloudtype> cco

    For example:

    ./core_installer.bin centos7 amazon cco

    Syntax:

    <ostype> = centos6, centos7, rhel6, rhel7

    <cloudtype> = amazon, openstack, vmware

  4. Remove the core_installer.bin file.

    rm core_installer.bin /root/cliqr_modules.log
  5. Log off and log back in as the root user to ensure JAVA Home is set

  6. Run the appliance installer to setup CCO.

    java -jar ccm-installer.jar cco-response.xml
  7. Reboot the CCO VM.
  8. REQUIRED: At this point, you must continue with the Appliance Process to configure the wizard properties and set up the VM.

Appliance Process

Launch a VM using the CCO appliance image. To launch the CCO wizard, follow this procedure.

    1. SSH into the CCO instance as a centos user.
    2. Edit the /etc/mongod.conf file and add the following line. 

      vi /etc/mongod.conf
      replSet=setname
    3. Modify bind_ip as follows.

      bind_ip=0.0.0.0
      
    4. Restart mongod.

      service mongod restart
      
  1. CCO_LB with the new IP address of the new CCO_PRIMARY server.
    1. SSH into the VM instance using the key pair that you used to launch the VM.

      systemctl stop haproxy
    2. Modify the HAProxy config file as follows to replace the old IP with the new IP address.

      vi /etc/haproxy/haproxy.cfg        
                                               
      # listen on 8443 with SSL certs and loadbalance
      frontend httpsalt-in
          mode tcp
          log global
          bind *:8443
          default_backend ccos
      
      backend ccos
          mode tcp
          balance roundrobin
          option ssl-hello-chk
          server  cco1 <CCO_PRIMARY_IP>:8443
          server  cco2 <CCO_SECONDARY_IP>:8443 
          server  cco2 <CCO_TERTIARY_IP>:8443 
      


    3. Start the HAProxy service and check the status, to ensure that it is active

       

      systemctl start haproxy
      systemctl status haproxy 
      

Health Monitor

The back up scenario is not applicable in the non-HA mode as there is no known data that can be persisted.

To restore, simply launch a new VM and reconfigure the VM.