// removed jquery ui css and js

Backup and Recovery in HA Mode

Recommendations

To effectively manage your CloudCenter deployment, backup your deployment on a daily basis – you can setup a cronjob to automatically perform this backup.

If you are upgrading the CloudCenter deployment, the process differs. See Upgrade/Migrate for additional context.

The backup and recovery procedure is performed on a per component basis and the procedure for backup is the same for all components. Instead of repeating this procedure for each role, the procedure calls out the applicable roles for each component in the HA and standalone modes.

Database (PostgreSQL)

Use this procedure for the following roles (see Virtual Appliance Overview > Modes and Roles for additional context).

  • MGMTPOSTGRES_MASTER
  • MGMTPOSTGRES_SLAVE

Backup

Backup your database and application (the following example uses /mnt, you can change this directory as applicable).

pg_dump -U cliqr -d cliqrdb > cliqrdb.sql
# At the prompt, enter the password, cliqr

tar czvf cliqrdb.tar.gz cliqrdb.sql
rm cliqrdb.sql  

Recover

This procedure assumes that the MGMTPOSTGRES_MASTER is terminated in the HA set up.

  1. On the existing MGMTPOSTGRES_SLAVE run below command.

    rm -rf /usr/local/osmosix/etc/.HAINSTALLED 
    
    sudo sed -i".bak" "/dbslave/d" /etc/hosts
    sudo sed -i".bak" "/dbmaster/d" /etc/hosts
    sudo sed -i".bak" "/dbslave/d" /root/.ssh/known_hosts
    
    su - postgres
    psql -d cliqrdb -c "select pg_drop_replication_slot('cliqr_rep_slot1');"
    psql -d cliqrdb -c "select pg_drop_replication_slot('cliqr_rep_slot');"
    exit
    
    pcs cluster stop
    pcs cluster destroy
    service pcsd stop
    service pcsd start
  2. Launch the MGMTPOSTGRES_(new)SLAVE VM.
    1. Phase 1: Prepare Infrastructure > MGMTPOSTGRES_MASTER/SLAVE
    2. Phase 2: Configure Firewall Rules > MGMTPOSTGRES_MASTER/SLAVE
    3. Phase 3: Run the Prerequisite Checker > MGMTPOSTGRES_MASTER/SLAVE
    4. Phase 4: Configure Components > MGMTPOSTGRES_MASTER/SLAVE – Use one of the processes (Installer Process or Appliance Process provided after this section) for explicit instructions.

Installer Process

On the new MGMTPOSTGRES VM, follow this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Download the following files from software.cisco.com to the /tmp folder. See Installation Overview > Installation Download Details for additional context.

    • core_installer.bin
    • ccm-installer.jar
    • ccm-response.xml
  3. Run Core installer to setup core system components using the following commands.

    sudo –i
    cd /tmp
    chmod 755 core_installer.bin
    
    
    #Set the following only if a local package store is setup export
    CUSTOM_REPO=<http://local_package_store ip>
    
    
    ./core_installer.bin <ostype><cloudtype> mgmtpostgres
    

    For example:

    ./core_installer.bin centos7 amazon mgmtpostgres

    Syntax:

    <ostype> = centos6, centos7, rhel6, rhel7

    <cloudtype> = amazon, openstack, vmware

  4. Remove the core_installer.bin file.

    rm core_installer.bin
  5. REQUIRED: At this point, you must continue with the Appliance Process to configure the wizard properties and set up the VM.

Appliance Process

Launch an appliance using the MGMTPOSTGRES appliance file. On the new MGMTPOSTGRES_MASTER, follow this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Exchange SSH keys between the new VM and existing MGMTPOSTGRES servers.
    1. Copy the files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from the same location on the existing VM to the same location on new VM.

      If  the .ssh directory doesn’t exist on the New VM, first create it (using the following commands) before copying the files. 

      sudo –i
      mkdir –p ~/.ssh
      chmod 600 ~/.ssh
    2. On new VM, run the following commands.

      sudo –i
      chmod 400 ~/.ssh/id_rsa*
      cat id_rsa.pub >> authorized_keys
      
    3. Verify mutual SSH access between the existing and new VM by running the following command on each server.

      sudo –i
      ssh root@<NEW VM/OLD VM>
      

CCM

In these procedures:

  • user = the username

  • server_ip = the server where you want to copy the required files

Backup


Copy the CCM_SA_PRIMARY and CCM_SA_SECONDARY conf files from the /etc/sysconfig/ folder to your desired location using the following commands.


sudo -i
cd /etc/sysconfig
scp mgmtserver.conf <user>@<server_ip>:/tmp
scp capacity-manager.conf <user>@<server_ip>:/tmp
scp filebeat <user>@<server_ip>:/tmp


This will copy all the files to the /tmp folder of your desired server.

You have now backed up the CCM_SA_PRIMARY and CCM_SA_SECONDARY servers.


Recover

This procedure assumes that the CCM_SA_PRIMARY is terminated in the HA set up.

On the existing CCM_ SA_SECONDARY, perform this procedure.

  1. SSH into CCM_SA_SECONDARY instance.

    sudo -i
  2. Remove the .unison folder:

    rm –rf .unison
  3. Verify and delete any running cron jobs containing the name unison.
  4. Navigate to /usr/local/osmosix/etc folder and delete the harole file.

    rm -f /usr/local/osmosix/etc/harole

Installer Process

On the new CCM_SA_PRIMARY VM, follow this procedure to use the Installer method.

  1. SSH into the CCM_SA_PRIMARY instance.

  2. Download the CCM installer files to the /tmp folder:

    1. core_installer.bin

    2. ccm-installer.jar

    3. ccm-response.xml

  3. Run Core installer to setup core system components using the following commands.

    sudo –i
    cd /tmp
    chmod 755 core_installer.bin
    
    
    #Set the following only if a local package store is setup 
    export CUSTOM_REPO=<http://local_package_store ip>
    
    
    ./core_installer.bin <ostype><cloudtype> ccm_sa

    For example:

    ./core_installer.bin centos7 amazon ccm_sa
  4. Log off and log back in as the root user to ensure JAVA Home is set
  5. Modify the ccm-response.xml file as follows:

    Response file option

    Value

    Notes

    <entry key="db_host" value="localhost"/> 

    Replace localhost with VIP (MGMTPOSTGRES_VIP_IP)

    Required for the CCMs  to connect to the master PostgreSQL database

  6. Run the appliance installer to setup CCM_SA_PRIMARY instance.

    java -jar ccm-installer.jar ccm-response.xml
  7. Reboot the CCM_SA_PRIMARY VM.
  8. Setup SSH communication between the CCM_SA_PRIMARY and CCM_SA_SECONDARY instances using root privileges.
    1. On the CCM_SA_PRIMARY server, if the .ssh directory does not exist for the root, create it using the following commands before copying the files. 

      sudo -i
      mkdir -p ~/.ssh
      chmod 700 ~/.ssh
    2. Copy the files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from the CCM_SA_SECONDARY to the same location on CCM_SA_PRIMARY server.   

    3. On the new CCM_SA_PRIMARY server, execute the following commands.

      chmod 400 ~/.ssh/id_rsa*
      cat id_rsa.pub >> authorized_keys
    4. Verify mutual SSH access between the CCM_SA_PRIMARY and CCM_SA_PRIMARY servers by running the following commands on each server.

      ssh root@<CCM_SA_PRIMARY>/<CCM_SA_SECONDARY>
  9. Copy the backup conf files to the new CCM_SA_PRIMARY using the following commands.

    sudo -i
    cd /etc/sysconfig
    scp <user>@<server_ip>:/tmp/mgmtserver.conf .
    scp <user>@<server_ip:/tmp/capacity-manager.conf .
    scp <user>@<server_ip>:/tmp/ filebeat .

    This will copy all the backup conf files to new CCM_SA_PRIMARY.

  10. Configure HA for CCM_SA_PRIMARY server.

    1. Invoke the CCM wizard on the CCM_SA_PRIMARY server.

      sudo -i
      /usr/local/cliqr/bin/ccm_config_wizard
    2. To configure HA between CCM_SA_PRIMARY and CCM_SA_SECONDARY instances, select Configure HA and enter the New CCM_SA_PRIMARY IP in the Primary Node Private IP field.

    3. Verify your changes and exit the CCM configuration wizard.

  11. Restart the CCM service on the CCM_SA_PRIMARY and CCM_SA_SECONDARY servers.

    service ccm stop
    service ccm start
  12. Update the CCM_LB haproxy.cfg file with new CCM_SA_PRIMARY IP.
    1. SSH into the CCM_LB instance and stop the HAproxy service.

      systemctl stop haproxy
    2. Modify the HAproxy config file as follows to replace the old IP with the new CCM_SA_PRIMARY IP address – append the following details to the HAProxy config file.

      vi /etc/haproxy/haproxy.cfg        
                                                            
      # configuration to listen on 443 with SSL certs and loadbalance
      frontend https-in
          mode http
          log global
          bind *:443 ssl crt /etc/haproxy/mgmtserver.pem ca-file /etc/haproxy/ca.pem
          default_backend ccms
      
      # configuration to listen on 8443 with SSL certs and loadbalance
      frontend httpsalt-in
          mode tcp
          bind *:8443
          default_backend nodes
      
      backend ccms
          balance roundrobin
          mode    http
          log global
          option httplog
          cookie SVR insert preserve nocache
          server  ccm1 <CCM_SA_PRIMARY_IP>:443 check cookie ccm1 ssl verify none
          server  ccm2 <CCM_SA_SECONDARY_IP>:443 check cookie ccm2 ssl verify none
      
      backend nodes
          mode tcp
          balance roundrobin
          option ssl-hello-chk
          server  ccm1 <CCM_SA_PRIMARY_IP>:8443 check
          server  ccm2 <CCM_SA_SECONDARY_IP>:8443 check
      
    3. Start the HAproxy service and check the status to ensure that it is active.

      systemctl start haproxy
      systemctl status haproxy

Appliance Process

On the new CCM_SA_PRIMARY VM, follow this procedure to use the Installer method.

  1. Setup SSH communication between the CCM_SA_PRIMARY and CCM_SA_SECONDARY instances using root privileges.
    1. On the CCM_SA_PRIMARY server, if the .ssh directory does not exist for the root, create it using the following commands before copying the files. 

      sudo -i
      mkdir -p ~/.ssh
      chmod 700 ~/.ssh
    2. Copy the files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from the CCM_SA_SECONDARY to the same location on CCM_SA_PRIMARY server.   

    3. On the new CCM_SA_PRIMARY server, execute the following commands.

      chmod 400 ~/.ssh/id_rsa*
      cat id_rsa.pub >> authorized_keys
    4. Verify mutual SSH access between the CCM_SA_PRIMARY and CCM_SA_PRIMARY servers by running the following commands on each server.

      ssh root@<CCM_SA_PRIMARY>/<CCM_SA_SECONDARY>
  2. Copy the backup conf files to the new CCM_SA_PRIMARY using the following commands.

    sudo -i
    cd /etc/sysconfig
    scp <user>@<server_ip>:/tmp/mgmtserver.conf .
    scp <user>@<server_ip:/tmp/capacity-manager.conf .
    scp <user>@<server_ip>:/tmp/ filebeat  .

    This will copy all the backup conf files to new CCM_SA_PRIMARY.

  3. Configure HA for CCM_SA_PRIMARY server.

    1. Invoke the CCM wizard on the CCM_SA_PRIMARY server.

      sudo -i
      /usr/local/cliqr/bin/ccm_config_wizard
    2. To configure HA between CCM_SA_PRIMARY and CCM_SA_SECONDARY instances, select Configure HA and enter the New CCM_SA_PRIMARY IP in the Primary Node Private IP field.

    3. Verify your changes and exit the CCM configuration wizard.

  4. Restart the CCM service on the CCM_SA_PRIMARY and CCM_SA_SECONDARY servers.

    service ccm stop
    service ccm start
  5. Update the CCM_LB haproxy.cfg file with new CCM_SA_PRIMARY IP.
    1. SSH into the CCM_LB instance and stop the HAproxy service.

      systemctl stop haproxy
    2. Modify the HAproxy config file as follows to replace the old IP with the new CCM_SA_PRIMARY IP address.
    3. Start the HAproxy service and check the status to ensure that it is active.

      systemctl start haproxy
      systemctl status haproxy

CCO

In these procedures:

  • user = the username

  • server_ip = the server where you want to copy the required files

Backup

Copy the  CCO_PRIMARY, CCO_SECONDARY, and CCO_TERTIARY conf files from the /etc/sysconfig/ folder to your desired location using the following commands.

sudo -i
cd /etc/sysconfig
scp gateway.conf <user>@<server_ip>:/tmp
scp cliqr-cis.conf <user>@<server_ip>:/tmp
scp filebeat <user>@<server_ip>:/tmp

This will copy all the files to the /tmp folder of your desired server.

You have now backed up the CCO_PRIMARY, CCO_SECONDARY, and CCO_TERTIARY servers.

Recover

This procedure assumes that the CCO_TERTIARY instance is terminated in the HA mode.

Installer Process

To launch a VM using the CCO installer image, follow this procedure.

  1. SSH in the new CCO_TERTIARY instance.

  2. Download the CCO_TERTIARY file to the /tmp folder.

    • core_installer.bin

    • cco-installer.jar

    • cco-response.xml

  3. Run core installer to setup the CCO_TERTIARY instance using the following commands:

    sudo –i
    cd /tmp
    chmod 755 core_installer.bin
    
    
    #Set the following only if a local package store is setup 
    export CUSTOM_REPO=<http://local_package_store ip>
    
    
    ./core_installer.bin <ostype><cloudtype> cco
  4. Log off and log back in as the root user to ensure JAVA Home is set.

  5. Run the appliance installer to setup the CCO_TERTIARY instanc.

    java -jar cco-installer.jar cco-response.xml
  6. Reboot the CCO_TERTIARY VM.

  7. Set up SSH communication between CCO_PRIMARY, CCO_SECONDARY,  and CCO_TERTIARY using root privileges.

    1. On the CCO_TERTIARY server, if the .ssh directory does not exist for the root create it using the following commands before copying the files.

      sudo -i
      mkdir -p ~/.ssh
      chmod 700 ~/.ssh
    2. Copy the files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from the CCO_SECONDARY to the same location on CCO_TERTIARY server.

    3. On the new CCO_TERTIARY server, execute the following commands.

      chmod 400 ~/.ssh/id_rsa*
      cat id_rsa.pub >> authorized_keys
    4. Verify if mutual SSH access between the CCO_PRIMARY, CCO_SECONDARY, and CCO_TERTIARY servers by running the following commands on each server. 

      ssh root@<CCO_PRIMARY>/<CCO_SECONDARY>/<CCO_TERTIARY>
  8. Copy the backup conf files on new CCO_TERTIARY.

    sudo -i
    cd /etc/sysconfig
    scp <user>@<server_ip>:/tmp/gateway.conf .
    scp <user>@< server_ip >:/tmp/cliqr-cis.conf .
    scp <user>@< server_ip >:/tmp/ filebeat  .
  9. Configure HA for CCO_TERTIARY.
    1. Invoke the CCO wizard on CCO_PRIMARY.

      sudo -i
      /usr/local/cliqr/bin/cco_config_wizard
    2. To configure HA between CCO_PRIMARY, CCO_SECONDARY, and CCO_TERTIARY instances, select Configure HA on each instance and enter the New CCO_TERTIARY IP in the Tertiary Node IP field.

    3. Verify your changes and exit the CCM configuration wizard.

  10. Restart the CCO service on the CCO_PRIMARY, CCO_SECONDARY, and CCO_TERTIARY servers.

    service cco stop
    service cco start
  11. Update the CCO_LB haproxy.cfg file with new CCO_TERTIARY IP.

    1. SSH into the CCO_LB instance and stop the HAproxy service.

      systemctl stop haproxy
    2. Modify the HAproxy config file as follows to replace the old IP with the new CCO_TERTIARY IP address  – modify HAProxy config file as follows.

      vi /etc/haproxy/haproxy.cfg        
                                               
      # listen on 8443 with SSL certs and loadbalance
      frontend httpsalt-in
          mode tcp
          log global
          bind *:8443
          default_backend ccos
      
      backend ccos
          mode tcp
          balance roundrobin
          option ssl-hello-chk
          server  cco1 <CCO_PRIMARY_IP>:8443
          server  cco2 <CCO_SECONDARY_IP>:8443 
          server  cco3 <CCO_TERTIARY_IP>:8443 
    3. Start the HAproxy service and check the status to ensure that it is active.

      systemctl start haproxy
      systemctl status haproxy

Appliance Process

To launch a VM using the CCO virtual appliance image, follow this procedure.

  1. Set up SSH communication between CCO_PRIMARY, CCO_SECONDARY,  and CCO_TERTIARY using root privileges.

    1. On the CCO_TERTIARY server, if the .ssh directory does not exist for the root create it using the following commands before copying the files.

      sudo -i
      mkdir -p ~/.ssh
      chmod 700 ~/.ssh
    2. Copy the files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from the CCO_SECONDARY to the same location on CCO_TERTIARY server.

    3. On the new CCO_TERTIARY server, execute the following commands.

      chmod 400 ~/.ssh/id_rsa*
      cat id_rsa.pub >> authorized_keys
    4. Verify if mutual SSH access between the CCO_PRIMARY, CCO_SECONDARY, and CCO_TERTIARY servers by running the following commands on each server. 

      ssh root@<CCO_PRIMARY>/<CCO_SECONDARY>/<CCO_TERTIARY>
  2. Copy the backup conf files on new CCO_TERTIARY.

    sudo -i
    cd /etc/sysconfig
    scp <user>@<server_ip>:/tmp/gateway.conf .
    scp <user>@< server_ip >:/tmp/cliqr-cis.conf .
    scp <user>@< server_ip >:/tmp/ filebeat  .
  3. Configure HA for CCO_TERTIARY.
    1. Invoke the CCO wizard on CCO_PRIMARY.

      sudo -i
      /usr/local/cliqr/bin/cco_config_wizard
    2. To configure HA between CCO_PRIMARY, CCO_SECONDARY, and CCO_TERTIARY instances, select Configure HA on each instance and enter the New CCO_TERTIARY IP in the Tertiary Node IP field.

    3. Verify your changes and exit the CCM configuration wizard.

  4. Restart the CCO service on the CCO_PRIMARY, CCO_SECONDARY, and CCO_TERTIARY servers.

    service cco stop
    service cco start
  5. Update the CCO_LB haproxy.cfg file with new CCO_TERTIARY IP.

    1. SSH into the CCO_LB instance and stop the HAproxy service.

      systemctl stop haproxy
    2. Modify the HAproxy config file as follows to replace the old IP with the new CCO_TERTIARY IP address.

    3. Start the HAproxy service and check the status to ensure that it is active.

      systemctl start haproxy
      systemctl status haproxy

AMQP

In these procedures:

  • user = the username

  • server_ip = the server where you want to copy the required files

Backup

Copy the  AMQP_PRIMARY and AMQP_ SECONDARY conf files from the /etc/sysconfig/ folder to your desired location using the following commands.

sudo -i
cd /etc/sysconfig
scp guacamole.conf <user>@<server_ip>:/tmp

This will copy all the files to the /tmp folder of your desired server.

You have now backed up the AMQP_PRIMARY and AMQP_SECONDARY servers.

Health Monitor

The back up scenario is not applicable in the non-HA mode as there is no known data that can be persisted.

To restore, simply launch a new VM and reconfigure the VM.


© 2017-2018 Cisco Systems, Inc. All rights reserved