Backup and Recovery in HA Mode

Recommendations

To effectively manage your CloudCenter deployment, backup your deployment on a daily basis – you can setup a cronjob to automatically perform this backup.

If you are upgrading the CloudCenter deployment, be aware that the process may differ. See the Upgrade in Non-HA Mode or Upgrade in HA Mode for additional context.

Use this procedure to backup and recover data for the following releases on a per-component basis:

  • CloudCenter 4.6.x
  • CloudCenter 4.7.x

The backup and recovery procedure is performed on a per component basis and the procedure for backup is the same for all components. Instead of repeating this procedure for each role, the procedure calls out the applicable roles for each component in the HA and standalone modes. For the non-HA modes, see Backup and Recover in Non-HA Mode.

Database (PostgreSQL)

Use this procedure for the following roles (see Component Modes and Roles for additional context).

  • MGMTPOSTGRES_MASTER
  • MGMTPOSTGRES_SLAVE

Backup

Backup your database and application (the following example uses /mnt, you can change this directory as applicable).

Backup from 4.6.x
pg_dump -U cliqr -d cliqrdb > cliqrdb.sql
# At the prompt, enter the password, cliqr

tar czvf cliqrdb.tar.gz cliqrdb.sql
rm cliqrdb.sql  

Recover

This procedure assumes that the MGMTPOSTGRES_MASTER is terminated in the HA set up.

  1. On the existing MGMTPOSTGRES_SLAVE run below command.

    rm -rf /usr/local/osmosix/etc/.HAINSTALLED 
    
    sudo sed -i".bak" "/dbslave/d" /etc/hosts
    sudo sed -i".bak" "/dbmaster/d" /etc/hosts
    sudo sed -i".bak" "/dbslave/d" /root/.ssh/known_hosts
    
    su - postgres
    psql -d cliqrdb -c "select pg_drop_replication_slot('cliqr_rep_slot1');"
    psql -d cliqrdb -c "select pg_drop_replication_slot('cliqr_rep_slot');"
    exit
    
    pcs cluster stop
    pcs cluster destroy
    service pcsd stop
    service pcsd start
  2. Launch the MGMTPOSTGRES_(new)SLAVE VM.
    1. Phase 1: Prepare Infrastructure > MGMTPOSTGRES_MASTER/SLAVE
    2. Phase 2: Configure Network Rules > MGMTPOSTGRES_MASTER/SLAVE
    3. Phase 3: Perform Network Compliance Check > MGMTPOSTGRES_MASTER/SLAVE
    4. Phase 4: Install Components > MGMTPOSTGRES_MASTER/SLAVE – Use one of the processes (Installer Process or Appliance Process provided after this section) for explicit instructions.

Installer Process

On the new MGMTPOSTGRES VM, follow this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Download the following files from software.cisco.com to the /tmp folder. See Installation Overview > Installation Download Details for additional context.

    • core_installer.bin
    • ccm-installer.jar
    • ccm-response.xml
  3. Run Core installer to setup core system components using the following commands.

    sudo –i
    cd /tmp
    chmod 755 core_installer.bin
    
    
    #Set the following only if a local package store is setup export
    CUSTOM_REPO=<http://local_package_store ip>
    
    
    ./core_installer.bin <ostype><cloudtype> mgmtpostgres
    

    For example:

    ./core_installer.bin centos7 amazon mgmtpostgres

    Syntax:

    <ostype> = centos6, centos7, rhel6, rhel7

    <cloudtype> = amazon, openstack, vmware

  4. Remove the core_installer.bin file.

    rm core_installer.bin
  5. REQUIRED: At this point, you must continue with the Appliance Process to configure the wizard properties and set up the VM.

Appliance Process

Launch an appliance using the MGMTPOSTGRES appliance file. On the new MGMTPOSTGRES_MASTER, follow this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Exchange SSH keys between the new VM and existing MGMTPOSTGRES servers.
    1. Copy the files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from the same location on the existing VM to the same location on new VM.

      If  the .ssh directory doesn’t exist on the New VM, first create it (using the following commands) before copying the files. 

      sudo –i
      mkdir –p ~/.ssh
      chmod 600 ~/.ssh
    2. On new VM, run the following commands.

      sudo –i
      chmod 400 ~/.ssh/id_rsa*
      cat id_rsa.pub >> authorized_keys
      
    3. Verify mutual SSH access between the existing and new VM by running the following command on each server.

      sudo –i
      ssh root@<NEW VM/OLD VM>
      
    1. SSH into the DB instance as a centos user.
    2. Run the following command:

      sudo -i
  3. Invoke the wizard.

    CCM Wizard Path
    /usr/local/cliqr/bin/db_config_wizard.sh
  4. Configure Postgres HA to ensure that the PostgreSQL database HA and enter the information in each field as follows:

    Database Properties

    Field

    Description

    Configure_Postgres_HA
    • Master Hostname
    The host name for the master database VM
    • Master Private IP
    The private IP address of the master database VM
    • Slave Hostname
    The host name for the slave database VM
    • Slave Private IP
    The private IP address of the slave database VM
    • VIP
      or
    • EIP

    The VIP/EIP IP for the database

    Use your mouse to select this option.

    Once the details are entered, the database server begins replication configuration between the database servers followed by HA configuration and finally presents the following status messages.

    • Configuring database for HA ...

    • Configuring database for replication

  5. Exit the configuration wizard.

  6. Go to the command line for each PostgreSQL server and enter the following command to review the status of the database and the HA connectivity:
    # pcs status

    1. Ensure that the PCSD Status for both database servers are Online.
    2. Ensure that the Daemon Status for corosync, pacemaker and pcsd are active/disabled.

 

CCM

Use this procedure for the following roles (see Component Modes and Roles for additional context).

  • CCM_SA_PRIMARY
  • CCM_SA_SECONDARY

Backup

This procedure assumes that the CCM_SA_PRIMARY is terminated in a HA setup.

  1. To backup your database and application, issue the following commands.

    The following example uses /mnt, you can change this directory as applicable.

    Backup from 4.6.x
    NOW=$(date +"%Y%m%d")
    bakdir="/mnt/bak/$NOW"
    mkdir -p $bakdir
    cd $bakdir
     
    cp -r /usr/local/tomcat/webapps/* . 

Recover

This procedure assumes that the CCM_SA_PRIMARY is terminated in the HA set up.

On the existing CCM_ SA_SECONDARY, perform this procedure.

  1. Navigate to /home/cliqruser.

    cd /home/cliqruser
  2. Remove the .unison folder:

    rm –rf .unison
  3. Verify and delete any running cron jobs containing the name unison.
  4. Navigate to osmosix/etc folder and delete the harole file.

    rm osmosix/etc/harole
  5. To launch the CCM_(new)SA_PRIMARY VM, follow this procedure.

    1. Phase 1: Prepare Infrastructure > CCM_SA_PRIMARY

    2. Phase 2: Configure Network Rules > CCM_SA_PRIMARY

    3. Phase 3: Perform Network Compliance Check > CCM_SA_PRIMARY

    4. Phase 4: Install ComponentsCCM_SA_PRIMARY – Use one of the processes (Installer Process or Appliance Process provided after this section) for explicit instructions.

Installer Process

On the new CCM_SA_PRIMARY VM, perform this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Download the CCM installer files to the /tmp folder:

    1. core_installer.bin
    2. ccm-installer.jar
    3. ccm-response.xml
  3. Run Core installer to setup core system components using the following commands.

    sudo –i
    cd /tmp
    chmod 755 core_installer.bin
    
    
    #Set the following only if a local package store is setup export
    CUSTOM_REPO=<http://local_package_store ip>
    
    
    ./core_installer.bin <ostype><cloudtype> ccm_sa

    For example:

    ./core_installer.bin centos7 amazon ccm_sa

    Syntax:

    <ostype> = centos6, centos7, rhel6, rhel7

    <cloudtype> = amazon, openstack, vmware

  4. Remove the core_installer.bin file.

    rm core_installer.bin
  5. Log off and log back in as the root user to ensure JAVA Home is set
  6. Modify the ccm-response.xml file as follows:

    Response file option

    Value

    Notes

    <entry key="db_host" value="localhost"/> 

    Replace localhost with VIP (MGMTPOSTGRES_VIP_IP)

    Required for the CCMs  to connect to the master Postgres database

  7. Run the appliance installer to setup CCM.

    java -jar ccm-installer.jar ccm-response.xml
  8. Reboot the CCM VM.
  9. REQUIRED: At this point, you must continue with the Appliance Process to configure the wizard properties and set up the VM.

Appliance Process

Launch a VM using the CCM_SA appliance image. To configure the CCM wizard properties, follow this procedure.

  1. On the CCM_SA_PRIMARY server, if the .ssh directory does not exist for the cliqruser, create it using the following commands before copying the files. 

    su - cliqruser
    mkdir –p ~/.ssh
    chmod 600 ~/.ssh
  2. Copy the files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from the CCM_SA_SECONDARY to the same location on CCM_SA_PRIMARY server. 

  3. On the new CCM_SA_PRIMARY server, execute the following commands.

    su - cliqruser
    chmod 400 ~/.ssh/id_rsa*
    cat id_rsa.pub >> authorized_keys
  4. Verify mutual SSH access between the CCM_SA_PRIMARY and CCM_SA_PRIMARY servers by running below command on each server.

    su – cliqruser
    ssh cliqruser@<CCM_SA_PRIMARY_IP/ CCM_SA_ SECONDARY_IP>
  5. On the CCM_SA_ SECONDARY, invoke the CCM configuration wizard and configure the properties. See CCM_SA_SECONDARY – Run Appliance Install for additional details.

    /usr/local/cliqr/bin/ccm_config_wizard.sh 
  6. Navigate to the Configure HA screen and enter the information in each field as follows:

    CCM HA Properties

    Field

    Description

    Configure_HA
    • Primary Node Private IP

    The private IP address of the primary server (CCM_SA_ SECONDARY).

    • Secondary Node Private IP

    The private IP address of the secondary server (CCM_SA_ PRIMARY).

    • Public DNS
    The public DNS of the slave database VM
    • Hazelcast IP
    • Comma separated IP addresses of the CCMs (CCM_SA_ SECONDARY, CCM_SA_PRIMARY).

    • Used internally by the CloudCenter platform.
    • External URL

    The VIP/EIP IP for the database

    Use your mouse to select this option.

    Once the details are entered, the database server begins replication configuration between the database servers followed by HA configuration and finally presents the following status messages.

    1. Configuring database for HA ...

    2. Configuring database for replication

  7. Exit the configuration wizard.

  8. Restart both servers.

  9. Update the CCM_LB with the new IP address of the new CCM_SA_PRIMARY server.
    1. SSH into the VM instance using the key pair that you used to launch the VM.

      systemctl stop haproxy
    2. Modify the HAProxy config file as follows to replace the old IP with the new IP address.

      vi /etc/haproxy/haproxy.cfg        
                                                            
      # configuration to listen on 443 with SSL certs and loadbalance
      frontend https-in
          mode http
          log global
          bind *:443 ssl crt /etc/haproxy/mgmtserver.pem ca-file /etc/haproxy/ca.pem
          default_backend ccms
      
      # configuration to listen on 8443 with SSL certs and loadbalance
      frontend httpsalt-in
          mode tcp
          bind *:8443
          default_backend nodes
      
      backend ccms
          balance roundrobin
          mode    http
          log global
          option httplog
          cookie SVR insert preserve nocache
          server  ccm1 < CCM_SA_PRIMARY_IP >:443 check cookie ccm1 ssl verify none
          server  ccm2 < CCM_SA_SECONDARY_IP >:443 check cookie ccm2 ssl verify none
      
      backend nodes
          mode tcp
          balance roundrobin
          option ssl-hello-chk
          server  ccm1 < CCM_SA_PRIMARY_IP >:8443 check
          server  ccm2 < CCM_SA_SECONDARY_IP >:8443 check
      


    3. Start the HAProxy service and check the status to ensure that it is active.

       

      systemctl start haproxy
      systemctl status haproxy 
      

AMQP

Use this procedure for the following roles (see Component Modes and Roles for additional context).

  • AMQP_PRIMARY
  • AMQP_SECONDARY

Backup

Backup the webapp folder containing the exploded war files to a backup folder (the following example uses /mnt, you can change this directory as applicable). This backup only applies to the Guacamole server, not the AMQP server.

NOW=$(date +"%Y%m%d")
bakdir="/mnt/bak/$NOW"
mkdir -p $bakdir
cd $bakdir

cp -r /usr/local/tomcatgua/webapps/* .

Recover

This procedure assumes that the AMQP_PRIMARY is terminated in a HA setup.

Launch the CCM_(new)SA_PRIMARY VM.

  1. Phase 1: Prepare InfrastructureAMQP_PRIMARY
  2. Phase 2: Configure Network Rules > AMQP_PRIMARY
  3. Phase 3: Perform Network Compliance Check > AMQP_PRIMARY
  4. Phase 4: Install ComponentsAMQP_PRIMARY – Use one of the processes (Installer Process or Appliance Process provided after this section) for explicit instructions.

Installer Process

On the existing AMQP_SECONDARY, perform this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Download the AMQP installer files to the /tmp folder:

    1. core_installer.bin
    2. cco-installer.jar
    3. conn_broker-response.xml
  3. Run Core installer to setup core system components using the following commands.

    sudo –i
    cd /tmp
    chmod 755 core_installer.bin
    
    
    #Set the following only if a local package store is setup export CUSTOM_REPO=<http://local_package_store ip>
    
    
    ./core_installer.bin <ostype><cloudtype> rabbit

    For example:

    ./core_installer.bin centos7 amazon rabbit

    Syntax:

    <ostype> = centos6, centos7, rhel6, rhel7

    <cloudtype> = amazon, azure, azurerm, azurepack, google,  opsource, openstack, softlayer, vmware, vcd

  4. Remove the core_installer.bin file.

    rm core_installer.bin 
  5. Log off and log back in as the root user to ensure JAVA Home is set.

    exit
    sudo -i
  6. Run the appliance installer to setup AMQP.

    java -jar cco-installer.jar conn_broker-response.xml
  7. Reboot the AMQP VM.
  8. REQUIRED: At this point, you must continue with the Appliance Process to configure the wizard properties and set up the VM.

Appliance Process

Launch a VM using the AMQP appliance image. To configure the GUA wizard properties, follow this procedure.

  1. Copy the cookie from AMQP_SECONDARY to AMQP_PRIMARY node of location /var/lib/rabbitm.  

    chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
    chmod 400 /var/lib/rabb­itmq/.erlang.cookie
    systemctl enable rabbitmq-server.service
    sudo service rabbitmq-server start
  2. Detach AMQP servers using the following command.

    rabbitmq-server –detached
    
  3. On the new AMQP_PRIMARY server, execute the following commands.

    rabbitmqctl stop_app       
    rabbitmqctl join_cluster rabbit@<AMQP_PRIMARY_HOSTNAME>
    rabbitmqctl start_app
  4. On the new AMQP_SECONDARY server, execute the following commands.

    rabbitmqctl stop_app       
    rabbitmqctl join_cluster rabbit@<AMQP_SECONDARY_HOSTNAME>
    rabbitmqctl start_app
  5. Check AQMP cluster status on both the servers.

    rabbitmqctl cluster_status
    
  6. Set the mirroring policy by issuing below command in one of the server.

    rabbitmqctl
    set_policy ha-all "^cliqr" '{"ha-mode":"all"}' -p /cliqr
    
  7. Verify the policy change using the following command.

    rabbitmqctl list_policies –p /cliqr
    
  8. Modify the rabbitmq config file on both servers by adding the following line to rabbit array.

    {cluster_partition_handling,autoheal}
    

    For example:

    [{rabbit,
       [{cluster_partition_handling, autoheal},
          {ssl_listeners, [5671]},
          {handshake_timeout, 300000},
          {ssl_handshake_timeout, 300000},
          {ssl_options,
             [{cacertfile, "/etc/rabbitmq/certs/cacert.pem"},
              {certfile, "/etc/rabbitmq/certs/cert.pem"},
              {keyfile, "/etc/rabbitmq/certs/key.pem"},
              {verify, verify_peer},
              {fail_if_no_peer_cert, false}]}]}]
  9. Restart rabbitmq server on the both the servers.

    /etc/init.d/rabbitmq-server restart
    
  10. Update the AMQP_LB with the new IP address of the new AMQP_PRIMARY server.
    1. SSH into the VM instance using the key pair that you used to launch the VM.

      systemctl stop haproxy
    2. Modify the HAProxy config file as follows to replace the old IP with new

      vi /etc/haproxy/haproxy.cfg
      
      # configuration to listen on 443 and loadbalance 
      
      frontend quacs-in
      	mode tcp
      	log global	
      	bind *:443
      	default_backend amqps
      
      backend quacs
          mode tcp
          balance roundrobin
          option ssl-hello-chk   
          server guac1 <GUAC_PRIMARY>:443 check
          server guac2 <GUAC_SECONDARY>:443 check
      


    3. Start the HAProxy service and check the status, it should be active

       

      systemctl start haproxy
      systemctl status haproxy 
      
    1. Configure the CCO and CCM properties.

      Write this down for future reference!

      Write down the Field details in a printed version of the Your Notes section for later use.
    2. Configure the properties for the CCM and CCO VMs:

      GroupPossible IP Addresses

      CCM_Info

      CCM Host:
      CCM_IP or  CCM_SA_IP or CCM_LB_IP
      CCO_InfoCCO Host:
      CCO_IP or  CCO_LB_IP
    3. Verify your changes and Exit the GUA configuration wizard.

  11. Exit the configuration wizard.

  12. Restart the server.

CCO

Use this procedure for the following roles (see Component Modes and Roles for additional context).

  • CCO
  • CCO_PRIMARY
  • CCO_SECONDARY
  • CCO_TERTIARY
  • CCO_LB

Backup

Backup the exploded war files to a backup folder (the following example uses /mnt, you can change this directory as applicable).

NOW=$(date +"%Y%m%d")
bakdir="/mnt/bak/$NOW"
mkdir -p $bakdir
cd $bakdir
 
cp -r /usr/local/tomcat/webapps/ROOT .

Recover

This procedure assumes that the:

  • CCO_TERTIARY is terminated in a HA setup.
  • Node1, Node2, Node 3 are part of CCO HA.

  • Node3 is down.

  • Node4 is the new Node

To launch the CCM_(new)SA_PRIMARY VM, follow this procedure.

  1. Phase 1: Prepare Infrastructure > CCO_PRIMARY/SECONDARY/TERTIARY
  2. Phase 2: Configure Network Rules > CCO_PRIMARY/SECONDARY/TERTIARY
  3. Phase 3: Perform Network Compliance Check > CCO_PRIMARY/SECONDARY/TERTIARY
  4. Phase 4: Install ComponentsCCO_PRIMARY/SECONDARY/TERTIARY – Use one of the processes (Installer Process or Appliance Process provided after this section) for explicit instructions.

Installer Process

On the existing CCO_SECONDARY VM, perform this procedure.

  1. SSH into the VM instance using the key pair that you used to launch the VM.
  2. Download the CCO installer files to the /tmp folder:

    1. core_installer.bin
    2. cco-installer.jar
    3. cco-response.xml
  3. Run Core installer to setup core system components using the following commands.

    sudo –i
    cd /tmp
    chmod 755 core_installer.bin
    
    
    #Set the following only if a local package store is setup export
    CUSTOM_REPO=<http://local_package_store ip>
    
    
    ./core_installer.bin <ostype><cloudtype> cco

    For example:

    ./core_installer.bin centos7 amazon cco

    Syntax:

    <ostype> = centos6, centos7, rhel6, rhel7

    <cloudtype> = amazon, openstack, vmware

  4. Remove the core_installer.bin file.

    rm core_installer.bin /root/cliqr_modules.log
  5. Log off and log back in as the root user to ensure JAVA Home is set

  6. Run the appliance installer to setup CCO.

    java -jar ccm-installer.jar cco-response.xml
  7. Reboot the CCO VM.
  8. REQUIRED: At this point, you must continue with the Appliance Process to configure the wizard properties and set up the VM.

Appliance Process

Launch a VM using the CCO appliance image. To launch the CCO wizard, follow this procedure.

    1. SSH into the CCO instance as a centos user.
    2. Edit the /etc/mongod.conf file and add the following line. 

      vi /etc/mongod.conf
      replSet=setname
    3. Modify bind_ip as follows.

      bind_ip=0.0.0.0
      
    4. Restart mongod.

      service mongod restart
      
  1. CCO_LB with the new IP address of the new CCO_PRIMARY server.
    1. SSH into the VM instance using the key pair that you used to launch the VM.

      systemctl stop haproxy
    2. Modify the HAProxy config file as follows to replace the old IP with the new IP address.

      vi /etc/haproxy/haproxy.cfg        
                                               
      # listen on 8443 with SSL certs and loadbalance
      frontend httpsalt-in
          mode tcp
          log global
          bind *:8443
          default_backend ccos
      
      backend ccos
          mode tcp
          balance roundrobin
          option ssl-hello-chk
          server  cco1 <CCO_PRIMARY_IP>:8443
          server  cco2 <CCO_SECONDARY_IP>:8443 
          server  cco2 <CCO_TERTIARY_IP>:8443 
      


    3. Start the HAProxy service and check the status, to ensure that it is active

       

      systemctl start haproxy
      systemctl status haproxy 
      

Health Monitor

The back up scenario is not applicable in the non-HA mode as there is no known data that can be persisted.

To restore, simply launch a new VM and reconfigure the VM.

 

  • No labels