// removed jquery ui css and js

Cross-Region Disaster Recovery

Overview

CCM Disaster Recover (DR) in a new or existing CloudCenter deployment requires:

  • New CCM and PostgreSQL (MGMTPOSTGRES) instances to be configured as standby servers on different regions.

  • These instances must be in sync with the main setup.

  • Synchronization is configured from the existing CCM setup to the CCM DR using Unison.

  • MGMTPOSTGRES DR is configured using a trigger-based master-slave replication process.

  • The MGMTPOSTGRES DR node will be recovering the data from the master node in case of an existing HA setup.

General Requirements:

  • Ports:

    PortDescription
    22Open (2-way) between the existing CCM(s) to the DR CCM and vice-versa.
    Open the password-less SSH access from:
    •  The CCM in the main setup to the DR CCM.

    •  The MGMTPOSTGRES master to the DR MGMTPOSTGRES.

    5432Open (2-way) between the existing MGMTPOSTGRES to the DR MGMTPOSTGRES and vice-versa.

    See Firewall Rules Overview for additional details.

  • SSL Certs:

    • Identical SSL certificates for the existing CCM and the DR CCM.

    • Only required after the failover for the CCO communication with DR CCM

    • Ensure to update when configuring DR CCM.

  • DNS Configuration:

    • Configure the existing CCM (CCM Load Balancer in case of existing setup) with a valid DNS name. In case of disaster, the DNS name points to the DR CCM and CCO in another region and can continue to communicate with the DR CCM with out any reconfiguration.
      OR

    • You can manage this scenario by having DNS entry in the hosts file of the CCO server
      OR

    • Reconfigure the Orchestrator in the DR CCM after the failover.

  • DR CCM: Stop the DR CCM before configure DR (using the systemctl stop ccm).

CCM HA DR Setup

HA CCM DR Architecture

CCM Standalone DR Setup

Standalone CCM DR Architecture

CCM Configuration

Invoke the CCM wizard in the primary CCM to configure synchronization between the both CCM servers and the CCM DR. This step involves file synchronization of applications, services, tenant logos, and so forth using Unison.

  1. Invoke the CCM wizard as a root user on the primary CCM (CCM1).

    /usr/local/cliqr/bin/ccm_config_wizard.sh
  2. Select Config_CCM_DR to configure CCM DR.

    Wizard MenuFieldDescription

    Config_CCM_DR

    Primary Node Private IPThe DNS (or IP address) of the CCM server.
    Provide this information for the primary CCM server or the standalone CCM.
    Secondary Node Private IPThe DNS (or IP address) of the CCM secondary server, in case of an HA environment.
    DR Public IP/DNS NameThe public IP reachable via Port 22 for both the primary and secondary servers.
  3. Select Yes to save your changes.

  4. Exit the CCM wizard.                                                                                          

PostgreSQL DR Setup

Invoke the MGMTPOSTGRES database wizard in the master database to configure trigger-based master-slave replication between the master MGMTPOSTGRES and the DR MGMTPOSTGRES with which the data in the database is replicated.

  1. Invoke the database wizard as a root user on the master MGMTPOSTGRES.

    /usr/local/cliqr/bin/db_configwizard.sh
  2. Select Config_DB_DR to configure MGMTPOSTGRES DR.

  3. Select Configure_DB_DR_failover.

  4. Select Yes to save your changes.

  5. Exit the database wizard.

DR CCM Restart

After a MGMTPOSTGRES failover, restart the DR CCM services by executing the following command on the CCM DR server.

systemctl restart ccm

To ensures that the CCM is ready to use after the service is restarted, view the log file message in the corresponding log file for the CCM service (/usr/local/cliqr/logs/mgmtserver.log):

2018-04-09 23:55:20,470 INFO  mgmtserver.MgmtserverServer [main]  - Started MgmtserverServer in 30.553 seconds (JVM running for 32.062)

DR CCM DNS Change

Change the DNS entry of the master CCM over to the DR CCM, so that the CCOs in the other regions can start communicating with the DR CCM and the users can access the CCM using the same DNS.

Restoring an Existing Setup after a Failover

The CCM server restore process occurs automatically if the Public IP addresses of Primary/Secondary CCMs are not changed.

If the IP address did change after the disaster, then  re-run the ccm_config_wizard.sh and configure it again to change the IP addresses, this ensures auto-recovery after a disaster.

See the CCM Configuration section above for additional details.

MGMTPOSTGRES Restore

In case of a HA setup, you may need to address the following additional considerations:

  • If the IP address changes or an error occurs in the pcs status command (only in case of HA setup), fix it first before you run a restore. Without this fix, the restore will not succeed.

  • Execute the pcs status command in the MGMTPOSTGRES Master. If it has any issues in the MGMTPOSTGRES consistency or has an IP address change, resolve those issues and execute the“pcs resource cleanup command, wait for 1 minute, and then execute the pcs resource cleanup mgmtpostgres command.

  • Check the status again by issuing the pcs status command again to see if it is successful this time around.

Invoke the MGMTPOSTGRES database wizard to restore the MGMTPOSTGRES slave.

  1. Invoke the database wizard as a root user on the slave MGMTPOSTGRES.

    /usr/local/cliqr/bin/db_config__wizard.sh
  2. Select Config_DB_DR to restore the MGMTPOSTGRES slave.

    Wizard MenuFieldDescription
    Config_DB_DR_retoreMaster Node Public IPThe DNS (or IP address) of the slave MGMTPOSTGRES server.
    The Public IP is required to communicate with the master database in an HA environment.
    Master Node Private IPThe DNS (or IP address) of the slave MGMTPOSTGRES server.
    Slave Node Public IPThe DNS (or IP address) of the master MGMTPOSTGRES server.
    The Public IP is required to communicate with the slave database in an HA environment.
    Slave Node Private IPThe DNS (or IP address) of the master MGMTPOSTGRES server.
    DR Public IP/DNS NameThe public IP reachable via Port 22 for both the master and slave servers.
  3. Select Yes to save your changes.

  4. Exit the database wizard.  


  • No labels
© 2017-2019 Cisco Systems, Inc. All rights reserved