// removed jquery ui css and js

Monitor and Recover PostgreSQL

Overview

You can monitor and recover the PostgreSQL database if running in HA mode. The utility runs on the CCM server and monitors the network and operational health of the PostgreSQL.

Prerequisites

Before configuring the monitor and recover utility in your CloudCenter HA environment, verify the following requirements:

  1. You have exchanged the SSH keys between the CCM and the PostgreSQL components. See Exchange SSH Keys between the CCM and PostgreSQL Components for additional details.

  2. Verify that you have installed the Python module Paramiko, if it is not already installed by the core-installer. This feature helps monitor PostgreSQL VM health and assist with the recovery process in HA environments.

    The PostgreSQL Database VMs should preferably not be moved after installation to ensure a safe recovery (in case of environmental factors).


    To use this feature, follow this process:

    1. Ensure that the PostgreSQL HA setup is in a healthy state – before the feature is enabled.

    2. Restore the cluster to a healthy state (have both DB nodes in working state)

    3. Upgrade to CloudCenter 4.10.

  3. Verify that you have installed CloudCenter 4.10 as this package includes the required scripts for this procedure. See Download Packages for additional details.

Exchange SSH Keys between the CCM and PostgreSQL Components

This procedure assumes that you have already performed the following tasks:

Verify mutual SSH access between the CCM_SA_PRIMARY server (and the CCM_SA_SECONDARY server) with the MGMTPOSTGRES_MASTER and MGMTPOSTGRES_SLAVE servers by running the following commands.

#From the CCM_SA_PRIMARY:
ssh root@<MGMTPOSTGRES_MASTER/MGMTPOSTGRES_SLAVE>

#From the CCM_SA_SECONDARY:
ssh root@<MGMTPOSTGRES_MASTER/MGMTPOSTGRES_SLAVE>


Monitor PostgreSQL HA

Invoke the CCM wizard in the primary CCM to monitor both CCM servers.

  1. Invoke the CCM wizard as a root user on either the primary or secondary CCM server.

    /usr/local/cliqr/bin/ccm_config_wizard.sh
  2. Select Config_HA_MR to configure CCM DR.

    Wizard MenuFieldDescription

    Config_HA_MR

    Monitor_DB_HAThe monitor utility performs a health check of the DB at every 'n' interval, and records the status under /var/log. In case of any discrepancies, the monitor sends an email to the CCM admin to notify the problem
    Master Node Private IPRequired. The DNS (or IP address) of the master MGMTPOSTGRES server.
    Slave Node Private IPRequired. The DNS (or IP address) of the slave MGMTPOSTGRES server.
    VIP IP/EIP IP/DNS

    Required. The VIP/EIP for the master database or slave database. See Phase 1: Prepare Infrastructure > Cloud Nuances for additional context.

    When you configure the High Availability Properties, you would have configured the VIP/EIP address for the db_config_wizard. Similarly, you must provide the EIP/VIP address for the CCM_SA_PRIMARY and the CCM_SA_SECONDARY servers.

    Interval (mins)Optional. Default = 1 minute.
    The interval in minutes for the monitor utility to probe the database.
    SMTP SERVERRequired. The IP address or DNS for your SMTP mail server.
    SMTP PORTRequired. The open port for your SMTP mail server.
    SMTP SERVER USERRequired. The username for the person administering your SMTP mail server.
    SMTP SERVER PASSWORDRequired. The password for the person administering your SMTP mail server.
    TO EMAIL ADDRRequired. The email address of the CCM admin who should be receiving this email notification. The email notification is only sent if the monitor utility detects a unhealthy database.
  3. Select Yes to save your changes.

  4. Exit the CCM wizard after configuring the remaining CCM Wizard settings.

     Configure Remaining CCM Wizard Settings

    Configure CCM Wizard Properties

    To configure the CCM wizard properties, follow this procedure.

    1. Invoke the CCM wizard as a root user (see Virtual Appliance Process > Cloud-Specific Setup Details for a sample setup).

      CCM Wizard Path
      /usr/local/cliqr/bin/ccm_config_wizard.sh
    2. Configure the server properties.

      Write this down for future reference!

      Write down the Field details in a printed version of the Installation Approach > Your Notes section for later use.

      Wizard Menu

      Field

      Description

      Server_Info – Configure Server Info (Required)

      Mgmtserver DNS Name

      DNS (or IP address) of the CCM management server.

      If you are configuring the HA environment, enter the CCM_LB DNS (or IP) in this field.

      DB – Configure DatabaseIP or Hostname

      DNS or IP of the database.

      DB Username and Password

      The following credentials are pre-populated:

      Default username = cliqr (cannot be changed)

      Default password = cliqr (must be changed)

      Be sure to change the default password immediately after your first login. See PostgreSQL Password for additional context.

      ELK_Info –  Configure Log Collector InfoELK Host

      Specify the IP address for the Log Collector host.

      Elasticsearch PortDisplays 8881 by default.
      Kibana PortDisplays 8882 by default.
      ELK UserThe default ELK Username = logreader.
      ELK PasswordThe default ELK Password is re@d0nly (zero between d and n) (change this password after the initial login – see Download Log File for additional context).
      Host Identifier

      A Unique ID for the server – be sure to prefix the unique identifier with CCM_ for example, CCM_1
      If not set, the CloudCenter platform uses the CCM server date.

      The Host Identifier cannot contain capital letters for both CCM and CCO configurations.

      Host Identifier List

      Only applies to environments using the HA mode – provide a list of comma separated unique host identifiers for all Log Collector hosts in a HA setup = for example, CCM_1,CCM_2,myCCM.

      The Host Identifier List cannot contain capital letters for both CCM and CCO configurations.

      In an environment operating in HA mode, if you have two CCM instances with unique IDs configured as CCM_1,CCM_2 in their respective server.properties file, then this property should state CCM_1,CCM_2 in both CCM instances. Each CCM must be aware of the unique ID of the other CCM(s) when in HA mode.

      Custom Certs Menu
      Generate_CertsTo generate new certificates for CloudCenter components. See Certificate Authentication > Generate and Update the certs.zip File on the CCM for additional context.
      Update_CertsTo update certificates for CloudCenter components. See Certificate Authentication > Generate and Update the certs.zip File on the CCM for additional context.
    3. Exit the CCM configuration wizard.

    4. Select Yes, to restart the CCM server and corresponding CloudCenter services.

    You have successfully installed the CCM instance! You can now proceed to the next step:

    • Configure the Log Collector details in the CCM wizard's Configure Log Collector Info menu.

    • If you are installing the Enterprise Service Bus (ESB), do so at this point.

    Back to: CCM (Required)

                                                                                            

Restore PostgreSQL HA

Invoke the CCM wizard in the primary CCM to restore the PostgreSQL database on both CCM servers.

  1. Invoke the CCM wizard as a root user on either the primary or secondary CCM server.

    /usr/local/cliqr/bin/ccm_config_wizard.sh
  2. Select Config_HA_MR to configure CCM DR. The default values are inherited from the monitor utility.

    Wizard MenuFieldDescription

    Config_HA_MR

    Restore_DB_HA

    In case of an issue the CCM admin, can run the second part of the utility to recover the database, resolve the issue, and restore the DB back to an operational state.

    Master Node Private IPRequired. The private IP address of the MGMTPOSTGRES_MASTER server.
    Slave Node Private IPRequired. The private IP address of the MGMTPOSTGRES_SLAVE server.
    VIP IP/EIP IP/DNS

    Required.The VIP/EIP for the master database and slave database. See Phase 1: Prepare Infrastructure > Cloud Nuances for additional context.

    When you configure the High Availability Properties, you would have configured the VIP/EIP address for the db_config_wizard. Similarly, you must provide the EIP/VIP address for the CCM_SA_PRIMARY and the CCM_SA_SECONDARY servers.

  3. Select Yes to save your changes.

  4. Exit the CCM wizard after configuring the remaining CCM Wizard settings.

     Configure Remaining CCM Wizard Settings

    Configure CCM Wizard Properties

    To configure the CCM wizard properties, follow this procedure.

    1. Invoke the CCM wizard as a root user (see Virtual Appliance Process > Cloud-Specific Setup Details for a sample setup).

      CCM Wizard Path
      /usr/local/cliqr/bin/ccm_config_wizard.sh
    2. Configure the server properties.

      Write this down for future reference!

      Write down the Field details in a printed version of the Installation Approach > Your Notes section for later use.

      Wizard Menu

      Field

      Description

      Server_Info – Configure Server Info (Required)

      Mgmtserver DNS Name

      DNS (or IP address) of the CCM management server.

      If you are configuring the HA environment, enter the CCM_LB DNS (or IP) in this field.

      DB – Configure DatabaseIP or Hostname

      DNS or IP of the database.

      DB Username and Password

      The following credentials are pre-populated:

      Default username = cliqr (cannot be changed)

      Default password = cliqr (must be changed)

      Be sure to change the default password immediately after your first login. See PostgreSQL Password for additional context.

      ELK_Info –  Configure Log Collector InfoELK Host

      Specify the IP address for the Log Collector host.

      Elasticsearch PortDisplays 8881 by default.
      Kibana PortDisplays 8882 by default.
      ELK UserThe default ELK Username = logreader.
      ELK PasswordThe default ELK Password is re@d0nly (zero between d and n) (change this password after the initial login – see Download Log File for additional context).
      Host Identifier

      A Unique ID for the server – be sure to prefix the unique identifier with CCM_ for example, CCM_1
      If not set, the CloudCenter platform uses the CCM server date.

      The Host Identifier cannot contain capital letters for both CCM and CCO configurations.

      Host Identifier List

      Only applies to environments using the HA mode – provide a list of comma separated unique host identifiers for all Log Collector hosts in a HA setup = for example, CCM_1,CCM_2,myCCM.

      The Host Identifier List cannot contain capital letters for both CCM and CCO configurations.

      In an environment operating in HA mode, if you have two CCM instances with unique IDs configured as CCM_1,CCM_2 in their respective server.properties file, then this property should state CCM_1,CCM_2 in both CCM instances. Each CCM must be aware of the unique ID of the other CCM(s) when in HA mode.

      Custom Certs Menu
      Generate_CertsTo generate new certificates for CloudCenter components. See Certificate Authentication > Generate and Update the certs.zip File on the CCM for additional context.
      Update_CertsTo update certificates for CloudCenter components. See Certificate Authentication > Generate and Update the certs.zip File on the CCM for additional context.
    3. Exit the CCM configuration wizard.

    4. Select Yes, to restart the CCM server and corresponding CloudCenter services.

    You have successfully installed the CCM instance! You can now proceed to the next step:

    • Configure the Log Collector details in the CCM wizard's Configure Log Collector Info menu.

    • If you are installing the Enterprise Service Bus (ESB), do so at this point.

    Back to: CCM (Required)


  • No labels
© 2017-2019 Cisco Systems, Inc. All rights reserved