Upgrade CloudCenter in HA Mode                                            

Be sure to review Upgrade Overview before starting this procedure!

  1.  Bundle Server Upgrade

    Bundle Store Installation (Optional)

    The Bundle Store is a repository that contains the agent and service bundles. You must download and install these bundles on the VMs (worker or application VMs) launched by CloudCenter as part of the application orchestration process.

    • With Internet Connection: The default bundle store is hosted at (cdn.cliqr.com) and CloudCenter deployments where the Application VMs have access to the internet can use the default bundle store.
    • Isolated Environments: For environments where connectivity to the internet is restricted, create a local bundle store and register it with the CCO(s).

    Configure a Bundle Store

    To configure a bundle store, follow this procedure.

    Isolated Environments

    For environments where connectivity to the internet is restricted, create a local bundle store and register it with the CCO(s).

    1. Set up the HTTP server.

      This setup assumes Apache2 on a CentOS server. If you use a different OS/HTTP server, adjust the following commands accordingly.

    2. Locate the document root of the HTTP server

      1. Change directory to /etc/httpd/conf

      2. Check httpd.conf for site-available/default files.

      3. Locate the DocumentRoot in one of these configuration files. Typically, it will be either /var/www or /var/www/html.

    3. Change directory to DocumentRoot directory. 

      cd <DocumentRoot>
    4. Create a directory to reflect the CloudCenter release you are installing (for example, 4.6.0) and create a bundle directory under the release folder level.

      mkdir release-<CloudCenter Version>
      cd release-<CloudCenter Version>
      mkdir bundle
    5. Change to the bundle directory.

      cd bundle
    6. Copy or download the bundle_artifacts.zip

    7. Unzip the bundle_artifacts.zip file

      unzip bundle_artifacts.zip
    8. Update the configuration files to set the repository location.

      chmod 755 ./set_bundle_location.sh
      ./set_bundle_location.sh http://your_HTTP_server/release-<CloudCenter Version>/bundle/

      For example:

      ./set_bundle_location.sh http://cdn.cliqr.com/release-4.6.0/bundle/

      If you do not include the trailing “/” in the command, you will receive errors at some point in the process.

    You have successfully configured the bundle store! You can now proceed to the next step.

     

    Back to Phase 4: Install Components

     

     

     

  2.  CCM-DB Upgrade in HA Mode

    CCM and MGMTPOSTGRES Upgrade in HA Mode

    Overview

    This section provides details on upgrading your CloudCenter deployment in HA mode. 

    Prerequisites

    Be aware that the CCM and DB servers will be offline during the upgrade process. Schedule some down time for your enterprise before starting this process.

    Verify these requirements before you begin the upgrade process:

    • Review the information provided in the Upgrade Overview section and validate the following requirements for the release to which you are upgrading:

      • Is an upgrade path available?
      • Is the core_upgrade.bin file required?
    • See HA Best Practices for HA considerations.

    • For each CCM (CCM primary and CCM secondary) and each PostgreSQL (DB master and DB slave) instance that must be upgraded, verify the following prerequisites:

      • Ensure that a version file (/usr/local/osmosix/etc/version) exists in both CCMs to be upgraded.

      • Verify that the version file contains the correct version number (for example, if your current CloudCenter release version is 4.7.2, ensure that the corresponding version value is 4.7.2).

      • See the corresponding release notes for release-specific information on the CloudCenter version to which you are upgrading. For example, the CloudCenter 4.6.0 Release Notes.

    Backup Database

    Backup your database and application (the following example uses /mnt, you can change this directory as applicable).

    Backup from 4.5.x
    NOW=$(date +"%Y%m%d")
    bakdir="/mnt/bak/$NOW"
    mkdir -p $bakdir
    cd $bakdir
    
    cp -r /usr/local/tomcat/webapps/* . 
    
    mysqldump -p -u root osmosixdb > osmosixdb.sql
    # At the prompt, enter the password, osmosix
    tar czvf osmosixdb.tar.gz osmosixdb.sql
    rm osmosixdb.sql

    Osmosix users do not have permission to use the -R option. CloudCenter uses the GetVendorList routine. To backup this routine along with the rest of the database, you must provide the -R option using your root user credentials.

    Backup from 4.6.x
    NOW=$(date +"%Y%m%d")
    bakdir="/mnt/bak/$NOW"
    mkdir -p $bakdir
    cd $bakdir
    
    cp -r /usr/local/tomcat/webapps/* . 
    
    pg_dump -U cliqr -d cliqrdb > cliqrdb.sql
    # At the prompt, enter the password, cliqr
    tar czvf cliqrdb.tar.gz cliqrdb.sql
    rm cliqrdb.sql

    Download the Upgrade Packages

    Download package files:

    See Installation Overview to understand the required components and the installation options.

    See Installer Overview to understand the types of files.

    1. SSH into the VM instance designated for this component by using the key pair that you used to launch the VM.
    2. Download the following required files for this component from software.cisco.com to the /tmp folder on that VM:

      • ccm-installer.jar
      • ccm-response.xml

      • core_upgrade.bin

        Ensure (by reviewing Upgrade Overview) that this file is required for your release upgrade path.

    Select Your Upgrade Scenario

    Your upgrade process differs depending on your instance setup. Ascertain the following considerations before you begin the CloudCenter upgrade.

    Scenario
    Existing Scenario
    Upgrade Scenario
    Follow the Process in this Section

    Scenario 1

    Use if upgrading from CloudCenter 4.5.x to 4.6.x


     

    • Instance 1 = CCM primary
    • Instance 2 = CCM secondary
    • Instance 3 = MySQL
    • Instance 4 = MySQL
    • Instance 1 = CCM primary
    • Instance 2 = CCM secondary
    • Instance 3 = PostgreSQL master
    • Instance 4 = PostgreSQL slave
    Double CCM and Double PostgreSQL – A

    Scenario 2

    Use if upgrading from CloudCenter 4.6.0 to a later version

    • Instance 1 = CCM primary
    • Instance 2 = CCM secondary
    • Instance 3 = PostgreSQL
    • Instance 4 = PostgreSQL
    • Instance 1 = CCM primary
    • Instance 2 = CCM secondary
    • Instance 3 = PostgreSQL master
    • Instance 4 = PostgreSQL slave
    Double CCM and Double PostgreSQL B
    Scenario 3
    • Instance 1 = CCM primary  
    • Instance 2 = CCM secondary   
    • MySql  in either or both CCM instances
    • Instance 1 = CCM primary  
    • Instance 2 = CCM secondary
    • Instance 3 = PostgreSQL master  
    • Instance 4 = PostgreSQL slave
    MySql in Either or Both CCM Instances
    Scenario 4
    • Instance 1 = CCM  
    • Instance 2 = MySQL  
    • Instance 3 = MySQL
    • Instance 1 = CCM  
    • Instance 2 = PostgreSQL master  
    • Instance 3 = PostgreSQL slave
    Single CCM and Double PostgreSQL
    Scenario 5
    • Instance 1 = CCM primary
    • Instance 2 = CCM secondary
    • Instance 3 = MySQL
    • Instance 1 = CCM primary
    • Instance 2 = CCM secondary
    • Instance 3 = PostgreSQL
    Double CCM and Single PostgreSQL
    Scenario 1: Double CCM and Double PostgreSQL – A
     Scenario 1: Each in Separate Instances

    Use this procedure if upgrading from CloudCenter 4.5.x to 4.6.x.

    Follow this process to upgrade the CCM-DB instances in HA mode.

    1. To upgrade both PostgreSQL database instances, follow this procedure.
      1. Install both database servers on the same cloud or datacenter.
      2. Ensure that the master and slave database servers are located on the same subnet. If the master and slave database servers are on different subnets (or VPCs) ensure that they are able to communicate with each other over private IP.

      3. Launch both instances using the MGMTPOSTGRES_MASTER IP and MGMTPOSTGRES_SLAVE IP.
      4. Run the following commands on each MGMTPOSTGRES_MASTER and MGMTPOSTGRES_SLAVE instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> mgmtpostgres
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7 

        <cloudtype> = amazon, openstack, vmware

      5. Set up SSH connectivity between the master and slave VMs. Exchange the SSH keys between the MGMTPOSTGRES_MASTER IP and MGMTPOSTGRES_SLAVE VMs.

        1. On the MGMTPOSTGRES_MASTER, execute the following to generate a new SSH key. 

          sudo -i
          ssh-keygen -t rsa
          cd ~/.ssh
          cat id_rsa.pub >> authorized_keys
        2. Copy the id_rsa files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from the MGMTPOSTGRES_MASTER to the same location on the MGMTPOSTGRES_SLAVE. On the MGMTPOSTGRES_SLAVE, if the .ssh directory does not exist, create it using the following commands before copying the files.

          sudo -i
          mkdir -p ~/.ssh
          chmod ~/.ssh
        3. On the MGMTPOSTGRES_SLAVE, execute the following to generate a new SSH key.

          sudo -i
          chmod 400 ~/.ssh/id_rsa*
          cat id_rsa.pub >> authorized_keys
        4. Verify mutual SSH access between the MGMTPOSTGRES_MASTER and MGMTPOSTGRES_SLAVE by running the following command on each VM.

          sudo -i
          ssh cliqruser@ <MGMTPOSTGRES_MASTER/MGMTPOSTGRES_SLAVE>
      6.  Start the database wizard and allow the CCM to access the database.

        MGMTPOSTGRES_MASTER – Configure High Availability

        To configure high availability for MGMTPOSTGRES_MASTER, follow this procedure.

          1. SSH into the DB instance as a centos user.
          2. Run the following command:

            sudo -i

          Invoke the wizard.

          CCM Wizard Path
          /usr/local/cliqr/bin/db_config_wizard.sh

           

          Configure Postgres HA to ensure the PostgreSQL database HA and enter the information in each field as follows:

          Write this down for future reference!

          Write down the Field details in a printed version of the Your Notes section for later use.

          Database Properties

          Description

          Configure_Postgres_HAMaster Hostname: The host name for the master database VM
          Master Private IP: The private IP address of the master database VM
          Slave Hostname: The host name for the slave database VM
          Slave Private IP: The private IP address of the slave database VM

           

          VIP or EIP: The VIP/EIP IP for the database

          Use your mouse to select this option.

           AWS Cloud Nuances for EIP
          To setup PostgreSQL as an RDS service in the SA or HA modes, see Configuring HA for PostgreSQL Database on AWS. 

          Once the details are entered, the database server begins replication configuration between the database servers followed by HA configuration and finally presents the following status messages.

          • Configuring database for HA ...

          • Configuring database for replication

        1. Exit the configuration wizard.

        2. Go to the command line for each PostgreSQL server and enter the following command to review the status of the database and the HA connectivity:
          # pcs status

          1. Ensure that the PCSD Status for both database servers are Online.
          2. Ensure that the Daemon Status for corosync, pacemaker and pcsd are active/disabled.

        If your database upgrade was successful, be aware that you can now stop both instances of MySQL as you no longer need these instances.

    2. Upgrade the CCM primary server.

      1. Run the following commands on the CCM instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> ccm_sa
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7, ubuntu1204 (Ubuntu12.04 is not recommended for a new install)

        <cloudtype> = amazon, openstack, vmware

      2. Edit the ccm-response.xml file to include the following values:

        db_host=<postgres_vip or postgre_elastic ip>
        mysql_host=<mysql_host_ip> 
      3. Run the following commands from your download folder.

        java -jar ccm-installer.jar ccm-response.xml
    3. Upgrade the CCM secondary server.

      1. Run the following commands on the CCM instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> ccm_sa
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7, ubuntu1204 (Ubuntu12.04 is not recommended for a new install)

        <cloudtype> = amazon, azure, azurepack,  google, opsource, openstack, softlayer, vmware, vcd

      2. Edit the ccm-response.xml file to include the following values:

        db_host=<postgres_vip or postgre_elastic ip>
        db_enable=false
      3. Run the following commands from your download folder:

        java -jar ccm-installer.jar ccm-response.xml
    Scenario 2: Double CCM and Double PostgreSQL B
     Scenario 2: Each in Separate Instances

    Use this procedure if upgrading from CloudCenter 4.6.0 to 4.6.x (once available).

    Follow this process to upgrade the CCM-DB instances in HA mode.

    1. Follow this procedure to upgrade both PostgreSQL database instances.
      1. Install both database servers on the same cloud or datacenter.
      2. Ensure that the master and slave database servers are located on the same subnet. If the master and slave database servers are on different subnets (or VPCs) ensure that they are able to communicate with each other over private IP.

      3. Launch both instances using the master IP and slave IP.
      4. Set up SSH connectivity between the master and slave VMs. Exchange the SSH keys between the DB master and DB slave servers.

        1. On the DB master, execute the following to generate a new SSH key. 

          sudo -i
          ssh-keygen -t rsa
          cd ~/.ssh
          cat id_rsa.pub >> authorized_keys
        2. Copy the id_rsa files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from DB master to the same location on DB slave. On DB slave, if the .ssh directory does not exist, create it using the following commands before copying the files.

          sudo -i
          mkdir -p ~/.ssh
          chmod ~/.ssh
        3. On the DB slave, execute the following to generate a new SSH key.

          sudo -i
          chmod 400 ~/.ssh/id_rsa*
          cat id_rsa.pub >> authorized_keys
        4. Verify mutual SSH access between the DB master and DB slave by running the following command on each VM.

          sudo -i
          ssh cliqruser@<DB master/DB slave>
      5. Create a security group with the specified ports and launch the instances using this security group.
      6. Run the following commands on each PostgreSQL instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> mgmtpostgres
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7 

        <cloudtype> = amazon, azure, azurepack, google, opsource, openstack, softlayer, vmware, vcd

      7.  Start the database wizard and allow the CCM to access the database.

        MGMTPOSTGRES_MASTER – Configure High Availability

        To configure high availability for MGMTPOSTGRES_MASTER, follow this procedure.

          1. SSH into the DB instance as a centos user.
          2. Run the following command:

            sudo -i

          Invoke the wizard.

          CCM Wizard Path
          /usr/local/cliqr/bin/db_config_wizard.sh

           

          Configure Postgres HA to ensure the PostgreSQL database HA and enter the information in each field as follows:

          Write this down for future reference!

          Write down the Field details in a printed version of the Your Notes section for later use.

          Database Properties

          Description

          Configure_Postgres_HAMaster Hostname: The host name for the master database VM
          Master Private IP: The private IP address of the master database VM
          Slave Hostname: The host name for the slave database VM
          Slave Private IP: The private IP address of the slave database VM

           

          VIP or EIP: The VIP/EIP IP for the database

          Use your mouse to select this option.

           AWS Cloud Nuances for EIP
          To setup PostgreSQL as an RDS service in the SA or HA modes, see Configuring HA for PostgreSQL Database on AWS. 

          Once the details are entered, the database server begins replication configuration between the database servers followed by HA configuration and finally presents the following status messages.

          • Configuring database for HA ...

          • Configuring database for replication

        1. Exit the configuration wizard.

        2. Go to the command line for each PostgreSQL server and enter the following command to review the status of the database and the HA connectivity:
          # pcs status

          1. Ensure that the PCSD Status for both database servers are Online.
          2. Ensure that the Daemon Status for corosync, pacemaker and pcsd are active/disabled.

         

         

    2. Upgrade the CCM primary server.

      1. Run the following commands on the CCM instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> ccm_sa
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7, ubuntu1204 (Ubuntu12.04 is not recommended for a new install)

        <cloudtype> = amazon, azure, azurepack, google, opsource, openstack, softlayer, vmware, vcd

      2. Edit the ccm-response.xml file to include the following values:

        db_host=<postgres_vip or postgres_eip> 
      3. Run the following commands from your download folder.

        java -jar ccm-installer.jar ccm-response.xml
    3. Upgrade the CCM secondary server.

      1. Run the following commands on the CCM instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> ccm_sa
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7, ubuntu1204 (Ubuntu12.04 is not recommended for a new install)

        <cloudtype> = amazon, azure, azurepack,  google, opsource, openstack, softlayer, vmware, vcd

      2. Edit the ccm-response.xml file to include the following values:

        db_host=<postgres_vip or postgres_eip>
        db_enable=false
      3. Run the following commands from your download folder:

        java -jar ccm-installer.jar ccm-response.xml
    Scenario 3: MySql in Either or Both CCM Instances
     Scenario 3: Each in a Separate Instance and MySQL with CCM

    Follow this process to upgrade the CCM-DB instances in HA mode.

    1. Follow this procedure to upgrade both PostgreSQL database instances.
      1. Install both database servers on the same cloud or datacenter.
      2. Ensure that the master and slave database servers are located on the same subnet. If the master and slave database servers are on different subnets (or VPCs) ensure that they are able to communicate with each other over private IP.

      3. Launch both instances using the master IP and slave IP.
      4. Set up SSH connectivity between the master and slave VMs. Exchange the SSH keys between the DB master and DB slave servers.

        1. On the DB master, execute the following to generate a new SSH key. 

          sudo -i
          ssh-keygen -t rsa
          cd ~/.ssh
          cat id_rsa.pub >> authorized_keys
        2. Copy the id_rsa files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from DB master to the same location on DB slave. On DB slave, if the .ssh directory does not exist, create it using the following commands before copying the files.

          sudo -i
          mkdir -p ~/.ssh
          chmod ~/.ssh
        3. On the DB slave, execute the following to generate a new SSH key.

          sudo -i
          chmod 400 ~/.ssh/id_rsa*
          cat id_rsa.pub >> authorized_keys
        4. Verify mutual SSH access between the DB master and DB slave by running the following command on each VM.

          sudo -i
          ssh cliqruser@<DB master/DB slave>
      5. Create a security group with the specified ports and launch the instances using this security group.
      6. Run the following commands on each PostgreSQL instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> mgmtpostgres
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7 

        <cloudtype> = amazon, azure, azurepack, google, opsource, openstack, softlayer, vmware, vcd

      7.  Start the database wizard and allow the CCM to access the database.

        MGMTPOSTGRES_MASTER – Configure High Availability

        To configure high availability for MGMTPOSTGRES_MASTER, follow this procedure.

          1. SSH into the DB instance as a centos user.
          2. Run the following command:

            sudo -i

          Invoke the wizard.

          CCM Wizard Path
          /usr/local/cliqr/bin/db_config_wizard.sh

           

          Configure Postgres HA to ensure the PostgreSQL database HA and enter the information in each field as follows:

          Write this down for future reference!

          Write down the Field details in a printed version of the Your Notes section for later use.

          Database Properties

          Description

          Configure_Postgres_HAMaster Hostname: The host name for the master database VM
          Master Private IP: The private IP address of the master database VM
          Slave Hostname: The host name for the slave database VM
          Slave Private IP: The private IP address of the slave database VM

           

          VIP or EIP: The VIP/EIP IP for the database

          Use your mouse to select this option.

           AWS Cloud Nuances for EIP
          To setup PostgreSQL as an RDS service in the SA or HA modes, see Configuring HA for PostgreSQL Database on AWS. 

          Once the details are entered, the database server begins replication configuration between the database servers followed by HA configuration and finally presents the following status messages.

          • Configuring database for HA ...

          • Configuring database for replication

        1. Exit the configuration wizard.

        2. Go to the command line for each PostgreSQL server and enter the following command to review the status of the database and the HA connectivity:
          # pcs status

          1. Ensure that the PCSD Status for both database servers are Online.
          2. Ensure that the Daemon Status for corosync, pacemaker and pcsd are active/disabled.

    2. Upgrade the CCM primary server.

      1. Run the following commands on the CCM instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> ccm_sa
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7, ubuntu1204 (Ubuntu12.04 is not recommended for a new install)

        <cloudtype> = amazon, azure, azurepack, google, opsource, openstack, softlayer, vmware, vcd

      2. Edit the ccm-response.xml file to include the following values:

        db_host=<postgres_vip or postgre_elastic ip>
        
      3. Run the following commands from your download folder.

        java -jar ccm-installer.jar ccm-response.xml
    3. Upgrade the CCM secondary server.

      1. Run the following commands on the CCM instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> ccm_sa
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7, ubuntu1204 (Ubuntu12.04 is not recommended for a new install)

        <cloudtype> = amazon, azure, azurepack, google, opsource, openstack, softlayer, vmware, vcd

      2. Edit the ccm-response.xml file to include the following values:

        db_enable=false
        mysql_host=<mysql_ip>
        db_host=<postgres_vip or postgre_elastic ip>

        if you have two MySQL servers (in cased of high availability setup), the mysql_host=<mysql_ip> value displays as mysql_host=<default>. In this case, you do not need to change the default value to mysql_ip.

      3. Run the following commands from your download folder:

        java -jar ccm-installer.jar ccm-response.xml
    Scenario 4: Single CCM and Double PostgreSQL
     Scenario 4: DBs in Separate Instances

    Follow this process to upgrade the CCM-DB instances in HA mode.

    1. Follow this procedure to upgrade both PostgreSQL database instances.
      1. Install both database servers on the same cloud or datacenter.
      2. Ensure that the master and slave database servers are located on the same subnet. If the master and slave database servers are on different subnets (or VPCs) ensure that they are able to communicate with each other over private IP.

      3. Launch both instances using the master IP and slave IP.
      4. Set up SSH connectivity between the master and slave VMs. Exchange the SSH keys between the DB master and DB slave servers.

        1. On the DB master, execute the following to generate a new SSH key. 

          sudo -i
          ssh-keygen -t rsa
          cd ~/.ssh
          cat id_rsa.pub >> authorized_keys
        2. Copy the id_rsa files (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub) from DB master to the same location on DB slave. On DB slave, if the .ssh directory does not exist, create it using the following commands before copying the files.

          sudo -i
          mkdir -p ~/.ssh
          chmod ~/.ssh
        3. On the DB slave, execute the following to generate a new SSH key.

          sudo -i
          chmod 400 ~/.ssh/id_rsa*
          cat id_rsa.pub >> authorized_keys
        4. Verify mutual SSH access between the DB master and DB slave by running the following command on each VM.

          sudo -i
          ssh cliqruser@<DB master/DB slave>
      5. Create a security group with the specified ports and launch the instances using this security group.
      6. Run the following commands on each PostgreSQL instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> mgmtpostgres
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7 

        <cloudtype> = amazon, azure, azurepack, google, opsource, openstack, softlayer, vmware, vcd

      7.  Start the database wizard and allow the CCM to access the database.

        MGMTPOSTGRES_MASTER – Configure High Availability

        To configure high availability for MGMTPOSTGRES_MASTER, follow this procedure.

          1. SSH into the DB instance as a centos user.
          2. Run the following command:

            sudo -i

          Invoke the wizard.

          CCM Wizard Path
          /usr/local/cliqr/bin/db_config_wizard.sh

           

          Configure Postgres HA to ensure the PostgreSQL database HA and enter the information in each field as follows:

          Write this down for future reference!

          Write down the Field details in a printed version of the Your Notes section for later use.

          Database Properties

          Description

          Configure_Postgres_HAMaster Hostname: The host name for the master database VM
          Master Private IP: The private IP address of the master database VM
          Slave Hostname: The host name for the slave database VM
          Slave Private IP: The private IP address of the slave database VM

           

          VIP or EIP: The VIP/EIP IP for the database

          Use your mouse to select this option.

           AWS Cloud Nuances for EIP
          To setup PostgreSQL as an RDS service in the SA or HA modes, see Configuring HA for PostgreSQL Database on AWS. 

          Once the details are entered, the database server begins replication configuration between the database servers followed by HA configuration and finally presents the following status messages.

          • Configuring database for HA ...

          • Configuring database for replication

        1. Exit the configuration wizard.

        2. Go to the command line for each PostgreSQL server and enter the following command to review the status of the database and the HA connectivity:
          # pcs status

          1. Ensure that the PCSD Status for both database servers are Online.
          2. Ensure that the Daemon Status for corosync, pacemaker and pcsd are active/disabled.

    2. Upgrade the CCM server.

      1. Run the following commands on the CCM instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> ccm_sa
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7, ubuntu1204 (Ubuntu12.04 is not recommended for a new install)

        <cloudtype> = amazon, azure, azurepack, google, opsource, openstack, softlayer, vmware, vcd

      2. Edit the ccm-response.xml file to include the following values:

        db_host=<postgres_vip or postgre_elastic ip>
        mysql_host=<mysql_ip> 
      3. Run the following commands from your download folder.

        java -jar ccm-installer.jar ccm-response.xml
      4. If your database upgrade was successful, be aware that you can now stop both instances of MySQL as you no longer need these instances.
    Scenario 5: Double CCM and Single PostgreSQL
     Scenario 5: CCMs in Separate Instances

    Follow this process to upgrade the CCM-DB instances in HA mode.

    1. Upgrade the PostgreSQL instance.
      1. Create a security group with the specified ports and launch the instance using this security group.
      2. Run the following commands on the PostgreSQL instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> mgmtpostgres
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7 

        <cloudtype> = amazon, azure, azurepack, google, opsource, openstack, softlayer, vmware, vcd

      3.  Start the database wizard and allow the CCM to access the database.

        MGMTPOSTGRES_MASTER – Configure High Availability

        To configure high availability for MGMTPOSTGRES_MASTER, follow this procedure.

          1. SSH into the DB instance as a centos user.
          2. Run the following command:

            sudo -i

          Invoke the wizard.

          CCM Wizard Path
          /usr/local/cliqr/bin/db_config_wizard.sh

           

          Configure Postgres HA to ensure the PostgreSQL database HA and enter the information in each field as follows:

          Write this down for future reference!

          Write down the Field details in a printed version of the Your Notes section for later use.

          Database Properties

          Description

          Configure_Postgres_HAMaster Hostname: The host name for the master database VM
          Master Private IP: The private IP address of the master database VM
          Slave Hostname: The host name for the slave database VM
          Slave Private IP: The private IP address of the slave database VM

           

          VIP or EIP: The VIP/EIP IP for the database

          Use your mouse to select this option.

           AWS Cloud Nuances for EIP
          To setup PostgreSQL as an RDS service in the SA or HA modes, see Configuring HA for PostgreSQL Database on AWS. 

          Once the details are entered, the database server begins replication configuration between the database servers followed by HA configuration and finally presents the following status messages.

          • Configuring database for HA ...

          • Configuring database for replication

        1. Exit the configuration wizard.

        2. Go to the command line for each PostgreSQL server and enter the following command to review the status of the database and the HA connectivity:
          # pcs status

          1. Ensure that the PCSD Status for both database servers are Online.
          2. Ensure that the Daemon Status for corosync, pacemaker and pcsd are active/disabled.

      4. If your database upgrade was successful, be aware that you can now stop the MySQL instance as you no longer need it.
    2. Upgrade the CCM primary server.

      1. Run the following commands on the CCM instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> ccm_sa
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7, ubuntu1204 (Ubuntu12.04 is not recommended for a new install)

        <cloudtype> = amazon, azure, azurepack, google, opsource, openstack, softlayer, vmware, vcd

      2. Edit the ccm-response.xml file to include the following values:

        db_host=<postgres_vip or postgre_elastic ip>
        mysql_host=<mysql_ip> 
      3. Run the following commands from your download folder.

        java -jar ccm-installer.jar ccm-response.xml
    3. Upgrade the CCM secondary server.

      1. Run the following commands on the CCM instance.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> ccm_sa
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax

        <ostype> = centos6, centos7, rhel6, rhel7, ubuntu1204 (Ubuntu12.04 is not recommended for a new install)

        <cloudtype> = amazon, azure, azurepack, google, opsource, openstack, softlayer, vmware, vcd

      2. Edit the ccm-response.xml file to include the following values:

        db_enable=false
        mysql_host=<mysql_ip>
        db_host=<postgres_vip or postgre_elastic ip>
      3. Run the following commands from your download folder:

        java -jar ccm-installer.jar ccm-response.xml

    You have upgraded the CCM and PostgreSQL servers in HA mode.

    Post Upgrade Tasks

    1. Verify Your Upgrade – Ensure that the version file (/usr/local/osmosix/etc/version) reflects the new release in each VM.

      cat /usr/local/osmosix/etc/version
    2. Reboot the CCM Servers.

      reboot

     

  3.  CCO Upgrade in HA Mode

     

    CCO Upgrade in HA Mode

    Overview

    This section provides details on upgrading your CCO in HA mode.

    Prerequisites

    Be aware that the CCO servers will be offline during the upgrade process. Schedule some down time for your enterprise before starting this process.

    Verify these requirements before you begin the upgrade process:

    • Review the information provided in the Upgrade Overview section and validate the following requirements for the release to which you are upgrading:

      • Is an upgrade path available?
      • Is the core_upgrade.bin file required?
    • See HA Best Practices for HA considerations.
    • Backup your database and applications before you begin this process. See Backup and Recovery in HA Mode for additional context.
    • For each CCO instance that must be upgraded, verify the following prerequisites:

      • Ensure that a version file (/usr/local/osmosix/etc/version) exists in both CCOs to be upgraded.

      • Verify that the version file contains the correct version number (for example, if your current CloudCenter release version is 4.7.2, ensure that the corresponding version value is 4.7.2).

      • See the corresponding release notes for release-specific information on the CloudCenter version to which you are upgrading. For example, the CloudCenter 4.6.0 Release Notes.

    • The upgrade procedure in this section assumes the following setup:
      • The MongoDB data is retained on the CCO_PRIMARY server – this is the initiating server.
      • The MongoDB data is deleted on the CCO_SECONDARY and CCO_TERTIARY – be sure to backup and delete the CloudCenter database (called cliqr) on these two servers.
        • The assumed path for this upgrade procedure is /var/lib/mongo
        • The mongodump directory is created as a dump sub-directory in  the specified directory: /var/lib/mongo/dump

        • To locate the path for your setup, see your /etc/mongod.conf file
      • The configuration files on the CCO_TERTIARY server must reflect the corresponding values for your deployment.

    Download package files:

    See Installation Overview to understand the required components and the installation options.

    See Installer Overview to understand the types of files.

    1. SSH into the VM instance designated for this component by using the key pair that you used to launch the VM.
    2. Download the following required files for this component from software.cisco.com to the /tmp folder on that VM:

      • cco-installer.jar 
      • cco-response.xml 
      • core_upgrade.bin

        Ensure (by reviewing Upgrade Overview) that this file is required for your release upgrade path.


    Upgrading CCO from HA to HA Mode

    The CCO upgrade procedure is the same for both CloudCenter 4.6 and 4.7 – except for the last step.

    To upgrade the Primary, Secondary, and Tertiary CCO instances, follow this process on each server.

    1. Select and execute the required upgrade scenario as you would for a standalone CCO upgrade – see 2. CCO Upgrade for additional details.
      1. Run the following commands on each CCO instance.

        Ensure (by reviewing Upgrade Overview) that this step is required for your release upgrade path.

        sudo –i
        cd /tmp
        chmod 755 core_upgrade.bin
        ./core_upgrade.bin <ostype> <cloudtype> cco
        
        
        #After the above process completes, remove the core_upgrade.bin file
        rm core_upgrade.bin
        cd /tmp
         Syntax
        <ostype> = centos6, centos7, rhel6, rhel7
        <cloudtype
        > = amazon, azure, azurepack, google, opsource, openstack, softlayer, vmware, vcd
      2. Run the following commands from your download folder.

        java -jar cco-installer.jar cco-response.xml
    2. In some cases, your deployment settings may need to be updated in the CCO server. Reassign the IP address information by running the wizard for both the CCO server.

    3. Verify the following configuration files to ensure that they reflect the right values for your deployment on the CCO_TERTIARY server:

      • /usr/local/tomcat/webapps/ROOT/WEB-INF/rabbit-gateway.properties on the CCO server (verify the gatewayHost value in particular).

      • /usr/local/tomcat/webapps/ROOT/WEB-INF/gateway.properties on the CCO server (verify the rabbit.gateway.brokerHost and rabbit.gateway.cluster.addresses values in particular).

       
    4. Backup the MongoDB data on the CCO_SECONDARY and the CCO_TERTIARY servers.
      1. Stop the MongoDB instance.

        sudo -i
        service mongod stop
      2. Create a directory named backup (to store the MongoDB data).

        cd /tmp
        mkdir backup
      3. Change to the backup directory and provide the location to backup the data files. 

        cd /tmp/backup
        mongodump -dbpath /var/lib/mongo
      4. Start the MongoDB instance.

        service mongod start
    5. Connect to the MongoDB instance as root and delete the CloudCenter database (called cliqr) on the CCO_SECONDARY and the CCO_TERTIARY instances – use one of the following options to delete the database.
      • Option 1

        sudo -i
        mongo --host <CCO_SECONDARY or CCO_TERTIARY> --port27017
        use cliqr
        db dropDatabase()
      • Option 2

        sudo -i
        cd /var/lib/mongo
        rm -rf cliqr*
    6. Set up SSH communication between the CCO_PRIMARY, CCO_SECONDARY, and CCO_TERTIARY instances
      1. Run the following commands on the CCO_PRIMARY to generate a new SSH key.

        sudo -i
        ssh-keygen -t rsa
        cd ~/.ssh
        cat id_rsa.pub>>authorized_keys
      2. Create the .ssh directory (if it does not exist) on the CCO_SECONDARY and CCO_TERTIARY instances.

        sudo -i
        mkdir -p ~/.ssh
        chmod 700 ~/.ssh
      3. Copy the id_rsa files (~/ssh/id_rsa and ~/ssh/id_rsa.pub) from the CCO_PRIMARY to the same location on the CCO_SECONDARY and CCO_TERTIARY instances.

        sudo -i
        chmod 700 ~/.ssh/id_rsa*
        cat id_rsa.pub>>authorized_keys
      4. Verify mutual SSH access between all three CCO instances by running the following commands on each CCO VM.

        sudo -i
        ssh root@<CCO_PRIMARY or CCO_SECONDARY or CCO_TERTIARY>

        You have now set up SSH on all three CCO instances.

       
    7. This step differs for CloudCenter 4.6 and 4.7:
      •  CloudCenter 4.6

        Start the CCO wizard, be sure to configure the Hazelcast IPList by providing a comma separated list of the CCO IP addresses in the primary and secondary CCO server.

        •  CCO Primary - Configure Wizard Properties

           

          CCO_PRIMARY – Configure CCO Properties

          You can configure the information for all three CCO servers when you invoke the CCO_PRIMARY wizard.

            1. SSH into the CCO instance as a centos user.
            2. Run the following command:

              sudo -i
          1. Invoke the CCO wizard.

            CCO Wizard Path
            /usr/local/cliqr/bin/cco_config_wizard.sh
          2. Configure the server properties.

            Write this down for future reference!

            Write down the Field details in a printed version of the Your Notes section for later use.
            GroupNotes

            AgentBundle

            Use the defaults.

            • If you are using the custom bundle, replace cdn.cliqr.com with the custom bundle store IP or DNS
            • If you are using the package store, replace repo.cliqrtech.com with the custom package store IP or DNS

            AMQP_Server

            • AMQP Server IP: AMQP_IP or AMQP_LB_IP
            • AMQP Port: 5671
            NetworkHostname: Configure the Network details for your CCO environment. This is an optional step to configure the Private IP of the VM. You can generally configure this information if the VM does not have preset IP or hostname or if you need to override an existing IP or Hostname.

            Guacamole

             

            • Connection Broker Host: IP address of the both Primary and Secondary AMQP VMs separated by commas
            • Connection Broker Port1: 7788
            • Connection Broker Port2: 7789

            Docker

            • Docker Registry URL: Set only if custom docker registry is used
            • Docker CACert URL: Set only if docker registry uses SSL with custom CA Certificates

            Configure_HA (CloudCenter 4.6.x)

             

            This configuration is only applicable for CloudCenter 4.6.x


            Hazelcast IP List: Comma separated list of CCO_PRIMARY_IP and CCO_SECONDARY_IP

            Configure_HA (CloudCenter 4.7.x)

            CCO HA Info: Specify the following details in the primary CCO server.

            This configuration is only applicable for CloudCenter 4.7.x

            • Primary Node IP: Enter the IP address of the Primary CCO instance.
            • Secondary Node IP: Enter the IP address of the Secondary CCO instance.
            • Tertiary Node IP: Enter the IP address of the Tertiary CCO instance.

            ELK_Info

            (Effective CloudCenter 4.7.0)

            • ELK Host: Specify the IP address for the ELK/Monitor host.
            • Elasticsearch Port: The Elasticearch Port displays 8881 by default.
            • Logstash Port: The Logstash Port displays 4560 by default.
            • Host Identifier: The Host Identifier is a Unique ID for the server – be sure to prefix the unique identifier with CCO_ for example, CCO_Openstack_regionOne or CCO_Amazon_east.
            • Host Identifier List: The Host Identifier List field only applies to environments using the HA mode – provide a list of comma separated unique host Identifiers for all ELK/Monitor hosts in a HA setup = for example, CCO1,CCO2,myCCO.

              In an environment operating in HA mode, if you have three CCO instances with unique IDs configured as CCO_1,CCO_2,CCO_3 in their respective server.properties file, then this property should state CCO_1,CCO_2,CCO_3 in each CCO instances. Each CCO must be aware of the unique ID of the other CCO(s) when in HA mode.

          3. Verify your changes and Exit the CCO configuration wizard.

          You have successfully configured the CCO! You can now proceed to the next step.

           

          Back to Phase 4: Install Components


        •  CCO Secondary - Configure Wizard Properties

          CCO_SECONDARY – Configure CCO Properties

            1. SSH into the CCO instance as a centos user.
            2. Run the following command:

              sudo -i
          1. Invoke the CCO wizard.

            CCO Wizard Path
            /usr/local/cliqr/bin/cco_config_wizard.sh
          2. Configure the server properties.

            Write this down for future reference!

            Write down the Field details in a printed version of the Your Notes section for later use.
            GroupNotes

            AgentBundle

            Use the defaults.

            • If you are using the custom bundle, replace cdn.cliqr.com with the custom bundle store IP or DNS
            • If you are using the package store, replace repo.cliqrtech.com with the custom package store IP or DNS

            AMQP_Server

            • AMQP Server IP: AMQP_IP or AMQP_LB_IP
            • AMQP Port: 5671
            NetworkHostname: Configure the Network details for your CCO environment. This is an optional step to configure the Private IP of the VM. You can generally configure this information if the VM does not have preset IP or hostname or if you need to override an existing IP or Hostname.

            Guacamole

             

            • Connection Broker Host: AMQP_IP or AMQP_LB_IP
            • Connection Broker Port1: 7788
            • Connection Broker Port2: 7789

            Docker

            • Docker Registry URL: Set only if custom docker registry is used
            • Docker CACert URL: Set only if docker registry uses SSL with custom CA Certificates

            Configure_HA

             

            Hazelcast IP List:Comma separated list of CCO_PRIMARY_IP and CCO_SECONDARY_IP
          3. Verify your changes and Exit the CCO configuration wizard.

          You have successfully configured the CCO! You can now proceed to the next step.

      •  CloudCenter 4.7

        Start the wizard in the primary server and configure the HA properties for all three CCO servers.

        • Required for all deployments

           CCO PRIMARY – Configure HA Properties

          CCO_PRIMARY – Configure CCO Properties

          You can configure the information for all three CCO servers when you invoke the CCO_PRIMARY wizard.

            1. SSH into the CCO instance as a centos user.
            2. Run the following command:

              sudo -i
          1. Invoke the CCO wizard.

            CCO Wizard Path
            /usr/local/cliqr/bin/cco_config_wizard.sh
          2. Configure the server properties.

            Write this down for future reference!

            Write down the Field details in a printed version of the Your Notes section for later use.
            GroupNotes

            AgentBundle

            Use the defaults.

            • If you are using the custom bundle, replace cdn.cliqr.com with the custom bundle store IP or DNS
            • If you are using the package store, replace repo.cliqrtech.com with the custom package store IP or DNS

            AMQP_Server

            • AMQP Server IP: AMQP_IP or AMQP_LB_IP
            • AMQP Port: 5671
            NetworkHostname: Configure the Network details for your CCO environment. This is an optional step to configure the Private IP of the VM. You can generally configure this information if the VM does not have preset IP or hostname or if you need to override an existing IP or Hostname.

            Guacamole

             

            • Connection Broker Host: IP address of the both Primary and Secondary AMQP VMs separated by commas
            • Connection Broker Port1: 7788
            • Connection Broker Port2: 7789

            Docker

            • Docker Registry URL: Set only if custom docker registry is used
            • Docker CACert URL: Set only if docker registry uses SSL with custom CA Certificates

            Configure_HA (CloudCenter 4.6.x)

             

            This configuration is only applicable for CloudCenter 4.6.x


            Hazelcast IP List: Comma separated list of CCO_PRIMARY_IP and CCO_SECONDARY_IP

            Configure_HA (CloudCenter 4.7.x)

            CCO HA Info: Specify the following details in the primary CCO server.

            This configuration is only applicable for CloudCenter 4.7.x

            • Primary Node IP: Enter the IP address of the Primary CCO instance.
            • Secondary Node IP: Enter the IP address of the Secondary CCO instance.
            • Tertiary Node IP: Enter the IP address of the Tertiary CCO instance.

            ELK_Info

            (Effective CloudCenter 4.7.0)

            • ELK Host: Specify the IP address for the ELK/Monitor host.
            • Elasticsearch Port: The Elasticearch Port displays 8881 by default.
            • Logstash Port: The Logstash Port displays 4560 by default.
            • Host Identifier: The Host Identifier is a Unique ID for the server – be sure to prefix the unique identifier with CCO_ for example, CCO_Openstack_regionOne or CCO_Amazon_east.
            • Host Identifier List: The Host Identifier List field only applies to environments using the HA mode – provide a list of comma separated unique host Identifiers for all ELK/Monitor hosts in a HA setup = for example, CCO1,CCO2,myCCO.

              In an environment operating in HA mode, if you have three CCO instances with unique IDs configured as CCO_1,CCO_2,CCO_3 in their respective server.properties file, then this property should state CCO_1,CCO_2,CCO_3 in each CCO instances. Each CCO must be aware of the unique ID of the other CCO(s) when in HA mode.

          3. Verify your changes and Exit the CCO configuration wizard.

          You have successfully configured the CCO! You can now proceed to the next step.

           

          Back to Phase 4: Install Components

        • Required if your deployment uses the Monitor component.

           CCO SECONDARY and TERTIARY – Configure ELK Properties

          CCO_SECONDARY and CCO_TERTIARY – Configure ELK Properties

            1. SSH into the CCO instance as a centos user.
            2. Run the following command:

              sudo -i
          1. Invoke the CCO wizard.

            CCO Wizard Path
            /usr/local/cliqr/bin/cco_config_wizard.sh
          2. Configure the properties for the ELK Information:

            GroupConfigurable Properties

            ELK_Info

            (Effective CloudCenter 4.7.0)

            • ELK Host: The IP address for the ELK/Monitor host.
            • Elasticsearch Port: Displays 8881 by default.
            • Logstash Port: Displays 4560 by default.
            • Host Identifier: A Unique ID for the server – be sure to prefix the unique identifier with CCO_ for example, CCO_Openstack_regionOne or CCO_Amazon_east.
            • Host Identifier List: Only applies to environments using the HA mode – provide a list of comma separated unique host Identifiers for all ELK/Monitor hosts in a HA setup = for example, CCO1,CCO2,myCCO.

              In an environment operating in HA mode, if you have three CCO instances with unique IDs configured as CCO_1,CCO_2,CCO_3 in their respective server.properties file, then this property should state CCO_1,CCO_2,CCO_3 in each CCO instances. Each CCO must be aware of the unique ID of the other CCO(s) when in HA mode.

          3. Verify your changes and Exit the CCO configuration wizard.

          You have successfully configured the CCO! You can now proceed to the next step.

    Upgrading from Non-HA to CloudCenter 4.7 HA Mode

    See Migrate CCO from Non-HA to HA mode.

     

     

     

     

  4.  AMQP Upgrade in HA Mode

     

    AMQP Upgrade in HA Mode

    Overview

    This section provides details on upgrading your AMQP in HA mode.

    Currently, the CloudCenter platform does not support HA for the Guacamole component.

    Prerequisites

    Verify these requirements before you begin the upgrade process:

    • Review the information provided in the Upgrade Overview section and validate the following requirements for the release to which you are upgrading:

      • Is an upgrade path available?
      • Is the core_upgrade.bin file required?
    • See HA Best Practices for HA considerations.
    • For each AMQP instance that must be upgraded, verify the following prerequisites:

      • Ensure that a version file (/usr/local/osmosix/etc/version) exists in both AMQPs to be upgraded.

      • Verify that the version file contains the correct version number (for example, if your current CloudCenter release version is 4.7.2, ensure that the corresponding version value is 4.7.2).

      • See the corresponding release notes for release-specific information on the CloudCenter version to which you are upgrading. For example, the CloudCenter 4.7.2 Release Notes.

    Download Packages

    Download package files:

    See Installation Overview to understand the required components and the installation options.

    See Installer Overview to understand the types of files.

    1. SSH into the VM instance designated for this component by using the key pair that you used to launch the VM.
    2. Download the following required files for this component from software.cisco.com to the /tmp folder on that VM:

      • cco-installer.jar
      • conn_broker-response.xml
      • core_upgrade.bin

        Ensure (by reviewing Upgrade Overview) that this file is required for your release upgrade path.

    Upgrading AMQP Instances

    To upgrade the Primary and Secondary AMQP instances, follow this process on each server.

    1. Login to the AMQP server and back up the data.

      cd $bakdir
      cp -r /usr/local/tomcatgua/webapps/access .
      cp -r /usr/local/tomcatgua/webapps/cliqr-connection-broker .
    2. Run the following commands on the AMQP server.

      Ensure (by reviewing Upgrade Overview) that this step is required for your release upgrade path.

      sudo –i
      cd /tmp
      chmod 755 core_upgrade.bin
      
      #Set the following only if a local package store is setup:
      export CUSTOM_REPO=<http://local_package_store ip>
      
      ./core_upgrade.bin <ostype> <cloudtype> rabbit
      
      #After the above process completes, remove the core_upgrade.bin file
      rm core_upgrade.bin 
      cd /tmp
       Syntax

      <ostype>= centos6, centos7, rhel6, rhel7

      <cloudtype>= amazon, azure, azurerm, azurepack, google, opsource, openstack, softlayer, vmware, and vcd

    3. Run the following commands from your download folder.

      java -jar cco-installer.jar conn_broker-response.xml

    Post Upgrade Tasks

    1. Verify Your Upgrade – Ensure that the version file (/usr/local/osmosix/etc/version) reflects the new release.

      cat /usr/local/osmosix/etc/version
    2. Reboot the AMQP servers – Be aware of the following consequences if/when you reboot the AMQP server.

      Reboot AMQP VM

      If you change the AMQP server's host name, the local AMQP database is renamed and you must reboot the AMQP VM.

      • To reboot the AMQP VM, run the following commands as root:

        rm /usr/local/osmosix/etc/.RABBITINSTALLED
        /usr/local/osmosix/bin/rabbit_config.sh
        reboot
      • If you reboot the VM, be aware of the following details:
        • You may end up with a new host name and database name after the reboot.

        • Some clouds set the host name automatically for each new instance or reboot – RabbitMQ uses a preset host name to set the database name.

        • If a database user exists and a login is not associated, this user may not be able to log into the AMQP server.

          • Ensure that the required users (cliqr and cliqr_worker) are setup in your database. If you have additional users in your database, they will also be displayed when you run the rabbitmqctl command.

            rabbitmqctl list_users
            Listing users ...
            cliqr [administrator]
            cliqr_worker []
          • If you do not see these users in your database, run the following commands as root (to recreate the users in the AMQP configuration):

            rm /usr/local/osmosix/etc/.RABBITINSTALLED
            bash /usr/local/osmosix/bin/rabbit_config.sh

    3.  Configure the Properties in Primary AMQP

      AMQP_PRIMARY – Configure CCM/CCO Properties for Guacamole Server

      Dedicated GUAC Setup?

      This GUA config wizard step is not required if you have set up a dedicated Guacamole server.

        1. SSH into the GUA instance as a centos user.
        2. Run the following command:

          sudo -i
      1. Invoke the GUA wizard.

        GUA Wizard Path
        /usr/local/cliqr/bin/gua_config_wizard.sh
      2. Configure the CCO and CCM properties.

        Write this down for future reference!

        Write down the Field details in a printed version of the  Your Notes section for later use.
      3. Configure the properties for the CCM and CCO VMs:

        GroupHostPossible IP Addresses

        CCM_Info

        CCM Host

        CCM Host: (Required)

        CCM_IP or  CCM_SA_IP or CCM_LB_IP

        CCO_InfoCCO HostCCO Host: (Required)
        CCO_IP or  CCO_LB_IP
      4. Verify your changes and Exit the GUA configuration wizard.

      5. Reboot the AMQP_PRIMARY VM.

       

    4.  Configure the Properties in Secondary AMQP

      AMQP_SECONDARY – Configure CCM/CCO Properties for Guacamole Server

      Dedicated GUAC Setup?

      This GUA config wizard step is not required if you have set up a dedicated Guacamole server.

        1. SSH into the GUA instance as a centos user.
        2. Run the following command:

          sudo -i
      1. Invoke the GUA wizard.

        GUA Wizard Path
        /usr/local/cliqr/bin/gua_config_wizard.sh
      2. Configure the CCO and CCM properties.

        Write this down for future reference!

        Write down the Field details in a printed version of the Installation Approach > Your Notes section for later use.
      3. Configure the properties for the CCM and CCO VMs:

        GroupFieldPossible IP Addresses

        CCM_Info

        CCM Host

        CCM Host: (Required)

        CCM_IP or  CCM_SA_IP or CCM_LB_IP

        CCO_InfoCCO HostCCO Host: (Required)
        CCO_IP or CCO_LB_IP
      4. Verify your changes and Exit the GUA configuration wizard.

      5. Reboot the AMQP_SECONDARY VM.

      You have successfully configured the AMQP server! You can now proceed to the next step.

    5.  Update the AMQP_LB Configuration

      AMQP_LB

      The AMQP load balancing can be done through HAProxy, NGiNX, Apache2, or a cloud that is natively available to services, like AWS Elastic Load Balancer (ELB). To configure the load balancer service and ensure AMQP load balancing, be sure to listen on port 5671 and balance the request at 443 on both the AMQP_PRIMARY and AMQP_SECONDARY servers.

      See AMQP_LB Ports for the complete list of ports that need to be open for your deployment.

      The following load balancing configuration was performed on CentOS7.x VM with HAProxy for the AMQP VM.

      1. SSH into the VM instance using the key pair that you used to launch the VM.
      2. Install HAProxy as the root user.

        yum –y install haproxy
        
      3. Modify HAProxy config file as below

        vi /etc/haproxy/haproxy.cfg
         
        #configuration to listen on 5671 and loadbalance
         
        frontend amqps-in
            mode tcp
            log global
            bind *:5671
            default_backend amqps
        backend amqps
            mode tcp
            balance roundrobin
            option ssl-hello-chk
            server amqp1 <AMQP_PRIMARY>:5671 check
            server amqp2 <AMQP_SECONDARY>:5671 check
        #configuration to listen on 443 and loadbalance
        frontend gua-in
            mode tcp
            log global
            bind *:443
            default_backend guas
        backend guas
           mode tcp
           balance source
           cookie SVR insert preserve nocache
           option ssl-hello-chk
           server amqp1 <AMQP_PRIMARY>:443 check
           server amqp2 <AMQP_SECONDARY>:443 check
        
      4. To bind to 5671 port you must disable SELinux – run the following command to disable SELinux.

        setenforce 0
        sed -i 's/=enforcing/=permissive/g' /etc/selinux/config*
        #This command ensures that SELINUX is disabled permanently and the changes are retained even in case of reboot 
      5. Start the HAProxy service and check the status, it should be active

         

        systemctl start haproxy
        systemctl status haproxy 
        

    6. Upgrade the Monitor instances – See Monitor Upgrade in HA Mode  for additional context.

     

  5.  Monitor Upgrade in HA Mode

     

    Health Monitor Upgrade in HA Mode

    Prerequisites

    For each CloudCenter deployment that needs to be upgraded, verify the following prerequisites:

    • Backup your database and applications before you begin this process.

    • You should have already performed the CCM-Database Upgrade procedure.

    • You should have already performed the CCO Upgrade procedure.

    • Review the information provided in the Upgrade Overview section and validate the following requirements for the release to which you are upgrading:

      • Is an upgrade path available?
      • Is the core_upgrade.bin file required?
    • Ensure that a version file (/usr/local/osmosix/etc/version) exists in both Monitors to be upgraded.

    • Verify that the version file contains the correct version number (for example, if your current CloudCenter release version is 4.6.0, ensure that the corresponding version value is 4.6.0).

    • See the corresponding release notes for release-specific information on the CloudCenter version to which you are upgrading. For example, the CloudCenter 4.6.0 Release Notes.

    Backup Database

    Backup the exploded war files to a backup folder (the following example uses /mnt, you can change this directory as applicable)

    NOW=$(date +"%Y%m%d")
    bakdir="/mnt/bak/$NOW"
    mkdir -p $bakdir
    cd $bakdir
    
    cp -r /usr/local/tomcat/webapps/* . 

    Download Packages

    Download package files:

    See Installation Overview to understand the required components and the installation options.

    See Installer Overview to understand the types of files.

    1. SSH into the VM instance designated for this component by using the key pair that you used to launch the VM.
    2. Download the following required files for this component from software.cisco.com to the /tmp folder on that VM:

      • monitor-installer.jar
      • monitor-response.xml
      • core_upgrade.bin

        Ensure (by reviewing Upgrade Overview) that this file is required for your release upgrade path.

    Upgrading Monitor Instances

    To upgrade the Primary and Secondary Monitor instances, follow this process on each server.

    1. Run the following commands:

      Ensure (by reviewing Upgrade Overview) that this step is required for your release upgrade path.

      sudo –i
      cd /tmp
      chmod 755 core_upgrade.bin
      ./core_upgrade.bin <ostype> <cloudtype> monitor 
      
      #After the above process completes, remove the core_upgrade.bin file
      rm core_upgrade.bin

      For example: ./core_upgrade.bin centos7 amazon monitor

      • <ostype> = centos6, centos7, rhel6, rhel7

      • <cloudtype> = amazon, azure, azurepack, azurerm, google, opsource, openstack, softlayer, vmware, vcd

    2. Run the following command from your download folder.

      java -jar monitor-installer.jar monitor-response.xml

    Post Upgrade Tasks

    1. Verify Your Upgrade – Ensure that the version file (/usr/local/osmosix/etc/version) reflects the new release.

      cat /usr/local/osmosix/etc/version
    2. Reboot the Monitor server.

      reboot
    3.  Configure the Properties in the Monitor Wizard.

      Monitor – Configure Monitor Properties

        1. SSH into the MONITOR instance as a centos user.
        2. Run the following command:

          sudo -i
      1. Invoke the wizard.

        Monitor Wizard Path
        /usr/local/cliqr/bin/monitor_config_wizard.sh
      2. Configure the properties for the Monitor instance.

        Write this down for future reference!

         Write down the Field details in a printed version of the Your Notes section for later use.

        GroupNotes
        CCM_Info
        • Monitor ID – A unique (alphanumeric) identifier used for the health check instance.
        • CCM Hostname/URL (Required)
          • CCM_IP or 
          • CCM_SA_IP or
          • CCM_LB_IP
        • Monitor User – The User ID configured on the CCM server to enable health check for cloud  regions.
          • To perform a health check on all activated cloud regions, set this value as 2 (2 is the CloudCenter’s root administrator’s User ID).
          • To perform a health check on specific cloud regions, create and activate a new user with those specific regions and use that user’s User ID as value for this property. To get the User ID, use the v1 User Management APIs.
        ELK_LoginFor the ELK/Monitor host.
        • ELK User: The default ELK Username = logreader.
        • ELK Password: The default ELK Password is re@d0nly (zero between d and n) (change this password after the initial login – see Download Log File for additional context).
      3. Verify your changes and Exit the Monitor configuration wizard.

      4. Select Yes, to restart the Tomcat service for the changes to take effect.

      You have successfully configured the Monitor instance! You can now proceed to the Per CloudCenter Region Installation section and install the CloudCenter components for each Cloud.


      Back to Phase 4: Install Components

    4.  Update the MON_LB Configuration

      MON_LB 

      Load balancing can be done through HAProxy, NGiNX, Apache2, or a cloud that is natively available to services, like AWS Elastic Load Balancer (ELB). To configure the load balancer service and ensure Monitor load balancing, be sure to listen on port 8443 and balance the request at 8443 on both the MON_PRIMARY and MON_SECONDARY servers.

      See MON_LB Ports for the complete list of ports that need to be open for your deployment.

       

      The following load balancing configuration was performed on CentOS7.x VM with HAProxy for the Monitor VM.

      1. SSH into the VM instance using the key pair that you used to launch the MON VM.
      2. Install HAProxy as the root user.

        yum –y install haproxy
        
      3. Modify HAProxy config file as below

        vi /etc/haproxy/haproxy.cfg
        
        # configuration to listen on 8443 and loadbalance
         
        frontend httpsalt-in
            mode tcp
            log global 
            bind *:8443 
            default_backend mons
         
         
        backend mons
            mode tcp
            balance roundrobin
            option ssl-hello-chk  
            server mon1 <MON_PRIMARY_IP>:8443 check 
            server mon2 <MON_SECONDARY_IP>:8443 check 
        
      4. Start the HAProxy service and check the status, it should be active

         

        systemctl start haproxy
        systemctl status haproxy 
        

       

      Back to Phase 4: Install Components

  6.  Docker Image Upgrade

    Docker Image Upgrade

    Overview

    A Docker image upgrade is not required with every release. If a version requires the CloudCenter-supported Docker image to be upgraded, use the procedure provided in this page. 

    When using External Services, the CloudCenter platform launches a Docker container to run the scripts. You may also need to customize this container using the applicable packages.

    The Docker image can reside in one of two places:

    • In the CCO VM (default, co-located)
    • In a Standalone Docker VM

    Versions Requiring a Docker Image Upgrade

    If you upgrade to CloudCenter 4.4.x or 4.5.x from any earlier CloudCenter version, you must upgrade the CloudCenter supported Docker image.

    CloudCenter 4.4.x and later includes the following Docker image enhancements:

    See External Service > Error Handling for additional context.

    Prerequisites

    For each CloudCenter deployment that needs to be upgraded, verify the following prerequisites:

    • Backup your database and applications before you begin this process.
    • Verify that the Docker version is ≤ Docker 1.11. Issue the following command on your Docker server: docker version
      You only need to upgrade your Docker image if your Docker image is not at Docker 1.11.

    • Review the information provided in the Upgrade Overview section and validate the following requirements for the release to which you are upgrading:

      • Is an upgrade path available?
      • Is the core_upgrade.bin file required?
    • Ensure that a version file (/usr/local/osmosix/etc/version) exists in both CCOs that need to be upgraded.

    • Verify that the version file contains the correct version number (for example, if your current CloudCenter release version is 4.5.1, ensure that the corresponding version value is 4.5.1).

    • See the corresponding release notes for release-specific information on the CloudCenter version to which you are upgrading. For example, the CloudCenter 4.6.0 Release Notes.

    Backup Database

    Backup the exploded war files to a backup folder (the following example uses /mnt, you can change this directory as applicable).

    NOW=$(date +"%Y%m%d")
    bakdir="/mnt/bak/$NOW"
    mkdir -p $bakdir
    cd $bakdir

    Download Packages

    Download package files:

    See Installation Overview to understand the required components and the installation options.

    See Installer Overview to understand the types of files.

    1. SSH into the VM instance designated for this component by using the key pair that you used to launch the VM.
    2. Download the following required files for this component from software.cisco.com to the /tmp folder on that VM:

      • cco-installer.jar 
      • docker.tar
      • Dockerfile

    Select and Execute Your Upgrade Scenario

    Your upgrade process differs depending on your instance setup. Ascertain the following considerations before you begin the CCO upgrade.

    ScenarioInstance SetupRelated Section
    1
    • Instance 1 = CCO + Docker
    CCO and Docker on the Same Instance
    2
    • Instance 1 = CCO
    • Instance 2 = Docker
    CCO and Docker on Separate Instances
    Scenario1: CCO and Docker on the same Instance
     CCO and Docker are Co-Located
    1. Run the following commands on the CCO-Docker Instance:

      Ensure (by reviewing Upgrade Overview) that this step is required for your release upgrade path.

      sudo –i
      cd /tmp
      chmod 755 core_upgrade.bin
      ./core_upgrade.bin <ostype> <cloudtype> cco
      
    2. Run the following commands from your download folder.

      tar xvf docker.tar
      
    3. Run the following commands to update the Docker image file.

      cd cliqr-container-worker
      
      # Optional - To customize docker image, add the commands under RUN in Dockerfile 
      vi Dockerfile
       
      #Remove the existing image and create a new image
      docker rmi -f cliqr/worker
      docker build -t 'cliqr/worker:latest' .

     

    Scenario 2: CCO and Docker on Separate Instances
     CCO and Docker are not Co-Located
    1. Upgrade your CCO instance. See 2. CCO Upgrade for additional context.
    2. Run the following commands on the Docker Instance:

      Ensure (by reviewing Upgrade Overview) that this step is required for your release upgrade path.

      sudo –i
      cd /tmp
      chmod 755 core_upgrade.bin
      ./core_upgrade.bin <ostype> <cloudtype> docker
      
    3. Run the following commands from your download folder.

      tar xvf docker.tar
      
    4. Run the following commands to update the Docker image file.

      cd cliqr-container-worker
      
      # Optional - To customize docker image, add the commands under RUN in Dockerfile 
      vi Dockerfile
       
      #Remove the existing image and create a new image
      docker rmi -f cliqr/worker
      docker build -t 'cliqr/worker:latest' .

    Post Upgrade Tasks

    1. Verify Your Upgrade – Verify that the Docker version is ≤ Docker 1.11. Issue the following command on your Docker server:

      # docker version
      Client:
       Version:      1.11.1
       API version:  1.23
       Go version:   go1.5.4
       Git commit:   5604cbe
       Built:        Wed Apr 27 00:34:42 2016
       OS/Arch:      linux/amd64
      Server:
       Version:      1.11.1
       API version:  1.23
       Go version:   go1.5.4
       Git commit:   5604cbe
       Built:        Wed Apr 27 00:34:42 2016
       OS/Arch:      linux/amd64
    2. Issue the following command and verify that the output is as follows:

      # docker images
      REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
      <none>              <none>              d54feac3ed0d        39 seconds ago      196.7 MB
      centos              7                   980e0e4c79ec        10 days ago         196.7 MB

     

  • No labels