// removed jquery ui css and js

Application Deployment and VM States

 

Deployment States

CloudCenter displays color-coded states for deployed applications in the Deployments page. The following table provides the description for each Deployment state.

UI Deployment State (Alphabetical Listing)

Lifecycle Order

DescriptionAPI Enumeration for deploymentStatusAPI Enumeration for jobStatus

Deployment Rollback

9There was an error during upgrade, because of which the deployment was rolled back.DeploymentRollbackJobRollback 

DeploymentRollback Error

10There was an error in running the rollback script after an upgrade failure.DeploymentRollbackErrorJobUpgradeRollback
Error 
Error22The deployment encountered an error state.DeploymentErrorJobError


Finished20The deployment has finished executing or was canceled successfully. The nodes remain available for reuse until the next hour boundary, at which time they will be cleaned up if they are not reused.DeploymentDoneJobFinished  
Migrating11The deployment is in the process of migrating to another cloud (see Termination for additional context).DeploymentMigrating

JobMigrating


Migration Error13The deployment encountered an error while migration to another cloud was in process.DeploymentMigrationErrorJobMigrationError

Migrate Pending

12The deployment is pending approval for migration.DeploymentMigratePendingJobMigratePending
Pending2If you launched a deployment in an environment that required approval and you are waiting for approval.DeploymentPending

JobPending


Reconfiguring6The node is being reconfigured with the new settings.DeploymentReconfiguringJobReconfiguring
Rejected4If you launched a deployment in an environment that required approval and the phase approver rejected your request (see the previous section to set up approval).DeploymentRejected

JobRejected

Resuming16

The deployment was suspended and is in the process of resuming.

DeploymentResumingJobResuming 
Running/Deployed3

The CCO has completed the orchestration steps and the job is running. The application is successfully deployed and all nodes are up and running (detected via a heartbeat within the heartbeat timeout interval window).

DeploymentDone(“Deployed")

JobRunning
Scaling5The deployment is in the process of increasing or decreasing the number of nodes for an app tier.DeploymentScalingJobScaling 

 

Stopped18The deployment has been stopped and all nodes associated with the job have been cleaned up.DeploymentStoppedJobStopped 
Stopping17The deployment is being stopped.DeploymentStoppingJobStopping
Stopping Error19An error occurred when the deployment was being stopped.DeploymentStoppingError

JobStoppingError


Submitted1The CCO has received the request and started to process it but the orchestration steps are not yet completed.DeploymentSubmittedJobSubmitted 

 

Suspended15The application VMs in this deployment have been powered off or shut down (not terminated).DeploymentSuspendedJobSuspended 
Suspending14

The application is in the process of being suspended.

DeploymentSuspendingJobSuspending
Terminated21The application VMs and external volumes launched as part of the deployment are terminated.DeploymentKilledJobCanceled


 

Upgrading7The deployment is being upgraded to a more recent application profile version. Application profiles can have numerous versions. When launching an application, you may prefer to launch using the latest version of the profile. If you choose to do so, the deployment lists this state.DeploymentUpgradingJobUpgrading
Upgrading Error8The deployment encountered an error when upgrading the application profile version.DeploymentUpgradeError

JobUpgradingError 

VM (Node) States

Use the Deployments > Application Deployments > Click Application page to view the VM state for each job:

VM State (Alphabetical Listing)
Lifecycle OrderSupported ActionsDescriptionAPI Enumeration for nodeStatus 
Cleaned11NoneThe node is cleaned of all its configurations and dependent services based on the cleanup script provided by the user.NodeCleaned 

 

Error12NoneThe node encountered an error as a result of the most recent action. You can see error details in the job status message. For example: Instance bootstrapping is timed out, possibly due to incorrect or missing agent bundle for node...

NodeError 

Not Reachable13None

The node is offline as a result of detecting a heartbeat loss.

NodeNotReachable
Reachable14NoneThe node is reachable after being in the not reachable state for any period of time.NodeReachable 
Rebooted8NoneThe node was restarted.NodeRebooted 
Resuming6NoneThe node was suspended and is in the process of resuming.NodeResuming 
Resumed7NoneThe node has resumed after the suspend state.NodeResumed 
Running3
  • Power Off
  • Reboot
The node is started and the agent is running. This is the normal status for a running node.NodeReady 
Started2
  • Power Off
  • Reboot
The node has started successfully and the initialization is complete.

NodeStarted 

Starting1None

This is the first status message from the agent to indicate that the node initialization has started.

NodeStarting 
Suspending4NoneThe node is in the process of being suspended.

NodeSuspending

Suspended5Power OnThe node has been powered off or shut down (not terminated).NodeSuspended 
Terminating9NoneThe node is in the process of being terminatedNodeTerminating
Terminated10None

The node is terminated in the cloud. Termination occurs as a result of the reduce action or the deployment termination action.

NodeTerminated 
Node not found15None

The node is not found by the CloudCenter platform either because the node is down for an unknown reason.

This state is also displayed when an imported node is deleted from the cloud provider console instead of the CloudCenter platform

NodeNotFound

Orchestration Lifecycle Threshold Settings

The success of the CCO lifecycle depends on the successful deployment of an application. You can set the timeout thresholds for some phases of the orchestration process. These threshold settings enable the CCO to proceed with the remainder of the lifecycle process when deploying an application. Some thresholds can be set in the gateway.properties file in the CCO and overridden through application parameters.

Any change to the gateway.properties requires a restart of Tomcat services. In case of a HA setup, this can be an intricate process.

Instead, you can also configure these settings at the application profile level and fine tune the settings for each job submission. By doing so, you do not need to restart the Tomcat service.

This is the high-level orchestration lifecycle process when launching VMs:

  1. Launch the required VMs as part of the application deployment process.
  2. The VMs go through the bootstrap process.
    1. If all the VMs are successfully bootstrapped (preInit) within their configured bootstrap timeout, then the VM bootstrap process is complete.
    2. If some of the VMs reach the bootstrap timeout, then they are terminated. If the number of healthy VMs is less than the configured minimum number of nodes, then all VMs are terminated and the deployment ends in a failure.
    3. If the number of healthy VMs is greater than or equal to the minimum number of nodes, then the CCO retries once to launch the failed VMs. Once again, the CCO terminates any VMs that reach the bootstrap timeout.
    4. If  the minimum number of VMs required for the deployment is met, the CCO proceeds to initialize the healthy VMs, thus ensuring partial success, even if all the VMs cannot be bootstrapped successfully.
  3. The CCO runs the initialization scripts on the healthy VMs.
    1. If the initialization scripts running on the VMs do not complete within the node ready timeout then the CCO terminates those VMs, without any retries.
    2. Once the initialization scripts either complete or time out for all the VMs, the CCO checks to see if the initialization was successful for the minimum number of nodes.
    3. If initialization was successful for the minimum number of nodes, then the deployment is completed for those VMs.
    4. Otherwise, the deployment ends in a failure.

The orchestration lifecycle steps for pre-initialization (pre-init) and initialization in the high-level process relies on the numbers that you configure for the following settings.

CloudCenter 4.8.0.2 and 4.8.1.2 and later

CloudCenter introduced the concept of partial success for the orchestration lifecycle process. In other CloudCenter releases, when you launched multiple VMs and even one of those VMs failed, the entire process was made to fail by the CloudCenter platform.

CloudCenter 4.8.0, 4.8.0.1, 4.8.1, and 4.8.1.1

If the CCO does not get any message from the application VM within this interval, the deployment ends in a failure and all nodes are terminated. 

SettingApplication Parametergateway.properties FileDescription
Bootstrap Timeout
(time in seconds)

cliqrNodeBootstrapTimeout

maximum.bootstrap.wait.time

Default = 3600

The Bootstrap Timeout setting identifies the maximum time available for VMs to bootstrap after they are launched. 

In CloudCenter 4.8.0.2, the deployment behaviour when few VMs fail to bootstrap is governed by the Minimum Number of Nodes setting. If some VMs time out and face a bootstrap failure, then those VMs are terminated. If the number of VMs that bootstrap successfully is greater than (or equal to) the Minimum Number of Nodes, then the CCO tries a fresh launch of the failed VMs.

Minimum Number of VMs (VM count)

deploymentMinClusterSize

Not available

You can specify the Minimum Number of Nodes and Maximum Number of Nodes in the General Settings section for each N-tier application. While these settings are important in the scaling context, they are also important in the bootstrap timeout context as CloudCenter 4.8.0.2 eases the failure process for VMs that have reached the timeout limit. The deployment proceeds as long as the number of VMs that are successfully bootstrapped (despite the Bootstrap Timeout) is greater than or equal to the Minimum number of Nodes setting and ensures partial success even if the system encounters a bootstrap failure for a few VMs.

To view the VM Details for each terminated VM:

  1. Navigate to the Job Deatils page for the required deployment
  2. Uncheck the hide terminated nodes option by unchecking the hide terminated nodes option.
Node Ready Timeout
(time in seconds)
cliqrNodeReadyTimeout
node.ready.timeout

Default = 36000

Sometimes, a VM may function but the associated application service may not have started on this VM when the initialization scripts are stuck or take too long to complete. The Node Ready Timeout setting ensures that the timed out VMs are terminated. In this case, the remaining VMs complete the application process as long as the requirement for the Minimum Number of Nodes is met. The termination of the VMs affected by the time out does not hinder the completion process for the remaining VMs and ensures partial success for the entire Orchestrator lifecycle process.

The nodes that are terminated by this process are not given another chance – the CloudCenter platform does not re-launch these terminated VMs – instead, it displays an Error message for this kind of a timeout. 


Permitted Job Operations and Actions

For each valid state, CloudCenter displays corresponding operations and actions based on configured permissions. 

Legacy Jobs (Parallel, Batch, Interactive jobs etc )

CloudCenter displays the following actions for Legacy jobs:


Job StatusSupported ActionsRequired User Privilege (See perms and Permission Control)
InProgress

Stop

User must have Manage privilege

Rerun

User must have Manage privilege on own/other deployments

SubmittedStop

User must have Manage privilege

RunningStop

User must have Manage privilege

Promote

User must have Move_out privilege

Rerun

User must have Manage privilege on own/other deployments

FinishedPromoteUser must have Move_out privilege
PendingApproveUser must have Approve privilege
DenyUser must have Approve privilege


N-Tier Jobs

CloudCenter displays the following actions for N-tier jobs:


Job StatusSupported ActionsRequired User Privilege (See perms and Permission Control)
Error

Terminate

Terminate and Hide

  • User must have Manage privilege on own/other deployments
  • Deployment must  not be terminated
Rejected

Terminate

Terminate and Hide

Hide

  • User must have Manage privilege on own/other deployments
  • Deployment must  not be terminated
Stopped Resume
  • User must have Manage privilege on own/other deployments

Terminate

Terminate and Hide

  • User must have Manage privilege on own/other deployments
  • Deployment is not terminated
Stopping Error

Terminate

Terminate and Hide

  • User must have Manage privilege on own/other deployments
  • Deployment is not terminated
  • Deployed 
  • Upgrade Rollback
  • Upgrade Rollback Error
  • Upgrade Error
Suspend
Stop
  • User must have Manage privilege on own/other deployments
  • Prevent Termination/terminateProtection must be disabled (off)
  • Suspend is not supported

Terminate

Terminate and Hide

  • User must have Manage privilege on own/other deployments
  • Prevent Termination/terminateProtection must be disabled (off)
  • Deployment is not terminated
Upgrade
  • User must have Manage privilege on own/other deployments
Promote
  • User must have Move_out privilege
Migrate
  • User must have Move_out privilege
  • Prevent Termination/terminateProtection must be disabled (off)
Enable/Disable Terminate Protection
  • User must have Manage privilege on own/other deployments
  •  Termination/terminateProtection must be supported
  • Migrating
  • Upgrading
  • Reconfiguring
Suspend
  • User must have Manage privilege on own/other deployments
  • Prevent Termination/terminateProtection must be disabled (off)
  • Suspend is supported
Stop
  • User must have Manage privilege on own/other deployments
  • Prevent Termination/terminateProtection must be disabled (off)
  • Suspend is not supported
Enable/Disable Terminate Protection
  • User must have Manage privilege on own/other deployments
  • Prevent Termination/terminateProtection must be supported
  • In Progress
  • Submitted

Terminate

Terminate and Hide

  • User must have Manage privilege on own/other deployments
  • Prevent Termination/terminateProtection must be disabled (off)
  • Deployment is not terminated
Enable/Disable Terminate Protection

User must have Manage privilege

Migration Error

Terminate

Terminate and Hide

  • User must have Manage privilege on own/other deployments
  • Prevent Termination/ terminateProtection must be disabled (off)
Enable/Disable Terminate ProtectionUser must have Manage privilege
Suspended

Terminate

Terminate and Hide

  • User must have Manage privilege on own/other deployments 
  • Prevent Termination/terminateProtection must be disabled (off)
Resume

User must have Manage privilege on own/other deployments

Enable/Disable Terminate ProtectionUser must have Manage privilege
  • Suspending
  • Resuming
  • Stopping

Terminate

Terminate and Hide

  • User must have Manage privilege on own/other deployments
  • Prevent Termination/terminateProtection must be disabled (off)
Enable/Disable Terminate ProtectionUser must have Manage privilege
  • Pending
  • Migrate Pending
 
ApproveUser must have Approve privilege
RejectUser must have Approve privilege
Enable/Disable Terminate ProtectionUser must have Manage privilege
Scaling  

Suspend

Terminate

Terminate and Hide

  • User must have Manage privilege on own/other deployments
  • Prevent Termination/terminateProtection must be disabled (off)
Enable/Disable Terminate ProtectionUser must have Manage privilege
TerminatedHideUser must have Manage privilege

Auto-Clean up of Resources

A Cron utility runs in the background at 02:00 hours (CCM system time) and automatically terminates and deletes jobs/deployments that meet the following conditions:

  • The job/deployment is in a JobError state.

  • The job/deployment is running on a Managed VM (see VM Management for additional context).

  • The instances/volumes related to this job/deployment continue to run or have not been terminated by the user, they will be terminated in the background.

  • The troubleshooting parameter, cliqrIgnoreAppFailure = false (see Troubleshooting Parameters for additional context).

    If cliqrIgnoreAppFailure = true, then the instances/volumes are left untouched.

The auto-clean utility:

  • Retrieves the list of jobs/deployments (latest 500 ordered by time) that meet the conditions listed above.

  • Deletes the related instances of the job/deployment at 02:0 hours (CCM system time).

  • Updates the status for this job/deployment in the VM Details page – you can check if the node has been terminated by this background Cron utility by referring to the corresponding VM details page in the CCM UI.



© 2017 Cisco Systems