Kubernetes Troubleshooting

Available Documentation

Log Files

Based on the error message that you see in the UI, you could perform basic troubleshooting steps if you have access to both the Kubernetes setup and to the CloudCenter platform:

IssueError Reference Location
Errors returned by the Kubernetes clusterGo to the Kubernetes dashboard and look for the event messages and log into the pod that you created for the CloudCenter platform.
Kubernetes cluster API interaction issuesRefer to the log file at /usr/local/cliqr/logs/containerblade.log in the CCO instance.
Orchestration or lifecycle issues

Refer to the log file at /usr/local/cliqr/logs/gateway.log in the CCO instance.

You may find the following warning message in the Kubernetes CCO logs – you can safely ignore this message as it does not impact product functionality.

WARNING!!! The linux bootstrap URL might be valid: http://build-rel.cliqr.com/..../bootstrap-cliqr-init.sh. If the CCO cannot access the file, all deployments would fail!
CCM-specific issues,Refer to the log file at /usr/local/cliqr/logs/mgmtserver.log in the CCM instance.

Failure to Deploy a New Container

If you are unable to deploy a new container, revisit the following steps to ensure that you follow the prescribed process:
See Configure a Kubernetes Cloud for additional details.

  • Check you clusterrole assignment and ensure that it is set to cluster-admin:
    Role binding the service account to the admin is essential to access the dropdown in the Cloud Defaults page.

    kubectl create clusterrolebinding <name> --clusterrole=cluster-admin
  • If the details in the previous bullet did not address the issue, then create a dedicated service account for CloudCenter. The following example, walks you through the required steps for this process

    kubectl create serviceaccount cloudcenterSA
    
    kubectl create clusterrolebinding cloudcentersabinding --clusterrole=cluster-admin --serviceaccount=default:cloudcenterSA
    
    #The following commands use jq. If not installed, you can install it using this command: sudo apt-get install jq
    
    kubectl get serviceaccount cloudcenterSA -o json | jq -Mr '.secrets[].name'
    
    #The cloudcenterSA-token-XXXXX name is unique and is gathered from this command -- be sure to replace the token in the following command
    
    kubectl get secrets cloudcenterSA-token-XXXXX -o json | jq -Mr '.data.token' | base64 -d
  • If the above two workarounds did not address the issue, verify the Kubernetes setting for the Default API version or the API version override. The API version is optional and not required.

    • To verify the version, access the Kubernetes Region UI > Kubernetes Settings.

    • If you have configured a specific API version in your environment, try leaving it blank and retry the deployment

Insufficient Permission

  • Issue: Your deployment fails with forbidden (networkpolicies.extensions is forbidden) or Code 403 (Received status: Status(apiVersion=v1, code=403) in the containerblade.log

  • Reason: The Service Account is associated with cluster role has insufficient permissions or has a non-existing cluster role

  • Solution: Update the Service Account to map to right cluster role. See Configure a Kubernetes Cloud for additional details.

Incorrect API Version

  • Issue: Your deployment fails with a Code 400 (no kind Network Policy is registered for v1 version. Received status:Status(apiVersion=v1, code=400) error in the containerblade.log

  • Reason: The API version for the object (Network Policy) in this case is not sent correctly. Either the user specified a wrong version or the CloudCenter platform could not auto-detect the version.

  • Solution: Try one of the following solutions:

    • Leave the Default and Override API versions blank – this often corrects the issue.

    • Alternately, find the right version by examining an existing object instance in the Kubernetes dashboard or using the kubectl GET API. In the CloudCenter Kubernetes region settings, set the API Version Override field with the identified version. For example, “NetworkPolicy:v1beta1”.




  • No labels
© 2017-2019 Cisco Systems, Inc. All rights reserved