CloudStack Kubernetes Service by shwstppr · Pull Request #3680 · apache/cloudstack

shwstppr · 2019-11-12T11:11:47Z

Description

Specification document https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Kubernetes+Service

Kubernetes Service plugin adds Kubernetes integration in the CloudStack. Plugin is disabled by default and an admin can enable it using the Global Setting. It enables users to run containerized services using Kubernetes clusters.

Kubernetes Service plugin uses a CoreOS based template for node VMs for the Kubernetes cluster. For installation of Kubernetes binaries on cluster nodes, a binaries ISO can be created for a particular Kubernetes version and can be added as a supported version by an admin. This allows faster, offline installation Kubernetes binaries and docker images along with support for adding multiple versions of Kubernetes for upgrades and running different clusters.

For deployment and setup of Kubernetes on cluster nodes, the plugin uses the Kubernetes tool, kubeadm. kubeadm is the command-line tool for easily provisioning a secure Kubernetes cluster on top of physical or cloud servers or virtual machines. Under the hood, master node(s) of the cluster starts a Kubernetes cluster using kubeadm init command with a custom token and worker nodes join this Kubernetes cluster using kubeadm join command with the same token. More about kubeadm, https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/. Weave Net CNI provider plugin is used for cluster netoworking. More about Weave Net provide plugin, https://www.weave.works/docs/net/latest/kubernetes/kube-addon/.

To access Kubernetes dashboard securely, the plugin provides access to kubeconfig file data which allows using the Kubernetes tool, kubectl, to run proxy locally and thereby access dashboard. More about kubectl, https://kubernetes.io/docs/reference/kubectl/overview/

The service allows creation of Kubernetes clusters using UI or API. Both UI and API functionalities to list, delete, scale upgrade, stop and start these clusters.

Enabling the Kubernetes Service

Kubernetes Service plugin is disabled by default. To enable it, go to Global Settings and set the following global configuration to true:

cloud.kubernetes.service.enabled

Restart the Management Server to enable the set configuration values.

service cloudstack-management restart

Once the Kubernetes service is running new API will become accessible and UI will start show Kubernetes Service tab.

Kubernetes Supported Versions
For faster and offline installation of Kubernetes and docker images on node virtual machines of a Kubernetes cluster, the service provides the functionality to manage supported Kubernetes versions.

Script named create-kubernetes-binaries-iso.sh has been provided in the cloudstack-common package for creating a new setup iso with the desired version of Kubernetes binaries and corresponding docker images.

Usage,

./create-kubernetes-binaries-iso.sh OUTPUT_PATH KUBERNETES_VERSION CNI_VERSION CRICTL_VERSION WEAVENET_NETWORK_YAML_CONFIG DASHBOARD_YAML_CONFIG

eg,

./create-binaries-iso.sh ./ 1.12.5 0.7.1 1.12.0 "https://cloud.weave.works/k8s/net?k8s-version=1.12.5" https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta1/aio/deploy/recommended.yaml

ISOs created using the script can be added as supported Kubernetes version for Kubernetes service.

Working with Kubernetes supported version
New Kubernetes versions can be added using both UI and API. UI provides the following form to add new supported version,

addKubernetesSupportedVersion API can be used by an admin to add a new supported version for the service. It takes following input parameters,

name (the name of the Kubernetes supported version)
semanticversion (the semantic version of the Kubernetes; Required)
zoneid (the ID of the zone in which Kubernetes supported version will be available)
isoid (the ID of the binaries ISO for Kubernetes supported version)
url (the URL of the binaries ISO for Kubernetes supported version)
checksum (the checksum value of the binaries ISO)

> add kubernetessupportedversion name=v1.13.2 semanticversion=1.13.2 url=http://172.20.0.1/files/setup-1.13.2.iso zoneid=34d23dd5-5ced-4e8b-9b0a-835a0b8ae2a6

{
"kubernetessupportedversion": {
"id": "6668e999-fe6c-4a91-88d8-d10bcf280d02",
"isoid": "78d45e9b-a482-46f4-8cbc-cf7964564b85",
"isoname": "v1.13.2-Kubernetes-Binaries-ISO",
"isostate": "Active",
"semanticversion": "1.13.2",
"name": "v1.13.2",
"supportsha": false,
"zoneid": "34d23dd5-5ced-4e8b-9b0a-835a0b8ae2a6",
"zonename": "KVM-advzone1"
}
}

Note:
The minimum Kubernetes version that can be added in the service is 1.11.

listKubernetesSupportedVersion API can be used to list existing supported versions. It takes id parameter as input to list details of a single supported version.

updateKubernetesSupportedVersion API has been provided for admins to update an existing supported version to mark them as enabled or disabled.

deleteKubernetesSupportedVersion API has been provided for admins to delete an existing supported version if it is not used by any Kubernetes cluster in the service. id parameter of the API can be used to pass Kubernetes version to be deleted.

Note:
addKubernetesSupportedVersion, updateKubernetesSupportedVersion and deleteKubernetesSupportedVersion API are available to admin only

Kubernetes clusters
Kubernetes service provides the functionality of running and managing Kubernetes clusters. Highly available, scalable Kubernetes clusters can be created to run containerized deployments without having trouble to set up Kubernetes on each container node manually. Service will automatically provision the desired number of virtual machines as per cluster size using the binaries from the given Kubernetes version. Additionally, the service provides the functionality to upgrade and scale clusters. Running clusters can be upgraded to a newer minor or patch Kubernetes version at a time. Running clusters can also be scaled for the number of worker nodes up and down and for the service offering used by each node.

The provides functionality to create Kubernetes clusters for Shared, Isolated and VPC networks in the CloudStack but such networks must be accessible to CloudStack management server for provisioning virtual machines of the cluster. Template and default network offering must be set Global Settings for the service to create Kubernetes clusters.

Following Global Settings value must be set to the name of Template to be used for deploying node virtual machines while creating a Kubernetes cluster,

cloud.kubernetes.cluster.template.name.hyperv
cloud.kubernetes.cluster.template.name.kvm
cloud.kubernetes.cluster.template.name.vmware
cloud.kubernetes.cluster.template.name.xenserver

CoreOS templates for CloudStack can be found here, http://dl.openvm.eu/cloudstack/coreos/x86_64/

For CoreOS template with VMware, rootDiskController should be pvscsi and nicAdapter should be Vmxnet3 else root disk resizing might fail apart from some other unexpected behaviours in a cluster deployment.

Following Global Setting value must be set to the name of Network Offering to be used for creating a new network when no network has been selected while creating a Kubernetes cluster,

cloud.kubernetes.cluster.network.offering

A new network offering named DefaultNetworkOfferingforKubernetesService has been added since 4.14.0

Note:

Multi-master, HA cluster can be created for Kubernetes version 1.16 and above only.
While creating multi-master, HA cluster over a shared network, an external load-balancer must be manually setup. This load-balancer should have port-forwarding rules for SSH, Kubernetes API server access. Service assumes SSH access to cluster nodes is available from port 2222 to (2222 + cluster node count -1). Similarly, for API access 6443 must be forwarded to master nodes. Over CloudStack isolated network these rules are automatically provisioned.

Managing Kubernetes clusters
For Kubernetes cluster management, service provides create, stop, start, scale, upgrade and delete APIs and similar features in the UI. APIs available,
createKubernetesCluster

> create kubernetescluster name=Test description=Test-Cluster zoneid=34d23dd5-5ced-4e8b-9b0a-835a0b8ae2a6 size=1 noderootdisksize=10 serviceofferingid=a4f280a1-9122-40a8-8f0c-3adb91060f2a kubernetesversionid=6668e999-fe6c-4a91-88d8-d10bcf280d02
{
"kubernetescluster": {
"associatednetworkname": "Test-network",
"cpunumber": "4",
"description": "Test-Cluster",
"endpoint": "https://172.20.20.12:6443/",
"id": "74e3cc02-bbf7-438f-bfb0-9c193e90c1fb",
"kubernetesversionid": "6668e999-fe6c-4a91-88d8-d10bcf280d02",
"kubernetesversionname": "v1.13.2",
"masternodes": 1,
"memory": "4096",
"name": "Test",
"networkid": "148af2cb-4b94-42a2-b701-3b6aa884cbb0",
"serviceofferingid": "a4f280a1-9122-40a8-8f0c-3adb91060f2a",
"serviceofferingname": "CKS Instance",
"size": 1,
"state": "Running",
"templateid": "17607ed6-1756-4ed7-b0f4-dbab5feff5b2",
"virtualmachineids": [
"da2cb67e-e852-4ecd-b16f-a8f16eb2c962",
"4179864a-88ad-4d6d-890c-c9b73c53589b"
],
"zoneid": "34d23dd5-5ced-4e8b-9b0a-835a0b8ae2a6",
"zonename": "KVM-advzone1"
}
}

stopKubernetesCluster can be used to stop a running cluster. It takes id of the cluster as the input parameter.

startKubernetesCluster can be used to start a stopped cluster. It takes id of the cluster as the input parameter.

scaleKubernetesCluster can be used to scale a running or stopped cluster.
Note:
Only upscaling is supported while scaling clusters for service offering.
Olny a running Kubernetes cluster can be scaled for size.

upgradeKubernetesCluster can be used to upgrade a running cluster.

deleteKubernetesCluster can be used to delete a cluster. It takes id of the cluster as the input parameter.
getKubernetesClusterConfig API can be used to retrieve kubeconfig file data for a cluster. It takes id of the cluster as the input parameter.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

shwstppr · 2019-12-05T07:59:10Z

@blueorangutan package

shwstppr · 2019-12-06T07:36:52Z

@blueorangutan package

yadvr · 2019-12-10T15:31:45Z

@shwstppr does this also have a doc PR to #apache/cloudstack-documentation?