...
The document has the Edge-node Cluster design considerations and detailed procedures for Cluster Creation; Network Instance Deployment; Volume Instance Deployment; Application Deployment; Application Migration and Status for both Controller and Onsite edge-nodes. It also has the Edge-Node API for the clustering operation.
EVE development has the extension to build the kubevirt image with kubernetes and kubevirt managing the user VMs, see document Cluster compute and storage support in EVE. This Edge-Node Clustering is built on top of that development.
Terminology
- Cluster - Edge-Node Cluster, with Kubernetes cluster on EVE edge-nodes, at least 3 local network connected nodes, managing the User applications on EVE devices at customer sites.
- Master - Kubernetes master node or server node, which runs with control and worker functions. In the initial phases, we only support exactly 3 master nodes for a cluster.
- Cluster-Interface - The EVE device physical Interface which the cluster uses for kubernetes communication among the kubernetes nodes. This may or may not be part of the EVE device management ports. It can be a Kubernetes dedicated ‘App Shared’ port on the device.
- Cluster-Prefix - A dedicated IP prefix is used on the ‘Cluster-Interface’ for kubernetes cluster to communicate with. This is configured regardless of the DHCP or manual usage on the site. It can be the secondary IP subnet on the interface.
- Seed-server - One of the master nodes is designated by the configuration to be the ‘Seed-server’ in the cluster. The ‘seed-node’ is responsible for generating the initial kubernetes configuration, the certificates, and the token in the cluster for other master nodes to join. A ‘seed-node’ may be changed from one device to another if the original ‘seed-node’ can not be brought up again.
- SSIP - Seed-Server IP address. This is used during initial bootstrap the cluster for the Non-Seed-Node to join the cluster.
- PVC - Persistent Volume Claim. User application and its volumes handle which is managed from kubernetes and longhorn systems.
- Kubevirt Image - EVE image compiled with ‘kubevirt’ HV mode, instead of the currently offered ‘kvm’ and ‘xen’ modes. All the cluster devices need to run this type of image.
- Single Node - A single kubernetes edge-node running with EVE Kubevirt image, the edge-node launches VM and container using kubernetes mechanism. This document does not discuss the details of single-node. If a cluster loses two edge-nodes on-site, then the only live edge-node can reduce to the single-node mode.
- Kube Container - An EVE service which is a `services` container, running in system containerd, parallel to wwan and pillar, called ‘kube’, which is responsible for kubernetes cluster either on Edge-Node Cluster or on a single-node. It runs the K3S, Longhorn, KubeVirt, etc. and EVE specific changed open-source kubernetes software.
- HA - High Availability for Cluster. In particular this is high availability for the kubernetes control plane functions. For the 3 nodes cluster, it can have one node down or disconnected but the cluster will still be functional for the kubernetes operations.
- ResourceLabel - Node or edge-node resource label. It is a pair of key and value strings, to represent some particular resource on an edge-node. This is used for Apps, NIs and VIs placement and App placement and migration be restricted on certain edge-nodes
- Auto-Migration - An user application can be automatically migrated from original node onto any node which has resources.
- Manual-Migration - An user application can only be migrated from the original node onto another node by user’s configuration change through the controller.
- Server Node - Kubernetes server or master node. We initially only support the cluster with 3 server nodes. The server nodes run control plane processes such as API-Server, Scheduler, Controller-Manager, Etcd, etc. They also serve work loads for running VMs and containers in our case. Unlike the ‘Server’ nodes, the agent nodes don’t have most of the kubernetes controller functions and they only run work loads.
- Agent Node - Kubernetes agent node or worker node. It does not run most of the processes with controller plane functionalities of the kubernetes cluster. Those agent nodes can not be one of the 3 HA controller kubernetes nodes.
- NI - Network Instance of edge-nodes
- VI - Volume Instance of edge-nodes
- DNId - Designated Node ID for a cluster application. The UUID of the edge-node in the cluster responsible for launching the application.
...
- Need to extend the type ‘filesystem storage’, currently only allows ‘Block Storage’ for non image usage.
- Same as in the NI case, the same VI name and UUID can be deployed onto each of the edge-nodes in the cluster.
- Same as in the NI case, may want to let users pick which edge-nodes need this VI or assign the ResourceLabel requirement.
- Same as in the NI case, if the ResourceLabel list exists, it does not need to be downloaded to the edge-nodes.
- Once each edge-node VI configuration is buildbuilt, it will be used in the device configuration for each edge-node selected
- If the VI does not need to be replicated by the cluster, then the user should only pick one edge-node
- Same as above NI question, should we reuse the same ‘VI’ object or create a new one for Edge-node Cluster VI?
- Normally once a VI is configured independent of the Application, it is downloaded onto the edge-node. This cluster case, we should allow users to pick up to three devices for this VI in the cluster. When the App is attached with this VI and if the App device selection does not match the VI device selection, we generate an error and ask user to modify. In the ‘Manual Migration’ case, only the DNId will get this VI configuration. In the ‘Auto-Migration’ of the App case, the other nodes may also get the VI configuration. A new DNId string is added to the ‘Volume’ API.
...
This is an example of the new ‘cluster’ API, is part of the device configuration of EVE API
message EdgeDevConfig {
...
// cluster configuration
EdgeNodeCluster cluster;
}
message EdgeNodeCluster {
// cluster name, in case it has multiple cluster on the same site
string ClusterName cluster_name;
// cluster UUID
string ClusterID cluster_id;
// Cluster-Interface
, for example "eth1"
string ClusterInterface cluster_interface;
// The ‘cluster-prefix’ IP address of the ‘Cluster-Interface’, e.g. 10.244.244.2/28
string ClusterIPPrefix cluster_ip_prefix;
// This device is ‘seed-server’ or not
bool IsSeedServer;
// This device is an ‘Agent’ node
bool IsAgent Is_agent;
// SSIP - Seed- Server IP address to join during bootstrapthe cluster. E.g. 10.244.244.1
string SeedServerIPAddress join_server_ip;
// encrypted token string, use edge-node TMP to decrypt
string EncryptedClusterToken;
// Resource label list, may not be in first phases
map <string, string> ResourceLabel;
}org.lfedge.eve.common.CipherBlock encrypted_cluster_token;
}
App Instance App Instance API
This is an example of the change of the ‘AppInstanceConfig’ API
message AppInstanceConfig {
…
// This edge-node UUID for the Designated Node for the Application
string DesignatedNodeID;
// If the App allows automatically migrated or not
bool AutoMigration;
// a set of ResourceLabels for node selection (may not be in first phases)
map<string, string> NodeSelectLabels string designated_node_id;
…
}
Volume API
This is an example of the change of the ‘Volume’ API
...
message Volume {
…
// To inform the edge-node if the device receives this Volume is
// responsible to create volume, convert PVC or not
String DesignateNodeID;
// replicated to devices with UUID array, number arrange from 0 to 2
repeated string ReplicaNodeID;
}
App Info Message API
string designated_node_id;
}
App Info Message API
This is an example of the change of the ‘Info’ API
enum ZInfoClusterNodeStatus {
Z_INFO_CLUSTER_NODE_STATUS_UNSPECIFIED;
Z_INFO_CLUSTER_NODE_STATUS_READY; // cluster reports our node is ready
Z_INFO_CLUSTER_NODE_STATUS_NOTREADY; // cluster reports our node is ready
Z_INFO_CLUSTER_NODE_STATUS_DOWN; // cluster API server can not be reached
}
message ZInfoClusterNode {
ZInfoClusterNodeStatus node_status;
}
message ZInfoMsg {
oneof InfoContent {
...
ZInfoClusterNode cluster_node;
}
}
message ZInfoApp {
…
Message ZInfoApp {
…
// The App in cluster mode is currently running on this edge-node
bool ClusterAppRunning;
… cluster_app_running;
}