...
The following table summarizes all resources. Original resources - either from the higher-level Adam abstraction or from the native EVE config - are in black, reused native Kubernetes resources are in blue, custom resources are in green. Where a native resource inherently works well, even the custom resource column will use the native resource, marked in blue.
EVE resource | Native resource | Custom resources |
---|---|---|
Onboarding | OnboardCertificate OnboardCertificateAuthority DeviceCertificateAuthority | |
Device serial | Device | |
Device certificate | Device property | |
Edge Device | Node | Device |
Global options | ConfigMap | ConfigMap |
Edge Device options | Annotations | Device Annotations |
Edge Application Instance | Pod | Application |
Base OS | Device Annotation | |
Device Config | Device properties | |
Network Config | NetworkConfig | |
Device Network | DeviceNetwork | |
Application Network | Annotations | |
Volume | Persistent Volume | Volume |
Data Store | StorageClass | StorageClass |
Content Tree | Image | Image |
Scheduling (controller) | Deployment DaemonSet | ApplicationDeployment ApplicationDaemonSet |
Items that require special treatment:
...
The Device is very similar to the native Node, except that specification items that would be loaded into annotations are made part of the core spec.
Code Block | ||
---|---|---|
| ||
apiVersion: eve.lfedge.org/v1beta1 |
...
kind: Device |
...
metadata: |
...
labels: |
...
beta.kubernetes.io/arch: amd64 |
...
beta.kubernetes.io/instance-type: eve |
...
beta.kubernetes.io/os: linux |
...
kubernetes.io/arch: amd64 |
...
kubernetes.io/hostname: eve-device.lab1 |
...
kubernetes.io/os: eve |
...
node.kubernetes.io/instance-type: eve |
...
annotations: |
...
eve.lfedge.org/node-type: virtual |
...
eve.lfedge.org/location: "texas/usa" |
...
eve.lfedge.org/activate: true |
...
name: eve-device.lab1 |
...
Namespace: enterprise1 |
...
spec: |
...
eve-os-version: 8.10.0-kvm-amd64 |
...
certificate: TFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVTTFla05EUVdNclowRjNTVUpCWjBsQ1FWUkJUa0puYTNGb2EybEhPWGN3UWtGUmMwWkJSRUZXVFZKTmQwVlJXVVJXVVZGRVJYZHdjbVJYU213S1kyMDFiR1JIVm5wTlFqUllSRlJKZVUxRWEzZE5WRUV6VGtSTmVrOVdiMWhFVkUxNVRVUm5lVTlVUVROT1JFMTZUMVp2ZDBaVVJWUk5Ra1ZIUVRGVlJRcEJlRTFMWVROV2FWcFlTblZhV0ZKc1kzcERRMEZUU1hkRVVWbEtTMjlhU1doMlkwNUJVVVZDUWxGQlJHZG5SVkJCUkVORFFWRnZRMmRuUlVKQlRDdG5DbWgxZVVzMEswZ3dSRGRXUVZWUUwwdEdWbFZRUkhRcllrTnhiMEZTYkRNeVlteEhaV0pxYlZJeGRuZEdZalJYYlhkSVdEWmhOVFZMU0hWbmNGb3lUbThLVVVjNVQzcEhkMjVNWmpKeGRGSlBZbkpvTHk5bFkxbFBaRmxDVjNWRWIySk1lbkYyYW05WllWZFlZVnB1TVhKVllYY3dSekZuYUhab1ZYUTVSbE56YlFwd1NHeHNXVzh3YzJOU1ZGRnhkMWRyUXpOaVprMXpkWFpYUkd0cGFsRlVjMnhzYTFBNVdXZENjMFJZYlZBNU4zaEdkR0pXT0ZodFVtOXpkREZhUTJOS0Nua3hiRUpSU0RGM1R6TlNNR2h4YkUxUGNGZG9ZVzlITUhObldqWldhR3g1YW5OVk0xZHFiWGhFVm1abE5tczNObmx0WmpCM1MxZGhVWHBoUjFGdlpGZ0tlQ3RQWTJSaWJHcDNlVXh4VWs1U2NWaFljRGRIT1VOWVlXRkVPWGxYT1cxTlZ6UkRiVFI2VDFweU1VZG5SMDF3YkZSMGFWVTBiWFJSY1dKUmNYTjFWQXBaVVVRekwwaEVTVGg0TmpSMVZWQk5UbXRGUTBGM1JVRkJZVTVEVFVWQmQwUm5XVVJXVWpCUVFWRklMMEpCVVVSQlowdHJUVUU0UjBFeFZXUkZkMFZDQ2k5M1VVWk5RVTFDUVdZNGQwaFJXVVJXVWpCUFFrSlpSVVpJVWt0SVdHRXZTRFpPUlZaUFIzUjJSV00xVkRWM1UybHVkMGhOUVRCSFExTnhSMU5KWWpNS1JGRkZRa04zVlVGQk5FbENRVkZDVDNKeWQzSllkbVo1U2xWUFIxaFZiMXB6ZEdoTFUxZ3ZhRGRtZVdGa01WSnNiV3RQYzBoMldtMDNORUpNZURGNVFncHhTREZMU1hjMlozQktWMUpHUkVKc01GUlBPWFYxWWtvemNHMXRaMWx1WW5wUFRqRXJVVFJPV1dscWNsYzVXVEpwTXk5bWNDdEpTWEZFV0VwTFRYZHpDbFpsYjNWeWFHZFRTVkExU0RaSFJVMHZUalJLVjBJeU4xVTFXVWswVlZKQlNEWjFkRU14Vm5sblZVOUNUMHN3Tm5wQmVXbEJabWROVERseE0wVkZXRFVLVlZwVmVUQnVSRUV2WVhGUFZqVkdZa3hMTmxWek1tWnRNREpXUW05SVZFNTRURXBWYlV4b2FFSTJWVkZQUkRKSlpGRTJiRWhhUXpNdk9HVk5PVTR3T0FvcmNWSklObXM1UjI5dGRHbElUM2R5WkVwd1dqTklVa3huYVdoV1ZrUjZhbVIxWkRoMFVWUnRhVGwyUTBsWVFXeFJiek5vZWxabVJUbERhVTVEVEVSRENtWkZia2xVWTA1Q2VXbEtRbU42TldaVVVuZFhWa1JKWlc5QlYzTkZabTVJY1ROSGVRb3RMUzB0TFVWT1JDQkRSVkpVU1VaSlEwRlVSUzB0TFMwdENnPT0= # base64 encoded |
...
serial: "6654abbcc44" |
...
onboard: TFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVTTFla05EUVdNclowRjNTVUpCWjBsQ1FWUkJUa0puYTNGb2EybEhPWGN3UWtGUmMwWkJSRUZXVFZKTmQwVlJXVVJXVVZGRVJYZHdjbVJYU213S1kyMDFiR1JIVm5wTlFqUllSRlJKZVUxRWEzZE5WRUV6VGtSTmVrOVdiMWhFVkUxNVRVUm5lVTlVUVROT1JFMTZUMVp2ZDBaVVJWUk5Ra1ZIUVRGVlJRcEJlRTFMWVROV2FWcFlTblZhV0ZKc1kzcERRMEZUU1hkRVVWbEtTMjlhU1doMlkwNUJVVVZDUWxGQlJHZG5SVkJCUkVORFFWRnZRMmRuUlVKQlRDdG5DbWgxZVVzMEswZ3dSRGRXUVZWUUwwdEdWbFZRUkhRcllrTnhiMEZTYkRNeVlteEhaV0pxYlZJeGRuZEdZalJYYlhkSVdEWmhOVFZMU0hWbmNGb3lUbThLVVVjNVQzcEhkMjVNWmpKeGRGSlBZbkpvTHk5bFkxbFBaRmxDVjNWRWIySk1lbkYyYW05WllWZFlZVnB1TVhKVllYY3dSekZuYUhab1ZYUTVSbE56YlFwd1NHeHNXVzh3YzJOU1ZGRnhkMWRyUXpOaVprMXpkWFpYUkd0cGFsRlVjMnhzYTFBNVdXZENjMFJZYlZBNU4zaEdkR0pXT0ZodFVtOXpkREZhUTJOS0Nua3hiRUpSU0RGM1R6TlNNR2h4YkUxUGNGZG9ZVzlITUhObldqWldhR3g1YW5OVk0xZHFiWGhFVm1abE5tczNObmx0WmpCM1MxZGhVWHBoUjFGdlpGZ0tlQ3RQWTJSaWJHcDNlVXh4VWs1U2NWaFljRGRIT1VOWVlXRkVPWGxYT1cxTlZ6UkRiVFI2VDFweU1VZG5SMDF3YkZSMGFWVTBiWFJSY1dKUmNYTjFWQXBaVVVRekwwaEVTVGg0TmpSMVZWQk5UbXRGUTBGM1JVRkJZVTVEVFVWQmQwUm5XVVJXVWpCUVFWRklMMEpCVVVSQlowdHJUVUU0UjBFeFZXUkZkMFZDQ2k5M1VVWk5RVTFDUVdZNGQwaFJXVVJXVWpCUFFrSlpSVVpJVWt0SVdHRXZTRFpPUlZaUFIzUjJSV00xVkRWM1UybHVkMGhOUVRCSFExTnhSMU5KWWpNS1JGRkZRa04zVlVGQk5FbENRVkZDVDNKeWQzSllkbVo1U2xWUFIxaFZiMXB6ZEdoTFUxZ3ZhRGRtZVdGa01WSnNiV3RQYzBoMldtMDNORUpNZURGNVFncHhTREZMU1hjMlozQktWMUpHUkVKc01GUlBPWFYxWWtvemNHMXRaMWx1WW5wUFRqRXJVVFJPV1dscWNsYzVXVEpwTXk5bWNDdEpTWEZFV0VwTFRYZHpDbFpsYjNWeWFHZFRTVkExU0RaSFJVMHZUalJLVjBJeU4xVTFXVWswVlZKQlNEWjFkRU14Vm5sblZVOUNUMHN3Tm5wQmVXbEJabWROVERseE0wVkZXRFVLVlZwVmVUQnVSRUV2WVhGUFZqVkdZa3hMTmxWek1tWnRNREpXUW05SVZFNTRURXBWYlV4b2FFSTJWVkZQUkRKSlpGRTJiRWhhUXpNdk9HVk5PVTR3T0FvcmNWSklObXM1UjI5dGRHbElUM2R5WkVwd1dqTklVa3huYVdoV1ZrUjZhbVIxWkRoMFVWUnRhVGwyUTBsWVFXeFJiek5vZWxabVJUbERhVTVEVEVSRENtWkZia2xVWTA1Q2VXbEtRbU42TldaVVVuZFhWa1JKWlc5QlYzTkZabTVJY1ROSGVRb3RMUzB0TFVWT1JDQkRSVkpVU1VaSlEwRlVSUzB0TFMwdENnPT0= # base64 encoded |
...
status: |
...
eve-os-version: 6.12.2 |
...
uuid: EC232B65-602A-F2A9-287B-5D95721116E6 |
...
addresses: |
...
- address: 172.19.0.3 |
...
type: InternalIP |
...
- address: k3d-k3s-default-server-0 |
...
type: Hostname |
...
allocatable: |
...
cpu: "4" |
...
ephemeral-storage: "296591664715" |
...
hugepages-1Gi: "0" |
...
hugepages-2Mi: "0" |
...
memory: |
...
16235544Ki pods: "110" |
...
capacity: |
...
cpu: "4" |
...
ephemeral-storage: |
...
304884524Ki hugepages-1Gi: "0" |
...
hugepages-2Mi: "0" |
...
memory: |
...
16235544Ki pods: "110" |
...
conditions: |
...
- lastHeartbeatTime: "2021-11-23T12:57:09Z" |
...
lastTransitionTime: "2021-10-10T10:33:38Z" |
...
message: kubelet is posting ready status |
...
reason: KubeletReady |
...
status: "True" |
...
type: Ready |
...
nodeInfo: |
...
architecture: amd64 |
...
bootID: dc703fd4-543b-4801-96be-4d6d29afb41e |
...
containerRuntimeVersion: containerd://1.4.9 |
...
kernelVersion: 5.10.1 |
...
machineID: "" |
...
operatingSystem: eve |
...
osImage: eve |
...
systemUUID: EC232B65-602A-F2A9-287B-5D95721116E6 |
A Device can be created in one of two ways:
...
Any device presenting this onboard certificate can self-register.
apiVersion: "eve.lfedge.org/v1beta1" |
Onboard CA
Any device presenting a certificate signed by this CA can self-register.
apiVersion: "eve.lfedge.org/v1beta1" |
Device CA
Any device presenting a device certificate signed by this CA can self-register.
apiVersion: "eve.lfedge.org/v1beta1" |
Networks
Node Network
Creating an EVE-style device network requires the usage of two CRDs, one for configuration information, which can be reused, and one for the on-device network itself.
Note that the CRD NetworkConfig
(below) is very similar in principle to the Kubernetes NetworkAttachmentDefinition.
Network configuration:
apiVersion: "eve.lfedge.org/v1beta1" |
Network instantiation:
apiVersion: "eve.lfedge.org/v1beta1" affinity: |
Workload Network
We leverage the cncf standard annotations on the workload to indicate desired networks on the actual workload.
annotations: |
Storage
The EVE semantics for storage are as follows.
...
- eve-blank: for a blank disk or mountpoint
- eve-quay: from container image on quay.io
- eve-docker: from container image on docker hub
- etc.
apiVersion: storage.k8s.io/v1 |
for blank:
apiVersion: storage.k8s.io/v1 |
Credentials secrets, if needed, are affiliated with the StorageClass as credentialsRef
.
We define Custom Resources for Image, and then use admissions controllers to validate that the requested resources exist when deploying a Pod that references them.
apiVersion: "eve.lfedge.org/v1beta1" type: user # can be any field; a controller may define special names; eve-os is reserved |
The Image name is then used in a PersistentVolumeClaim
. See below.
...
Golden filesystem image stored on FTP site, mounted as a filesystem. Defined using the StorageClass eve-ftp
.
apiVersion: v1 |
Golden VM image stored on FTP site, mounted as a block device. Defined using the StorageClass eve-ftp
.
apiVersion: v1 |
Blank disk volume.
kind: PersistentVolumeClaim |
Status
The state of an Application
, as reported by the controller, is set on the ApplicationStatus
. For example:
apiVersion: eve.lfedge.org/v1beta1 |
The ApplicationStatus
field is similar to the Kubernetes PodStatus, albeit not identical. The fields are as follows.
...
The states of the application are the ones currently supported by the EVE API. E.g. BOOTING
, RUNNING
, STARTED
.
Complete Example
apiVersion: eve.lfedge.org/v1beta1 |
Scheduling
We define higher-level scheduling constraints, specifically ApplicationDeployment
, ApplicationDaemonSet
, ApplicationStatefulSet
. These are optional; a controller MAY implement them, but is not required to do so.
...
In Kubernetes one normally does not create a node via the API; the node exists by virtue of its joining a cluster. However, it is possible to create one via the API. It is unclear how the node, upon joining, will reconcile with the existing node resource.
Kubernetes Node Certificate | EVE Device Certificate | |
Validation | Signed by valid CA | Actual certificate in controller |
Kubernetes Token | EVE Onboard Certificate | |
Validation | Shared secret | Actual certificate in controller |
Usage | Generate node certificate | Accept presented device certificate |
All additional features and properties of the node that are not directly related to the cluster itself, including taints and tolerations, are handled via metadata, specifically labels and annotations. Since these are semi-arbitrary key-value pairs, anything can be placed here.
...
In order to indicate that the image
field references an Image
to be referenced rather than a normal OCI image to be pulled from a registry, we set an annotation on the Pod:
annotations: |
These are identical to the CRD image solution, except that the annotation is necessary only when using native Kubernetes pod resources.
...