NEW:
🎛 wwan: Replace uqmi with qmicli
This is due to a limitation of the QMI protocol and the linux driver qmi_wwan, which require that there is at most one client talking to the modem. For this reason, qmicli provides a proxy that multiplexes multiple requests under one session. But uqmi is not able to use this proxy, therefore we have to completely migrate from uqmi to qmicli.
🔓Send empty escrow key if TPM not enabled
To complete the attestation sequence we should send EncryptedVaultKeyFromDevice. Let’s leave data empty to indicate that we do not want to send it to the controller(if no TPM is enabled).
☑️ DPC verify: more sensitive handling of DNS errors
In this case, the DPC manager should wait instead of falling back to the previous DPC. But to detect this, Send* functions inside the cloud had to be improved to return an error value that allows unwrapping errors from all the send attempts.
🚦Add shutdown command to EdgeDevConfig
We added support to shutdown node gracefully to EdgeDevConfig.
DOCS:
🚦 Add shutdown/shutdown-poweroff commands to LPS API
We added DevInfo to the Local Profile API documentation. Publish the current state of the device to the local server and optionally obtain a command to execute.
🎛 Add document to list supported RAID configuration
ZFS provides a rich set of functionality but at a cost of extra resource usage. We added a table of RAID configurations for GRUB.
FIX:
🛠 Fix acl test to handle multiple IPs from dig
Dig command may return several IPs to properly handle them during the creation of the network we should join lines from output into one with a comma separator.
🛠 Fix accidental loop for poweroff command
This was introduced in PR #2609 and discovered as part of the review for PR #2610
🛠 Fix attestation restart on 403 config response
We should restart attestation on 403 code as config response and this code is assumed as an error, so we should move handling to the proper place inside error from SendOnAllIntf handling.
🛠 Do not restart a crashed VM within the handleModify function.
We added a check for `status.HasError()` in the condition in the `handleModify` function to eliminate the restart of crashed VM earlier than in `config.timer.reboot` seconds.
STATS:
Github:⭐️347(+3) DockerHub: 282591(+1585) pulls
Changelog: https://github.com/lf-edge/eve/compare/8.0.0...8.1.0
EVENTS:
The EVE Design Summit is taking place on June 23 in Berlin, Germany. Top EVE developers and contributors will collaborate with industry users and community members to plan the way forward. An open-source community keynote will be followed by lunch, then we’ll complete the remaining technical expert talks and break into groups to strategize for the future. We’ll wrap the day with an evening social event.
https://www.lfedge.org/event/eve-design-summit-2/
8.1.0 Released https://github.com/lf-edge/eve/releases/tag/8.1.0 🎉
NEW:
📍Geographic coordinates reported by EVE #2600
EVE is able to obtain location information from a GNSS receiver integrated into an LTE modem.
This information is then propagated to 3 destinations:
- to the controller as ZInfoMsg with newly added ZInfoLocation into InfoContent
- to the Local profile server
- to any locally deployed application via meta-data service
By default, location reporting is disabled and has to be explicitly enabled under the cellular configuration.
Learn more https://wiki.lfedge.org/display/EVE/GPS+coordinates+exposed+by+EVE
DOCS:
🎛 add edgeview container/api doc
Edge-View as a service on EVE, it needs to receive/update user configurations from the controller; and it needs to send/update Edge-View running status to the controller.
FIX:
⏱proto changes for app delay start interval && application instance staggered start built proto files
The crux of this feature is to introduce a delay between the time EVE is ready to process application instance configuration and the time the application is started. This delay interval should be added per application instance.
⬇️Remove /config/uuid compatibility for downgrade to older than 5.21
This includes removing some old unneeded code and relying on the 5-second timer when we have a UUID. Also, remove the /config/uuid compatibility for a downgrade to older than 5.21.
🛠Fix leaking locked mutex in netmonitor
watcher of LinuxNetworkMonitor would leave the mutex in a locked state in some cases, causing DPC Manager to deadlock.
📺Add missing iptables rule to allow local VNC traffic
STATS:
Github:⭐️344(+2) DockerHub: 282591(+861) pulls
Changelog: https://github.com/lf-edge/eve/compare/8.0.0...8.1.0
NEW:
DOCS:
FIX:
- Fix ntpd.pid watchdog after server change 🛠
We sometimes see a watchdog due to ntpd.pid not running in the case when EVE-OS picks up a NTP server from DHCP thus we see a log line of the form: NTP server changed from poll.ntp.org to $ns. The attempt to track this down has been to record the exit value from starting ntpd after it was killed, but that returns zero even when ntpd was not actually started. So adding a wait for the killed process to go away. - Fix filtering of unecessary info messages 🛠
Functions *HasRealChange() in zedagent that try to filter uninteresting changes from being logged have unintended side effects and cause zedagent to lose all timestamps and other information. This is because internally they clear these frequently-changing fields before comparing new and previous values, but do so without properly deep-copying values and thus changing the originals. - Fix multiple DNS servers configured for network instance 🛠
Multiple DNS servers (to advertise) should be configured for dnsmasq as a one-line "dns-server" DHCP option, with comma-separated DNS server IP addresses. Putting DNS servers each on a separate line is not correct, dnsmasq will only advertize the last entry. - Fix verification of persisted DPCs 🛠
With persisted DPCs from the previous run but with last-resort DPC disabled, there is only dpcTestTimer that will trigger DPC verification. And since this timer is set (by default) to 5 minutes, there is quite a delay until device applies working DPC after a reboot. This commit makes sure that persisted DPCs are tested as soon as possible after a reboot. DPC manager only waits for the global configuration before it starts verification. - Fix locally triggered purge 🛠
PR introduces a local generation counter for volume, which is added to the remote generation counter (from the controller) to form a volume key changing remotely as well as a locally issued purge. Similarly, the PR adds separate purge and restart counters for locally triggered operations to the application config. Note that most of the logic for local operations is currently handled by zedagent. Later, this could be refactored and moved to zedmanager to keep only config parsing and info/metrics publishing inside zedagent.
7.11.0 Released https://github.com/lf-edge/eve/releases/tag/7.11.0
NEW:
- NIM refactoring ✍️
This PR substantially refactors NIM microservice. Contains several commits to improve the code of NIM, split code into several files, and avoid files with 2000+ lines. - Support zfs raid levels during install 🎛
Provides grub parameters to explicitly install zfs and pick the raid level. Support for single disk installation of ZFS is added. - Show string status for zpool in case of not online ℹ️
Show string status for zpool in case of not online (i.e. One or more devices have been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.). The library we use has no support for showing this, so I parse the output of zpool status here. - Support for multiple top-level vdevs in pool 🛠
We will fill children info for multiple top-level vdevs and fill CurrentRaid as the lowest redundancy of all included vdevs. - Support to install EVE and ZFS on same disk 🎛
Installed with grub parameters eve_install_zfs_with_raid_level=none eve_install_disk=nvme0n1 - Pass grub config file for iso installer 💿
Seems we still need to have the possibility to pass options from grub.cfg comes with config.img file in case of iso installer.
The solution to sort disks (#2309) discussed #2303 (comment) is not enough and we still need to define explicitly the disk to install EVE onto with eve_install_disk option
FIX:
- Use fixed device/partition UUIDs with ZFS 🛠
For unknown reasons, ZFS creates partition 1 and partition 9 when adding to the pool. As we later add to the pool, we want to ensure that otherwise identical hardware ends up with identical PCR5 measurements from the TPM. - potentialUUIDUpdate multiple access fix 🛠
Check if zedclient process is not running before run new one. Add mutex to handle multiple function calls from goroutines - Need longer wait for zboot reset 🛠
On a RPI4 running ZFS with lots of USB sticks as drives and one of them in a failed/hosted state the zboot reset (and other zboot commands) takes more than 20 seconds.
STATS:
Github:⭐️340(+1) stars 121(+3) forks DockerHub: 280472(+1351)pulls🚀
Changelog: https://github.com/lf-edge/eve/compare/7.11.0...8.0.0
7.10.0 Released https://github.com/lf-edge/eve/releases/tag/7.10.0
NEW:
- Extract raw filename from mime
This is necessary to handle new version of the mime package and allow creating the CDROM directory layout for cloud-init. Note that the handling guards against the directory/filename escaping from the target directory. - Use 32byte TPM keys only for vault protection 🔒
Starting this commit a new install of EVE-OS will create a vault config file on systems with TPM support. That file will be used to determine whether to use only the TPM key or merge the TPM and controller key. This applies to both ext4 and zfs filesystems.
- Add functionality to send information about disks ℹ️
Add functionality to send information about disks via a separate HardwareInfo message with a rare sending rate. Add serial number assembly for disks in ZFS. Add information about the disk from which information could not be retrieved. Also rewrote the GetSerialNumberForDisk function because an error occurred if the input disk name was a partition (eg /dev/sda1).
- Run tests against zfs-kvm 🛠
We can use zfs-KVM HV to run the single-disk zfs mode of EVE and use it in our tests. We changed the version of EVE in eveupdate tests to recent ones.
- Add S.M.A.R.T data collector for disks 🎛
This PR adds features for collecting disk information, including SMART attributes. Also, a package has been added here that allows you to read the file system, to obtain information about available disks and information about them.
- API update to send more disks information for storage system to EVE ℹ️
Update API to send more disk information (including s.m.a.r.t ones) for the storage system in EVE.
- Add possibility to define nested structures in DisksConfig 🎛
To use stripe of two pairs of mirrored disks we should define DisksConfig without disks with array_type DISKS_ARRAY_TYPE_RAID0 with two children with properly defined disks inside and with array_type DISKS_ARRAY_TYPE_RAID1 and empty children
- Set max_sectors explicitly to run Windows VM with vhost-scsi-pci 🪟
We can see [ 259.573575] vhost_scsi_calc_sgls: requested sgl_count: 2649 exceeds pre-allocated max_sgls: 2048 in kernel messages and Windows VM do not boot with zfs/vhost-scsi-pci. As discussed in https://edk2.groups.io/g/discuss/topic/windows_2019_vm_fails_to_boot/74465994: I/O size exceeds the max SCSI I/O limitation(8M) of vhost-scsi in KVM and we should adjust options to run Windows VM with vhost-scsi-pci.
- Split bucket and path from ds config for AWS ☁️
We can have files located in directories inside the bucket, but now path from datastore assuming as bucket name. We should split the path into bucket names and file paths if we can see '/' inside the bucket.
- Have installer default to fixed disk/partition UUIDs 📝
This is needed to make PCR5 in the TPM measured boot be the same for otherwise identical hardware and firmware/software. The new eve_install_random_disk_uuids can be set to get the old behavior. storage-init recreates as fixed if IMGA has the fixed UUID.
STATS:
Github:⭐️339 stars 119(+1) forks DockerHub: 279121(+ 1606) pulls🚀
Changelog: https://github.com/lf-edge/eve/compare/7.9.0...7.10.0
7.9.0 Released https://github.com/lf-edge/eve/releases/tag/7.9.0
NEW:
FIX:
STATS:
Github:⭐️339(+2) stars 118 forks DockerHub: 277515(+1190) pulls🚀
Changelog: https://github.com/lf-edge/eve/compare/7.8.0...7.9.0
7.8.0 Released https://github.com/lf-edge/eve/releases/tag/7.9.0
NEW:
- Run potentialUUIDUpdate on 400 and on attest problems ☑️
As described in the APIv2 documentation, we should assume that the device does not exist in the controller if the controller returns 400.
Also seems we do not run potentialUUIDUpdate before successful attestation, but we should. Also, we must remove the old attest message on change and push new. - CONFIG_IGC for Intel Ethernet Controller 🎛
CONFIG_IGC for Intel Ethernet Controller I225-LM/I225-V/I225-IT - Use TLS with S3 🔑
Some old code had this disabled, thus we relied on the image SHA256 for verification. However, this means that firewalls need to open up outbound port 80 when port 443 should be sufficient. Verified that the S3 downloads work correctly even when a TLS MiTM proxy is in use thus the proxy certificate is passed into the S3 download code. - Implement appinfo extension for purge/restart command requests ℹ️
This commit implements the extension to the /api/v1/appinfo local profile endpoint, which allows the server to submit purge/restart commands for locally running application instances. This functionality is already documented in api/PROFILE.md under "AppInfo". Plus test lf-edge/eden#744 - Rework ECO to show information to log and VNC 📺
Now we cannot see information from the app in logs if VNC is enabled, with this change we will output information to both places. - Allow /30 subnets for local network instance 📡
The current MinSubnetSize of > 8 is too restrictive. We these changes we can handle /30 subnet, which means that there is one IP address available for an app instance (and one for "zedrouter").
FIX:
- Wait for no qmp socket in cleanup for kvm ⌛️
We can see in EdenGCP tests for KVM errors like PURGING: [description:"Qmp not found: error dial unix /run/hypervisor/kvm/ae84d4b7-9634-4abd-83ef-56f04ec01a27.1.5/qmp: connect: no such file or directory"...]. Errors occur after retry in our experiments. We should wait for the qmp socket gone after killing of containerd task. - Fix unlock vault on controller key for zfs 🔒
We should unlock zfs vault in a different way on the controller key received
STATS:
Github:⭐️337 stars 118 forks DockerHub: 276325 pulls🚀
Changelog: https://github.com/lf-edge/eve/compare/7.7.0...7.8.0
7.7.0 Released https://github.com/lf-edge/eve/releases/tag/7.7.0
NEW:
- add RoL Rpi4 test steps to eden GCP workflow 🎉
Now the first implementation of the cli and rest api of the Rack of Labs system is ready. But it is in the stage of active development and testing, and at this stage, we think it is inappropriate to include it as part of EDEN. RoL was added to the Github Actions together with the Google Cloud Platform. - Add debug.enable.vga support 📺
We only return VGA devices that were marked as boot devices.Console output won't be visible on others anyway. It also allows us to debug issues using VGA console while other GPUs are still assigned to applications - Implement ZFS minimum requirement check ☑️
Minimum supported system requirements to install ZFS storage is 64GB memory and 3 physical disks set in eve_persist_disk. eve_install_skip_zfs_checks should be set in the installation config to override the requirement check for experimental installs. - Use context with timeout in sending to the cloud functions ⏱
We can possibly consume a long time when we try to send information to the cloud with long delays/a lot of interfaces. We add context with a timeout to not hit watchdog. - zfs: performance patches 🏎
Patches significantly reduce the write amplification, therefore less data needs to be written, and fewer unnecessary syncs are issued. Another feature introduced in the patches is the write bandwidth smoothing algorithm which prevents huge latency spikes under the heavy load on the storage.
FIX:
- Rework zfs_arc_max get logic 🛠
Zedmanager asks volummgr to create the new volume, and when that is done it halts the application and restarts it. - Fix dynamic IPs allocation for several interfaces in the same network 🛠
If we allocate several interfaces on the same network instance all of them receive the same IP. It comes because we save and check only networkInstance - app pairs, so, we can allocate only one IP. We moved UUIDPairToNum to UUIDPairAndIfIdxToNumKey and allow to allocate different appNum (and IPs) for different interfaces on the same networkInstance.
Changelog: https://github.com/lf-edge/eve/compare/7.6.0...7.7.0
EVE 7.6.0 https://github.com/lf-edge/eve/releases/tag/7.6.0
NEW:
- limit dhcp range with BitMapMax
We cannot allocate more than BitMapMax dynamic IPs so seems we can reduce the capacity of dhcp range down to it. In that case for default network instance (/16) we will not hit the problem with allocated static IPs from the end (and inside) of dhcp range comes from the controller. - kernel: make patches am-able
Only a few of our patches can be applied using `git am`. This commit turns all the patches into a proper format. This makes it a bit easy to kick-start new kernel activity. - appinfo for local profile server implementation
Implementation for sending compact info (uuid, name, state and error) about apps on EdgeNode to api/v1/appinfo endpoint on local profile server. - Add ipxe.efi to release artifacts
ipxe.efi will be published with ipxe.efi.cfg and ipxe.efi.ip.cfg
FIX:
- fix broken calculation of downloaded parts for azure ☁️
In case of successfully downloading the last part (it is a special one, which is in general not equal SingleMB), on retry we can hit. The range specified is invalid for the current size of the resource because of the wrong check for the last partNum (Developer mistakenly checked not for partNum == partsCount - 1 in #2420) and we hit the situation with downloading 0 bytes starting from the end of the file. In this PR I removed complex logic and just check if the range for download is greater than 0 or not. - fix make-raw to properly handle stdin and to not adjust partitions for usbconf 🧹
It comes from lines where we check for /parts and if it does not exist, we extract file comes from tar in stdin. But /parts exist, we create it in Dockerfile. So, the developer adds a check for tty connected to stdin, if not, we assume that it is a pipe from tar. - Remove unused package 🧹
There is a CVE flagged against a dependency in pkg/lisp and since we no longer use it the easiest resolution is to remove the code.
Full Changelog: https://github.com/lf-edge/eve/compare/7.5.0...7.6.0
EVE 7.5.0 https://github.com/lf-edge/eve/releases/tag/7.5.0
NEW:
- Reserve space to maintain performance in ZFS 💾
It is recommended that storage usage should not go above 80% of available space. Because pool performance can degrade when a pool is very full and file systems are updated frequently, such as on a busy mail server. - parse-pkgs generates all known; docker-compose standardized 🧹
The primary benefit of this change is that it lets the compose file be standard. This makes it easier for editors to see and parse, people onboarding to find. It also allows a normal docker-compose up to simply execute it, while complaining that the env vars were not set. In short, we take advantage of standard tooling without changing functionality. - sort disks in mount_disk.sh 🗂
In case of multiple disks, we will have unordered output from find and can have swapped mounts.
as we expect this order from VM config (we rely on the same order between lines in mountPoints file and block device enumeration.) - pkg: grub: arm: moving to 2.06 ⬆️
Use grub tag instead of commit for arm64 and riscv64. Grub verify module is no longer available for build under arm with grub version 2.06. Grub verify module is no longer available for build under arm with grub version 2.06. Now coreutils package is used for arm too. - pkg: kernel: moving to 5.10.76 version ⬆️
As a result of testing, updating the kernel to 5.10.76 helps to eliminate the error -Synchronous exception at 0x000000005EAED180 on the RPi4 with hypervizor kvm. Previously, the error occurred when starting, restarting, and shutting down EVE App instances with a 5% chance.
FIX:
- block sending stacks in metrics message 🛠
there was a bug that caused the process stack collection to be always null, which was fixed in ec37884. but that fix caused the metrics message size to be more than doubled, e.g. on 'zc1' from 470k/hour before to 1.23M/hour after that fix, about 20Mbytes/day increase for that device. This patch is to skip uploading the process stack in metrics message to bring down the metrics message size to pre 6.12 release. - Euresys Frame Grubber Full XR doesn't work under VM 🛠
The PCIe card works fine on bare metal but doesn't work under VM. Neither Linux nor Windows. - Fix wrong permission with initrd.img 🛠
docker run lfedge/eve:tag installer_net to create network installer the output file initrd.img gets 600 instead of 644
DOCS:
Full Changelog: https://github.com/lf-edge/eve/compare/7.4.0...7.5.0
EVE 7.4.0 https://github.com/lf-edge/eve/releases/tag/7.4.0
NEW:
- Do not allow to impose radio silence during EVE update testing
If edge node is going through EVE update and radio silence is imposed during the 10 minutes testing period, then the access to the controller may be lost and device will fallback to the previous release. This is in violation with the radio silence requirements, which state that edge node should not trigger port config or EVE image fallback during a (temporarily) imposed radio silence. To prevent the EVE fallback from happening, zedagent will simply return error back to the Local profile server if radio silence is requested during EVE update testing period. - azure partial download
This allow us to keep information about downloaded parts for s3 datastore in memory and resume download from the previous try. New updates added Azure support. Also added file with .progress in the end to keep information about downloaded parts across reboots.
FIX:
- configure network broadcast address on container interfaces
Network bcast address on container interfaces is currently not being set. It shows as 0.0.0.0 in ifconfig output from inside the container.
This fix should make sure that ip command while setting the ucast address also computes the bcast address and adds it to interface of containers. - Fix publishedEdgeNodeCerts set too early.
Even in case messages are deferred due to failures we should not set publishedEdgeNodeCerts until after the ZAttestReqType_ATTEST_REQ_CERT message has been sent. - fix an issue of tlsconfig initialization in diag.go
this crash was due to a change in PR #2333 that added at the diag start of init tlsConfig to the session resume, but not the caroot.
this is to remove that init, and during the tryPing time to get the tlsConfig normally and add the session-resume option
DOCS:
Full Changelog : https://github.com/lf-edge/eve/compare/7.3.0...7.4.0
EVE 6.10.0 https://github.com/lf-edge/eve/releases/tag/6.10.0
NEW:
- retry eve image update
Possibility to retry eve image update by incrementing the counter (EdgeDevConfig->Baseos->RetryUpdate→Counter). If the currently configured image is in FAILED state in the other partition, retry the image upgrade. ELSE Do nothing. Just update the BaseosUpdateCounter counter in the Info message and send an Info message to zedcloud. Warning! if you try this from the UI and the device is running <6.10 it will silently be a no-op.
- generated vendor files for PR #2163
- Add display print example plus + qemu test
- Refactor ledmanager to handle different display
- Avoid setting freeUplink in model; better handling of wwan and wlan
- run selfRegister to overcome node re-creation
- support zfs zvols as volumes
- mdev rules for zvol
- add zfs-udev into eve-alpine
- bump edge-containers version
- datastore https with certificate chain
- change keyserver for eve-vtpm
FIX:
Full Changelog : https://github.com/lf-edge/eve/compare/6.9.0...6.10.0