...

Currently, EVE OS is able to manage a single QMI or MBIM enabled LTE modem. There is a separate wwan service, running a shell script which periodically checks the wwan connectivity and tries to (re)start and (re)connect the modem using uqmi and mbimcli tools. wwan service communicates with zedbox using files written under /run directory. Specifically, nim publishes the user configured APN into /run/accesspoint/wwan0 and the wwan service publishes state data and metrics as several json files under /run/wwan. These are periodically read and included into device info/metrics by zedagent.

Currently, the wwan service is limited to one LTE network with one APN.

The wwan service behaviour can be described using the following pseudo-code:

...

Firstly, given that the feature should be managed locally, we propose to leverage a recently added Local Profile server. The idea was to essentially allow a locally deployed application to act as a local controller and override a (very small) subset of the device configuration received from the controller. Currently, only the “profile” field can be overridden, hence the name. However, the feature was designed knowing that it could be extended in the future with more device configuration options that need local override. The API of the Local Profile server defines endpoint /api/v1/local_profile from which EVE should periodically obtain a profile name and use it instead of the default profile selected by the controller. The endpoint content is a protobuf-serialized message, which makes it easily extensible and allows to add more fields for local override. However, for airplane mode functionality we will define a separate POST endpoint, making it easier to remove/disable should the need for this feature go away. For API details see “EVE API Additions” below

The behaviour behavior (and to some degree the implementation) of this feature would be inherited from the Local Profile Server.

...

• The application to act as a Local Profile Server is selected via the controller
• Local profile server has to authenticate itself to EVE OS using a token
• Configuration from Local Profile server overrides the configuration from the controller (even when the Local server is not currently running/accessible/responsive). In this case, the airplane mode is not actually configurable by the controller. The behaviour behavior is to act as if the feature was disabled (i.e. airplane mode is false) by the controller. This means that only Local profile server can enable it and by default (i.e without local profile server app being deployed) the feature is disabled (i.e. radio is on).
• The configuration from the Local Profile Server is preserved between system reboots using the /persist directory partition

The requested airplane mode state would be periodically obtained by zedagent from the Local Profile server. The handler of this configuration option would be the wwan service. The file-based communication between the zedagent/nim and wwan service will be used to exchange the intended/actual airplane mode state. This is described in more details below under “EVE API Additions”.

...

The actual radio on/off operation will be done by wwan service using QMI / MBIM protocol as opposed to modem-specific AT commands. As it has been already mentioned, there are CLI tools available for both protocols that allow to change and read the radio power state. For now we therefore do not have to have a solution of preserving the airplane mode state immediately after booting, instead we should focus on avoiding a device reboot in the first place.

Out of Scope

For the time being it is not required by the customer to apply the airplane mode for WiFi modemsadapters. This proposal therefore covers radio on/off switch for cellular modems only.

...

Even though this document discusses the challenges associated with a device reboot and what effect it may have on the airplane mode, for this particular customer it is actually unacceptable for a device to reboot during the airplane mode (due to the nature of their use-case). For now we therefore do not have to have a solution of preserving the airplane mode state immediately after booting, instead we should focus on avoiding a device reboot in the first place.

EVE API Additions

Local Profile Server (new endpoint)

...

message RadioStatus {
	// true if enabled AND successfully applied
	bool airplane_mode = 1;
	// for every LTE network
	repeated CellularStatus cellular_status = 1;
	// later we can add status for every WiFi network
}

message CellularStatus {
	string networkUUID = 1;
    CellularModem modem = 2;
	OperatingState operatingState = 3;
	string msisdn = 4;
	string imei = 5;
	// if the last configuration change failed, error message is reported here)
	string error = 10;

}

message CellularModem {
	string model = 1;
	string revision = 2;
}  

enum CellularOperatingState {
	OPERATING_STATE_UNSPECIFIED = 0;
	OPERATING_STATE_ONLINE = 1;
	OPERATING_STATE_RADIO_OFF = 2;  (AKA airplane mode)
	OPERATING_STATE_OFFLINE = 3;
	...
}

...

A new /run/wwan/<network-uuid>/config.json will be added for each LTE network to submit LTE configuration from nim to wwan service, including the required airplane mode state, e.g.:

...

One thing that we still need to figure out is how to appropriately reference an LTE modem in NetworkConfig/WirelessConfig, i.e. how to go from some human readable (and persistent) “modem ID” to the corresponding CDC device path and the interface name. Until then nim will publish hard-coded wwan0 and cdc-wdm0 values. Also, the probing IP (by which wwan service determines if the connection is working) will be hard-coded to google DNS IP but eventually it will be made configurable (i.e. making all these configurable is out-of-scope for this proposal).

...

{
	"protocol": "qmi" | "mbim",
	"operating-mode": "online" | "radio-off" | etc.
	"msisdn": “918369110173”,
	"imsi": “310170845466094”,
	"model": "QUECTEL Mobile Broadband Module",
	"revision": "EC21ECGAR06A04M1G",
	"error": "",
	// modification Unix timestamp of config.json applied at the time of publishing this info
	"config-mtime": 1629446126
}

...

• Application provides UI on the user-side and a HTTP server with /api/v1/radio on the EVE-side
• The UI should allow to:
• Toggle the airplane-mode intended state (ON/OFF) and as a result change the content that the HTTP server will respond with to /api/v1/radio POST calls.
• Show the last received actual state and the last error if there was any (periodically POSTed to /api/v1/radio by EVE)
• Show that an operation of changing the radio state is still ongoing (e.g. a “loading gif”). This starts from a moment of user changing the intended state in the UI, continues through the next POST API call (from which EVE learns the new intended state, which in this case would differ from the actual state), up to the second next POST API call with the actual radio state after making the attempt to apply the new intended config . (please remember that for simplicity sake there will be no POST API calls while a change is ongoing).

zedbox Implementation Changes

...

type ZedAgentStatus struct {
	...
	AirplaneMode AirplaneMode   
}

AirplaneMode struct {
	Enabled     bool // from application
	InProgress  bool // from NIM
	RequestedAt time.Time // recorded by zedagent after POST call
}

nim will subscribe for zedagent status updates. If AirplaneMode.Enabled has changed, nim will trigger the operations of switching all radios ON/OFF. This actually means to publish the new configuration into wwan (and later also wlan) service and (asynchronously) wait for updated state data (acknowledging the latest config using a timestamp). Once wwan service responds, nim will publish the new state back into zedagent using DeviceNetworkStatus, which will be also extended with the same AirplaneMode structure (.RequestedAt copied from ZedAgentStatus; .Enabled and .InProgress updated by nim). Inside, the per-port NetworkPortStatus will contain WirelessStatus. Zedagentzedagent will make another POST call to /api/v1/radio only after it has received DeviceNetworkStatus with AirplaneMode of the same timestamp and with InProgress being false.

...

Please note that there is also a timer inside EVE to reboot the device if it hasn’t had controller connectivity for an entire week. Since it is not expected that airplane mode will be turned ON for such an extended period of time in practice, this behaviour behavior will not change.

wwan Service Implementation Changes

The shell script of the wwan service will undergo few implementation changes. Firstly, it will stop using hard-coded configuration values and instead will wait for and read /run/wwan/<network-uuid>/config.json (written by nim for each LTE network). It will then keep monitoring config.json files and will apply any changes, including the airplane-mode on/off switch.

A secondary goal of these implementation changes is to prepare (either fully or at least partially) the wwan service for scenarios with multiple LTE modems and multiple APNs.

The new wwan service behaviour behavior can be described using the following pseudo-code:

wwan main:
    repeat indefinitely:
        for every LTE network:
            if /run/wwan/<network-uuid>/config.json changed/appeared:
                detect protocol (QMI or MBIM)
                reset modem (i.e. disconnect)
                reload(re)load config
                set operating mode (online or low-power)    
            if not in the airplane mode (operating mode is "online") and connectivity check failed:
                reset modem if connected
                wait for device registration on the network
                start network(s) (take APN(s) from config.json)
                wait for data connectivity and IP address
                configure interface and resolv.conf
            publish state data under /run/wwan/<network-uuid>/ (power state, signal strength, IP configuration, etc.)
        sleep 5mins, break from sleep if any /run/wwan/<network-uuid>/config.json changes/(dis)appears

...

At the same time the application could publish the state information obtained from EVE into another file with a hardcoded hard-coded path. With this behaviour behavior it will be easy to prepare test-scripts under eden.

More challenging for the automated testing is the LTE modem side. Unless we can somehow emulate a cellular modem, we will need a physical device with a modem and preferably also with a registered SIM card available in the a lab.

Issues and Risks

The main issue of this proposal is that the operation of switching the airplane mode on/off is handled asynchronously with a considerable delay. Whereas longer delay may be acceptable when edge device is managed from the cloud, there may be different user expectations for local management. Also, there is a risk that the user may impatiently start to toggle the airplane mode button, triggering configuration changes before the previous one(s) had been fully handled and potentially hitting some race condition scenarios.

In order to limit the latency and the risk of race conditions, we proposed a separate and a shorter time period between EVE calls to radio POST API. Additionally, EVE will not make radio POST calls while a change is ongoing, thus enforcing at most one configuration change to be in progress at a time. Conversely, when an airplane mode change is finalized, EVE may call radio POST API with radio state update immediately, instead of waiting for the timer to fire. Internally, we will also try to minimize delay in communication between microservices (zedagent, nim, wwan).

Another issue is that when an external modem (e.g. USB dongle) gets physically re-connected, or if the entire device reboots, the operational state of the radio resets back to online. It was at least the case with the LTE modem tested for this proposal. Between modem resetting or edge device booting and wwan service noticing that the state is not as intended, there is a delay during which the device might emit power over radio, which is a safety risk. In Linux, the rfkill subsystem provides a kernel command line option to disable/enable radio on boot automatically. We may need to implement something similar for EVE. Perhaps the radio could be always disabled on boot, with only wwan service having the control to enable it once the configuration is received with airplane mode disabledturned off. But please note that device reboots during the airplane mode are not acceptable for this particular customer, hence the priority is to keep the device running.

One last issue, or rather an open question, is whether we can actually combine the Local Profile server with the airplane mode or if there is a customer requirement to have them separated and managed from different applications. This remains to be communicated with the customer. However, by having /api/v1/radio and /api/v1/local_profile as separate endpoints, it should be possible and relatively easy to allow them to be handled by different applications.