Motivation

An initial support for cellular connectivity was added to EVE OS already back in the version 4.3.0. Instead of using Hayes AT command set, which differs between modem manufactures, the decision was to use (proprietary) QMI and later also (standardized) MBIM protocols, providing standardized interface for controlling and managing various aspects of cellular modems, including functions like data transmission, voice calls, SMS messaging, and network configuration. These protocols can be used from open-source client libraries libqmi and libmbim, developed under the freedesktop.org project. However, there are no Golang bindings, hence we decided to use CLI tools provided alongside these libraries: qmicli and mbimcli. PoC for cellular connectivity in EVE simply consisted of a new container “wwan” with installed libqmi and libmbim libraries and their CLI binaries, plus a very simple shell script implementing a “management agent”. This script periodically (every 5 minutes) checked if modem (only one was supported at the time) is connected (using qmicli/mbimcli) and if not, it would read APN from a certain file under /run/wwan (where it was written down by NIM) and start connection with this APN string as the only parameter. The script then obtained IP settings and applied them to the wwan0 interface. List of DNS servers was published to NIM via another file stored under /run/wwan. No status data or metrics were published from this script.

...

1. The “management agent” for modems is way too complex for a shell script. What we have today abuses the power of shell for something that should be implemented using a proper programming language. We are using nesting, cycles, running commands and functions in the background, using pipes for exchanging data between processes and other features to the degree far beyond what the shell is meant to be used for. The outcome is a very inefficient implementation that starts and stops some processes all the time (e.g. to poll the modem state; for string manipulations; to produce json output; etc.).
2. The CLI tools qmicli and mbimcli are also not very efficient. Every call requires to start a new process, establish connection with the modem, request a so-called client ID, use that ID in the request, receive the response and print it to stdout, release client ID, close connection, stop process (and then we have to parse stdout). Furthermore, these tools often get stuck (maybe allocation and release of client IDs has some limits) and we have to run them inside “timeout” to avoid triggering watchdog. It is better to keep a single connection open and use the same client ID, but that’s only possible by directly using libqmi and libmbim libraries.And lastly, not all QMI/MBIM APIs have CLI commands provided.
3. CLI tools do not allow watching for modem notifications. This means that our shell script has to periodically poll for status updates (e.g. to react to connectivity loss). This adds delay and results in poor reaction time. It also makes the shell script even more inefficient because it keeps re-running the same set of commands quite often. With multiple modems, some command (i.e. process) is being run almost all the time.
4. As much as the shell script is complicated, it is not that sophisticated after all. For example, on any config change, everything is re-created (i.e. reconnected) for every modem. Handling every possible config change individually and effectively in this shell script is beyond imagination.
5. Because we use shell as opposed to Golang, input/output is done using files instead of pubsub. So wwan is rather disconnected from other microservices.
6. Our solution is apparently not very robust based on the number of customer issues received over time. This is partially because the QMI protocol is proprietary and specification is not publicly available. All we have is some very limited documentation provided for qmicli (more like comments). It is not always clear what is the right sequence of commands that we should call and what the parameters should be (some are not even documented). To a large degree we are therefore maintaining our script using a trial and error method.
7. Every modem model has its “nuances”, basically a “is-it-a-bug-or-is-it-a-feature” behavior added accidentally by manufacturers. It is therefore necessary to handle different models slightly differently, but that’s not realistic in our script.
8. At this point, I cannot imagine evolving this shell script further. For some advanced use-cases like 5G and beyond we will have to find a better way (such as what is being presented and evaluated in this document).

Introducing ModemManager

ModemManager (sometimes abbreviated to MM in this document) is a system daemon which controls WWAN (2G/3G/4G/5G) devices and connections. Alongside NetworkManager, ModemManager is the default mobile broadband management system in most standard GNU/Linux distributions (Debian, Fedora, Ubuntu, Arch Linux…), and is also available in custom systems built with e.g. buildroot, yocto/openembedded or ptxdist. ModemManager may also be used in routers running openwrt.

...

• tell ModemManager what it should do, e.g. connect modem at <DBus-path> with parameters <apn,user,password,...>
• obtain IP settings (IP address, gateway IP, DNS servers) and apply them in the network stack (i.e. this is done by NetworkManager, not MM, which as a result does not depend on the Linux network stack)
• obtain modem state data to then display via UI and nmcli (NetworkManager CLI tool)
• trigger reconnect (if enabled by user) when MM sends notification about a modem losing connection

EVE microservice controlling ModemManager

In order to evaluate the feasibility of using ModemManager inside EVE to control cellular modems, I ended up preparing pretty much complete integration inside my EVE fork. All features of mobile connectivity currently offered by EVE are covered by this integration and now implemented using ModemManager.

...

Full difference between the current EVE master and this MM integration can be seen here.

Evaluation

After testing ModemManager integrated with EVE on a device with two modems (Sierra Wireless EM7565 and QUECTEL EC21), I can now present the evaluation results, list the pros and cons of this solution, and share my thoughts on the way forward.

...

But apart from the watchdog, which is not an EVE problem, I was able to test all features of modem connectivity that EVE should support and all common scenarios without issues. This included enabling/disabling a modem, changing configuration, fail-over from eth to wwan connectivity, radio silence mode triggered from LPS, publishing location info, scanning visible providers, setting user credentials and more. A major improvement is the reaction time. With MM, our agent is getting notification as soon as modem state changes. We are therefore able to react to connectivity loss, modem discovery and other events very quickly. With our currently used shell script, we run polling with some period and this period is an extra delay that limits the reaction time.

Image Size

An important aspect that needs evaluation is the image size, which was expected to increase because we only added stuff to the wwan container and nothing substantial in size was removed (libqmi and libmbim had to remain). The only real advantage of the shell script is its small size. Having microservice written in Go adds several MBs into the image size as we are already aware of, having put most of our microservices under one zedbox binary. For the time being, I decided to have mmagent as a separate binary for cleaner separation, but nothing really prevents me from putting it under zedbox in the pillar container. To communicate with MM over DBus, only /run/dbus/system_bus_socket needs to be shared between the containers, which it already is.

...