Versions Compared

Old Version 3

changes.mady.by.user Milan Lenco

Saved on

compared with

New Version 4

changes.mady.by.user Milan Lenco

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

An initial support for cellular connectivity was added to EVE OS already back in the version 4.3.0. Instead of using Hayes AT command set, which differs between modem manufactures, the decision was to use (proprietary) QMI and later also (standardized) MBIM protocols, providing standardized a unified interface for controlling and managing various aspects of cellular modems, including functions like data transmission, voice calls, SMS messaging, and network configuration. These protocols can be used from open-source client libraries libqmi and libmbim, developed under the freedesktop.org project. However, there are no Golang bindings, hence we decided to use CLI tools provided alongside these libraries: qmicli and mbimcli. PoC for cellular connectivity in EVE simply consisted of a new container “wwan” with installed libqmi and libmbim libraries and their CLI binaries, plus a very simple shell script implementing a “management agent”. This script periodically (every 5 minutes) checked if modem (only one was supported at the time) is connected (using qmicli/mbimcli) and if not, it would read APN from a certain file under /run/wwan (where it was written down by NIM) and start connection with this APN string as the only parameter. The script then obtained IP settings and applied them to the wwan0 interface. List of DNS servers was published to NIM via another file stored under /run/wwan. No status data or metrics were published from this script.

...

  1. The “management agent” for modems is way too complex for a shell script. What we have today abuses the power of shell for something that should be implemented using a proper programming language. We are using nesting, cycles, running commands and functions in the background, using pipes for exchanging data between processes and other features to the degree far beyond what the shell is meant to be used for. The outcome is a very inefficient implementation that starts and stops some processes all the time (e.g. to poll the modem state; for string manipulations; to produce json output; etc.).
  2. The CLI tools qmicli and mbimcli are also not very efficient. Every call requires to start a new process, establish connection with the modem, request a so-called client ID, use that ID in the request, receive the response and print it to stdout, release client ID, close connection, stop process (and then we have to parse stdout). Furthermore, these tools often get stuck (maybe allocation and release of client IDs has some limits) and we have to run them inside “timeout” to avoid triggering watchdog. It is better to keep a single connection open and use the same client ID, but that’s only possible by directly using libqmi and libmbim libraries. And lastly, not all QMI/MBIM APIs API methods have CLI commands provided.
  3. CLI tools do not allow watching for modem notifications. This means that our shell script has to periodically poll for status updates (e.g. to react to connectivity loss). This adds delay and results in poor reaction time. It also makes the shell script even more inefficient because it keeps re-running the same set of commands quite often. With multiple modems, some command (i.e. process) is being run almost all the time.
  4. As much as the shell script is complicated, it is not that sophisticated after all. For example, on any config change, everything is re-created (i.e. reconnected) for every modem. Handling every possible config change individually and effectively in this shell script is beyond imagination.
  5. Because we use shell as opposed to Golang, input/output is done using files instead of pubsub. So wwan is rather disconnected from other EVE microservices.
  6. Our solution is apparently not very robust based on the number of customer issues received over time. This is partially because the QMI protocol is proprietary and specification is not publicly available. All we have is some very limited documentation provided for qmicli (more like comments). It is not always clear what is the right sequence of commands that we should call and what the parameters should be (some are not even documented). To a large degree we are therefore maintaining our script using a trial and error method.
  7. Every modem model has its “nuances”, basically a “is-it-a-bug-or-is-it-a-feature” behavior added accidentally by manufacturers. It is therefore necessary to handle different models slightly differently, but that’s not realistic in our script.
  8. At this point, I cannot imagine evolving this shell script further. For some advanced use-cases like 5G and beyond we will have to find a better way (such as what is being presented and evaluated in this document).

...

MM provides a standardized and consistent interface for interacting with different types of modems, such as USB dongles, embedded cellular modules or RS232 modems, enabling seamless integration of these devices into the Linux ecosystem. MM uses plugin architecture, where each plugin is a dynamically loaded library implementing a MMPlugin interface for a class of modems. For example, there is libmm-plugin-sierra.so implementing support for modems from Sierra Wireless. As of this writing, there are 48 plugins in total, covering all relevant modem manufacturers out there.

ModemManager is an actively developed project with contributions from a diverse group of developers and maintainers, the main one being Aleksander Morgado. A wide range of companies from different sectors are using and frequently contributing to this project, including: Google (Chromium), Ericsson, T-Mobile US (used here), Deutsche telekom, RedHat, Canonical, Samsung, Quectel, Huawei and many less known companies. The project is hosted on freedesktop.org and licensed under GNU LGPLv2.1. It is written in C, using glib and gio.

Here are some useful links related to ModemManager:

...

In order to evaluate the feasibility of using ModemManager inside EVE to control cellular modems, I ended up preparing pretty much a complete integration inside my EVE fork. All features of mobile connectivity currently offered by EVE are covered by this integration and now implemented using ModemManager.

The only notable limitation is that between Static, DHCP and PPP bearer IP methods, only Static is supported. But this is also the case with our current solution based on the shell script. With modern modems implementing QMI/MBIM protocols, the PPP method, which merely emulates a legacy analog modem, is no longer recommended and rarely used these days due to its performance and other limitations. It is therefore up to our consideration if supporting PPP for cellular connectivity in EVE is worth the extra effort and additional dependencies (pppd or some implementation in Golang). And as for DHCP, we haven’t yet received any request from EVE users to support DHCP client running on the wwan interface. It seems that at least the modems supported and verified on EVE always pass full IP configuration to the host, thus it is not necessary to run a DHCP client on the host. However, since we already support DHCP for ethernet interfaces, this could be implemented with little effort and no extra dependencies (NIM would be told by the wwan microservice to start dhcpcd for wwan* interface as well).

...

Of course, there are significant differences between NetworkManager and mmagent that stems from EVE not being a standard Linux distribution. The main difference being is that mmagent receives configuration from a remote controller. It comes in the form of WwanConfig pubsub publication from NIM microservice, which builds it from DevicePortConfig that zedagent created based on a portion of EdgeDevConfig.

...

An important aspect that needs evaluation is the image size, which was expected to increase because we only added stuff to the wwan container and nothing substantial in size was removed (libqmi and libmbim had to remain). The only real advantage of the shell script is its small size. Having microservice written in Go adds several MBs into the image size as we are already aware of, having put most of our microservices under one zedbox binary. For the time being, I decided to have mmagent as a separate binary for cleaner separation, but nothing really prevents me from putting it under zedbox in the pillar container. To communicate with MM over DBus, only /run/dbus/system_bus_socket needs to be shared between the containers, which it already is.

...

  • More efficient implementation: C and Go code as opposed to shell; notifications used as opposed to polling; MM reusing QMI/MBIM connection with a modem; communication with MM is in binary format as opposed to parsing string output from CLI tools; config changes handled efficiently as opposed to recreating everything
  • Better stability:This still needs more testing across a wider range of modems, devices and cellular networks to confirm, but my initial experience suggests that ModemManager will provide a more stable solution with less customer issues. After all, customers often complain that there is a problem with mobile connectivity on EVE while the same works on Ubuntu (where MM is running). During my testing I didn’t experience any issues and MM was even able to restart and recover my modem when it got stuck and a kernel watchdog was triggered.
  • Better reaction time: MM uses so called QMI/MBIM indications to get state updates from modems and mmagent uses DBus signals to watch for changes published by MM. This allows us to avoid polling and instead get notified when something changes. mmagent can therefore react almost immediately to lost connectivity and other events.
  • Better expertise: MM community has deeper understanding and more experience working with cellular modems than we do. Furthermore, they have access to QMI /MBIM specs spec and possibly other material from modem manufacturers that is not publicly available. Using MM we will be in position to post questions on the mailing lists, create tickets in GitLab, etc. We cannot expect them to help us with our current custom solution.
  • Wider modem support: With MM, it should be easier to bring in new modems and use them with EVE. It already provides 48 plugins covering all relevant modem manufacturers. It also supports modems connected via serial ports (as opposed to USB) and also some that do not understand QMI/MBIM protocols (in that case AT commands are typically used instead).
  • Integration with pillar/pubsub: mmagent uses pubsub and other common packages of pillar, such as logging and agentbase, for a native integration with pillar microservices. Meanwhile, the shell script requires the use of files inside the in-memory /run file-system to transfer config/status/metrics and fsnotify to get notified about changes. We can avoid this special solution and unify wwan with other EVE microservices.

...

  • Image size increase: The size of EVE rootfs image will increase by 12MB. This can be reduced by merging mmagent with pillar, at the cost of making pillar and wwan containers tightly-coupled. In the current integration of EVE with MM, the interaction between pillar and wwan is defined only by pubsub channels and their messages, meaning that for pillar it is irrelevant how wwan is implemented.
  • Missing features: There are few things not exposed by ModemManager API. For example, packet and drop counters are not available and only byte counters can be retrieved. Also, it is not possible to differentiate between inactive SIM slot with SIM card inserted and inactive SIM slot without SIM card. With qmicli we can tell these states apart. Lastly, for a visible network provider (which is scanned and published if wwan.query.visible.providers is enabled), I we cannot tell if roaming is required (we publish that as a boolean flag).
    Neither of these are particularly important features and these gaps could be filled in with our future contributions to ModemManager.  

  • More dependencies: Using ModemManager introduces new dependencies, notably the D-Bus daemon and udev. In standard Linux distributions, these daemons are utilized by multiple services, which justifies their presence. However, in EVE, they would serve the sole purpose of managing cellular connectivity (although there is some potential for more uses of udev, especially for handling of hot-plug devices). On the other hand, it appears that neither of these daemons consume significant resources, as confirmed by checking with 'top'.

    Code Block
     PID  PPID USER 	STAT   VSZ %VSZ CPU %CPU COMMAND
 28   1    root     S      706m  4%   0   0% mmagent
 25   1    root     S      16576 0%   0   0% ModemManager --debug
 38   1    root     S      9760  0%   0   0% /usr/libexec/qmi-proxy
 12   1    root     S      5596  0%   0   0% udevd --debug --daemon
  1   0    root     S      1616  0%   0   0% {mm-init.sh} /bin/sh /usr/bin/mm-init.sh
 10   1    messageb S      1472  0%   0   0% dbus-daemon --system


...

But please note that I tested this only with my personal device and just two modems. I suggest that some of the EVE users that plan to rely on modems heavily could give it a try on their devices and provide feedback. Images for testing are published on dockerhub:

Should we decide to use ModemManager in EVE, there is a follow-up initiativestory to productize this integration. This mostly involves more testing, writing documentation, cleaning up the code a bit and getting it through reviews.

...