...
Rsyslog supports batching of log messages before sending to output plugin. Rsyslogd starts a transaction, sends a a bunch of logs and then ends transaction. Plugin has the option to selectively acknowledge a subset of log messages or reject all. Rsyslogd does not mark these log messages as completed until EVE plugin acknowledges/accepts these log messages. This mechanism can be used for putting back pressure to rsyslogd in the event of network or other failures.
Envisioned list of log sources on an EVE edge-node
- EVE specific agents. This include EVE agents like zedagent, zedrouter, domainmgr, verifier, zedmanager, downloader, identitymgr, vaultmgr, baseosmgr, nim, nodeagent, ledmanager, wstunnelclient, lisp-ztr and python lisp control plane.
- External tools used by EVE. This include dnsmasq, radvd, watchdog, dhcpcd
- EVE specific scripts and short running executables. eg. client, diag, device-steps.sh
- Other containers running in EVE control domain (dom0). eg. wwan, wlan, guacd, sshd, vtpm (vtpm_server) and ntpd containers.
- Hypervisor and associated tools. eg. xen, xenstored, xenconsoled, xl, qemu.
- Kernel logs.
These log sources can broadly be grouped into three categories based on how their logs are output (log destinations):
- Sources that send logs only to files. Such sources include hypervisor, qemu, lisp control plane, xen-tools.
These logs will need special handling. We can start with having such logs be dumped to /var or /persist and use imfile module of rsyslogd to pick up from there.
We should later invent a mechanism like LD_PRELOAD or named pipes to makes these sources send logs to rsyslogd without using files. - Sources that are flexible and can be made to change their log destination easily. Such sources include EVE agents, short lived EVE executables and scripts.
EVE agents for examples can be changed (since they use logrus for their logging needs) to send their logs directly to syslog (/dev/log) or stdout and then re-direct to rsyslogd using logger tool. - Sources that send logs to both syslog and files. Eg. dnsmasq, dhcpcd, radvd, watchdog etc.
No special handling will be required in this case. - Sources that run inside containers and the logs of which are collected by memlogd. Eg. wlan, wwan, sshd, guacd, ntpd, vtpm etc.
Linuxkit has a module called memlogd that collects container logs in a circular buffer. Linuxkit's memlogd module is shipped with logread tool that can read logs from memlogd and output to it's stdout.
Output of logread can be piped into logger and subsequently sent to rsyslogd. - Kernel logs (easily collected by rsyslogd). Rsyslogd has a module called imklog that can read kernel logs into rsyslogd.