Page History

Zedrouter and NIM are already one of the most complicated microservices within EVE. This is mostly due to the fact that they have to deal with many configuration items (routes, bridges, interfaces, etc.). A single high-level configuration object defined by the controller, such as a device port or a network instance, is actually built using multiple configuration primitives inside the (Linux) network stack. EVE may even start additional processes, like dnsmasq or radvd, configured accordingly to serve requested network services.

EVE has to deal with the intended (also called desired) state on one side, which in this case is EdgeDevConfig received from the controller, and with the actual (also known as the current) state on the other side - the configuration currently applied and running on the device. The intended state is defined (using protobuf) to be more high-level, describing the intent but not the implementation. EVE first has to map it to low-level configuration primitives that implement the desired functionality. Next, it has to determine the difference between the currently running configuration and the new intended state. Finally, it performs all changes necessary in the form of Create/Modify/Delete operations to transition from the current state to the new intended state. Ordering of these operations matters and it has to respect any dependencies that exist between configuration items. For example, an application virtual interface (VIF) cannot be inserted into a bridge if that bridge has not yet been created. This is further complicated by the fact that the configuration space is split between microservices, sending updates between each other using pubsub.

...

However, the topic of this proposal is the tasks 2. and 3., which can be solved generically using a dependency graph. Every configuration primitive rendered from the step 1. can be represented by a single graph node. Dependencies between items are modeled using directed edges. For the start, we will need to support dependency with the semantics "must existsexist". For example, if a route depends on a particular network interface to be configured first, there would be an edge originating at the route's graph node and pointing to the node of the network interface. A visual example with a dependency graph representing network configuration is shown below:

...

1. Store the currently running state
2. Allow to prepare the new intended state
3. Move the current state towards the new intended state - this means to:
   1. Determine "diff" between the current and the new state
   2. Determine ordering of Create/Modify/Delete operations that respects the dependencies
   3. Run operations
   4. Store and expose any errors returned by Create/Modify/Delete operations

For the step 1., the graph must also allow to represent configuration items managed by microservices other that the one that owns the graph (let's label them as "external"). For those, the graph will never trigger Create/Modify/Delete operations and will use them only for the purposes of dependency management (e.g. create A only after another microservice has already created B).

For the step 3.c, the graph needs to have an access to handlers of configuration items (those which are not external). For the graph this can be a structure that implements an interface with Create/Modify/Delete methods. For every distinct configuration item type (like "Linux route", "container", "Linux bridge", "dnsmasq"), there will be a separate handler registered with the graph. For the graph, these handlers (in the preliminary implementation called "Configurators") are essentially backends or drivers, that the graph calls as needed to synchronize the current state with the latest desired state.

...

whenNetworkConfigChanges() {
	determineObsoleteVlans()
	removeObsoleteVlans()
	determineObsoleteBonds()
	removeObsoleteBonds()
	changeInterfaceIPsIfNeeded()
	ifSomethingChangedRestartDnsmasq()
	addNewBonds()
	addNewVlans()
	...
}

whenNetworkConfigChanges() {
	newConfig := []ConfigItem{
		interface(params),
		arpEntry(params),
		arpEntry(params),
		route(params),
		route(params),
		bridge(params),
		dnsmasq(params),
		// A comment explaining why this config item is here…
		iptablesChain(params),
		iptablesChain(params),
	    ...
	}
	graph.Cluster(<network-instance-name>).Put(newConfig)
	err := graph.Sync()
    ...
}

Note that the example also presents the concept of clustering (subgraphs), that the depency graph will support and which was borrowed from graphviz. Having support for graph clustering will allow us to group items which are in some way related to each other. For example, all components of the same application (domain, volume, VIFs) could be grouped under one cluster. This will be mostly done to simplify modifications to the intended state. As shown demonstrated in the example, the intended state of a single specific network instance will can be replaces replaced with just one function call: graph.Cluster(<network-instance-name>).Put(newConfig)

Note that the new approach is not only easier for the developer and therefore less bug-prone, but also allows to explicitly express the intent (= newConfig), while the steps (the sequence of configuration changes) needed to take to get there are implicit. Compare that with the current approach, where the steps are explicit, but the programmer's intent is implicit. To determine what the program is trying to configure, one must study the code thoroughly and build a mental image of the intended state. If the programmer made a mistake in that complex code, one might get a wrong idea of what the intended state is.

Lastly, with the dependency graph, it will be much easier to add new features. A programmer will only need to implement handlers for new configuration items and describe their dependencies. The rest is being taken care of by the graph.