RIOT Runtime Configuration Registry

This is a continuation of the effort previously done in #10622, but its architecture has dramatically changed since then.

Abstract

This PR implements a system level runtime configuration system for RIOT.

A runtime configuration system is in charge of providing a mechanism to set and get the values of Configuration Parameters that are used during the execution of the firmware, as well as a way to persist these values. Runtime configurations are deployment-specific and can be changed on a per node basis.
Appropriate management tools could also enable the configuration of nodes.

Examples of runtime configurations are:

• Transmission duty cycles
• Sensor thresholds
• Security credentials
• System state variables
• RGB-LED colors

These parameters might have constraints, like a specific order to be applied (due to interdependencies) or value boundaries.

The main advantages of having such a system are:

• Easy to apply per-node configuration during deployment
• No need to implement a special mechanism for per-node configurations during firmware updates (only in the case of migration), as the parameters persist.
• Common interface for modules to expose their runtime Configuration Parameters and handle them
• Common interface for storing Configuration Parameters in non-volatile storage devices

Modules

The RIOT Registry consists of various modules that are not all needed for minimal operation:

registry
Core functionality such as set, get, export and apply operations to retrieve and set configurations.

registry_storage
Adds capability to safe parameters to non-volatile storage devices.

registry_int_path
Allows accessing configuration parameters via a path of integers.

registry_string_path
Allows accessing configuration parameters via a path of strings.

registry_cli
Allows accessing configuration parameters via a shell command.

Design

Architecture

The proposed architecture, as shown below, is formed by one or more Applications or Configuration Managers and the RIOT Registry.
The RIOT Registry acts as a common interface to access Runtime Configurations and store them in non-volatile storage.
All runtime configurations can be accessed either from the RIOT application using the provided RIOT Registry interfaces or through the interfaces exposed by the Configuration Managers.
A RIOT Application may interact with a Configuration Manager in order to modify access control rules or enable different exposed interfaces.

Path Based Configuration Managers (Needs int_path or string_path extension)

These Configuration Managers are a simple representation of the default configuration structure of the RIOT Registry.
They use either the int_path or the string_path Registry extension module to expose the parameters using a path.

Custom Schema Based Configuration Managers

These Configuration Managers have their own configuration structure (custom predefined object models etc.) and can not automatically be mapped to / from the RIOT Registry itself.
To make them work, a custom mapping module needs to be implemented, which maps each Configuration Parameter from the registry to the correct format of the Configuration Manager.

Namespaces and Storages

The RIOT Registry interacts with RIOT modules via Configuration Schemas, and with non-volatile storages via Storages.
This way the functionality of the RIOT Registry is independent of the functionality of a module or storage implementation.
It is possible to get or set the values of Configuration Parameters.
It is also possible to transactionally apply configurations or export their values to a buffer or print them.
To persist Configuration Values, it is possible to store them in non-volatile storages.

Any mechanism of security (access control, encryption of configurations) is not directly in the scope of the Registry but in the Configuration Managers and the specific implementations of the Configuration Schemas and Storages.

The graphic below shows an example of two Configuration Namespaces (SYS and APP).
The APP namespace contains a application specific My app Configuration Schema and the SYS namespace specifies a WLAN and a LED Strip Configuration Schema.
The application My app uses the custom My app Configuration Schema to expose custom Configuration Parameters to the RIOT Registry and the drivers WS2812, SK6812 and UCS1903 contain instances of
the LED Strip Configuration Schema to expose common LED Strip Configuration Parameters.

Also, there are two Storages available: MTD and VFS.
The MTD Storage internally uses the RIOT MTD driver and the VFS Storage internally uses the RIOT VFS module.

Components

The RIOT Registry is split into multiple components as can be seen in the graphic below:

Registry Core

This component holds the most basic functionality of the RIOT Registry.
It allows to set and get Configuration Values, transactionally apply them to make the changes come into effect and export all Configuration Parameters that exist in a given Configuration Namespace, Configuration Schema or Configuration Group.

Furthermore it is possible to add Configuration Namespaces or Configuration Schema Instances.

Registry Namespace

The Configuration Namespaces such as SYS or APP and their respective Configuration Schemas are not part of the Registry itself.
It is possible to add custom Configuration Namespaces depending on the given needs.

Storage

The Storage component provides an interface to load Configuration Values from a persistent Storage implementation or to save the current Registry configuration to it.

Registry Storage [Extension]

The implementations of a Storage such as VFS or MTD are not part of the Registry itself and can be switched out with implementations that are most suitable to the given needs.

Integer Path [Extension]

The int_path component provides helper functions that convert a path of up to 4 integer values to the respective pointer of a Configuration Namespace, Configuration Schema, Configuration Schema Instance, Configuration Group or Configuration Parameter and the other way around.

The structure of an integer configuration path is the following:
namespace_id / schema_id / instance_id / group_id | parameter_id

For example:

String Path [Extension]

The string_path component provies helper functions that convert a string path to the respective pointer of a Configuration Namespace, Configuration Schema, Configuration Schema Instance, Configuration Group or Configuration Parameter and the other way around.

The structure of a string configuration path is the following:
namespace_name / schema_name / instance_id (/ group_name)* / parameter_name.

The amount of path items is flexible, so the path could only consist of the namespace_name or only the namespace_name and the schema_name and so on.

For example:\

API

The graphic below shows the API of the RIOT Registry.
The top shows the Core API to manage Configuration Parameters.
On the right-hand side are functions to set and get Configuration Parameters, transactionally apply them and export them to a buffer or terminal.
On the left-hand side are setup functions to add Configuration Namespaces and Configuration Schema Instances to the Registry.

The bottom shows the storage API to manage the persistance of Configuration Parameters.
The left-hand side shows functions to load” and save Configuration Parameters to and from the persistent Storage.
The right-hand side shows functions to add Storage Sources (for reading) and to set a Storage Destination (for writing).
The Registry can have multiple Storage Sources, but always only one Storage Destination.
This allows to migrate from an old Storage to a new one.

The functionality of these functions is explained in the following paragraphs.

Core API

Get

A Configuration Value can be retrieved using the registry_get function.
The function takes the registry_node_t and a registry_value_t pointer (to return the value) as its arguments.

int registry_get(
    const registry_node_t *node,
    registry_value_t *value
);

Set

A Configuration Value can be set using the registry_set function.
The function takes the registry_node_t, a void* buffer and the buffer size as its arguments.

The buffer must contain the value in its correct c-type.
If the Registry expects a u8, but a u16 is provided, the operation will fail.
Furthermore the registry can specify constraints like minimum and maximum values and an array of allowed or forbidden values.
If these constraints are not fulfilled, then the operation will fail as well.

int registry_set(
    const registry_node_t *node,
    const void *buf,
    const size_t buf_len,
);

Apply

Once the value(s) of one or multiple Configuration Parameter(s) are changed by the registry_set function, they still need to be applied, so that the new values are taken into effect.

Configuration Parameter(s) can be applied using the registry_apply function.
In this case the Registry provides multiple apply functions to allow applying in varying degrees.
The provided functions are registry_apply, this function applies every Configuration Parameter currently available in the Registry within the specified scope, specified by the registry_node_t argument.

When a whole Schema Instance, or a single Configuration Parameter is applied, it will be passed on to the apply_cb handler of the Configuration Schema Instance, provided by the module that needs runtime configuration.
This way the module gets notified, when the Configuration Parameter has been applied and can apply the changes accordingly.

int registry_apply(const registry_node_t *node);

Export

Some times it is convenient to have a way to see what Configuration Namespaces, Configuration Schemas, Configuration Schema Instances, Configuration Groups or Configuration Parameters are available within our current RIOT Registry deployment.
To get this information there is the registry_export function.

This function exports every Configuration Object currently available in the Registry within the scope of the provided registry_node_t argument.

When a node in the schema is exported, it will be passed on to the export_cb handler provided as an argument of each registry_export* function.

int registry_export(
    const registry_node_t *node,
    const registry_export_cb_t export_cb,
    const uint8_t tree_traversal_depth,
    const void *context
);

Add Namespaces to the Registry

To be able to use Configuration Schemas and their Parameters etc. it is necessary to add a Configuration Namespace that holds the required Configuration Schemas to the Registry.

This is possible using the REGISTRY_ADD_NAMESPACE macro, providing the name of the Configuration Namespace and a pointer to a registry_namespace_t object as arguments.

#define REGISTRY_ADD_NAMESPACE(_name, _namespace)

Add Configuration Schema Instances to the Registry

To implement runtime configuration functionality into a module, it is necessary to add a Configuration Schema Instance of the needed Configuration Schema to the Registry.

This is possible using the registry_add_schema_instance function, providing the Configuration Schema and the Configuration Schema Instance as arguments.

int registry_add_schema_instance(
    const registry_schema_t *schema,
    const registry_instance_t *instance,
);

Storage API (Extension)

Load from Storage

It is often needed to load Configuration Parameters from a non-volatile Storage device.
For example when a device restarts after a shutdown.

This is possible using the registry_storage_load function.
This function takes a registry_storage_instance_t as its arguments.
The the registry_storage_instance_t contains data such as the mount point.
Internally the Storage Instance searches its persistent storage device for Configuration Values.
If a Configuration Value is found, the Storage Source calls the registry_set function and provides the necessary arguments.

int registry_storage_load(
    const registry_storage_instance_t *storage_instance,
);

Save to Storage

To save Configuration Values to a non-volatile Storage device, the Registry provides the registry_storage_save function.
This function saves all available Configuration Parameters within the scope specified by the registry_storage_node_t argument.

Internally these functions call the save handler of the provided Storage Instance for each Configuration Parameter that has to be saved to storage.
The save handler of the Storage, of the provided Storage Instance takes a Storage Instance providing data such as the mount point, a registry_node_t, which describes the location inside the configuration tree and a Configuration Value as its arguments.
These values provide the needed information to load them back into the registry.

The way how the Storage stores these values internally is not specified.

int registry_storage_save(
    const registry_storage_instance_t *storage_instance,
    const registry_node_t *node,
);

Set Storage Instances

To be able to expose storage instances to configuration managers such as the RIOT CLI, we have a function called registry_storage_set_instances, which lets you provide an array of storage instances.

registry_error_t registry_storage_set_instances(
    const registry_storage_instance_t **storage_instances,
);

Get Storage Destination

To be able to retrieve a list of existing storage instances, we have a function called registry_storage_get_instances, which takes a pointer to an array of storage_instance_t structs as an output parameter.

registry_error_t registry_storage_get_instances(
    const registry_storage_instance_t ***storage_instances,
);

Comparison to Apache Mynewt Config

While originally our work on the RIOT Registry was heavily inspired by Apache Mynewt Config, it has since evolved to provide features such as Type Safety, make String Paths optional, introduce Integer Paths and provide a more modular Pointer-Based API.

The table below shows the difference between Apache Mynewt Config and the proposed RIOT Registry.

The Idea here is not to talk down Apache Mynewt Config, as its simplicity of course has its own advantages, but to point out more clearly how our solution differs to it.

External Configuration Managers

CLI

The only available Configuration Manager in this PR is a CLI.
It can be tried in the example under examples/registry.

The CLI uses int paths separated by /.
E.g.: 0/0/0/0.

CoAP API

The idea is to use the registry_int and/or registry_path module to provide a simple CoAP API.

/get endpoint

The get function could be mapped to the CoAP GET function.

/set endpoint

The set function could be mapped to the CoAP PUT function.

/apply endpoint

This is more tricky.
Possible solutions could be to provide an /apply suffix or prefix at the end or beginning of a path, to tell the registry, that this is an apply operation.

/export endpoint

The export function could be implemented using a CoAP GET function using /export as a suffix or prefix of the path.
CoAP would then return one response containing all the requested objects for example structured using CBOR.

LwM2M

A LwM2M integration would require to write a mapping between each LwM2M Object to the respective RIOT Registry Configuration Schemas.

In this case every LwM2M set operation would immediately trigger a registry_apply as LwM2M does not provide an apply operation.
This is not a drawback, as LwM2M allows to set multiple values at the same time, thus covering the same use-case as the registry_apply function.

Testing procedure

Testing commands unter tests/unittests:

make tests-registry term

make tests-registry_storage term

make tests-registry_storage_heap term

make tests-registry_storage_vfs term

make tests-registry_int_path term

make tests-registry_string_path term

Issues/PRs references

See also #10622
See also #19557