Mastering Linux Kernel Modules with Kmod

The Linux kernel supports dynamically loadable modules to extend functionality at runtime. This enables customizing kernel capabilities without rebuilding it entirely. The kmod utility in Linux makes managing these modules user-friendly.

In this advanced guide, we will dive deeper into Linux kernel module concepts and see how seasoned developers and administrators can harness modules effectively using kmod.

Kernel Modules – A Primer

Since Linux 2.6, kernel modules have been imperative for adapting the kernel to diverse systems. But what exactly are modules?

As per the Linux Kernel Module Programming Guide, a kernel module "allows a developer to add code to the kernel without having to reboot the system."

Key advantages include:

• Adding new hardware support on demand – especially important on production systems.
• Implementing new filesystems, drivers, network stacks.
• Fine-grained control over customizations.
• Faster development – just compile required modules without full kernel builds.
• Focused component testing for quality.

Internally, modules explicitly register provided functionality via exported symbols. The kernel references these symbols to use the registered capabilities.

Kernel Module Statistics

A snapshot of the modules in a standard Linux 5.15 kernel provides insights on scale and usage.

Total modules – Evident from Ubuntu 22.04 LTS:

$ lsmod | wc -l
176

Module sizes – Ranging from just 486 bytes to several MB:

Functional categories – Covering diverse capabilities likeperf events,virtualization,networking etc:_

This data indicates that despite the monolithic architecture, the Linux kernel relies heavily on modules for extension and flexibility.

Understanding these OS design principles helps gauge the role of kmod better. Now let‘s move on to utilizing it for simplified management.

Kmod Module Loading Sequence

The standard tools for loading kernel modules are modprobe and insmod. But kmod acts as an intermediate layer that handles dependencies before invoking them under the hood.

This sequence is applicable when using modprobe for loading modules:

1. User runs modprobe for a module say mydriver.
2. Modprobe contacts kmod to load dependencies first.
3. Kmod parses module metadata to get dependency info.
4. Any required modules like kernel APIs or libraries are loaded using modprobe.
5. Once dependencies initialize successfully, mydriver module is loaded.
6. Kmod passes control back to modprobe after loading finished.

So kmod handles the heavy lifting of dependency resolution based on symbols and versioning info.

This facilitates simpler modprobe usage for end-users without bothering about intricate dependencies.

Why Does Kmod Simplify Module Management?

Kmod was created due to deficiencies of earlier tools like lsmod, insmod, rmmod and modprobe as noted by author Lucas De Marchi:

• Dependency resolution was manual and error-prone.
• No guarantees of module unload order or multi-version support.
• No checks for symbol clashes while loading modules.
• Limited information on in-use modules.

By addressing these gaps in tools like modprobe and consolidating module management, kmod delivered superior usability.

Let‘s now move on to leveraging kmod for more efficient module debugging and customization.

Debugging Modules with Kmod

When working with custom modules, issues like failed loads, version mismatches or clashes can occur. Kmod aids rapid diagnosis by unveiling dependencies and symbol exports.

Consider troubleshooting a modified i915 GPU driver(drm_i915) failing to load on Ubuntu 22.04:

Error from modprobe

modprobe: ERROR: could not insert ‘i915‘: Unknown symbol in module

Step 1 – Check symbols offered by module

Kmod can detect unresolved symbols using modinfo:

$ modinfo i915 | grep symbol
depends:        drm
vermagic:       5.15.0-41-generic SMP mod_unload modversions 
parm:           modeset:Disable/Enable modesetting (int)

No symbol issues visible. So why won‘t it load? Time to get dependencies.

Step 2 – Examine dependency chain

Using kmod show reveals that drm_i915 needs lib/modules/5.15.0-41-generic/kernel/drivers/gpu/drm/i915/i915.ko:

$ kmod show drm_i915
Libkmod version: 27
Module                  drm_i915
Module Path            /lib/modules/5.15.0-41-generic/kernel/drivers/gpu/drm/i915
Module Depends         drm
Module Version         5.15.0-41-generic

Aha! It requires the parent drm module, which itself depends on other graphics and DRM kernel APIs.

Step 3 – Load missing part of dependency chain

$ sudo modprobe drm
$ sudo modprobe i915

Now i915 loads successfully! Kmod provided visibility into non-obvious deps issue.

Similarly, obsure issues due to custom/third party modules can be diagnosed using modinfo and kmod show.

Customizing Module Parameters

Module functionality can be tuned by passing load-time parameters to alter behavior.

For example, the nfsd module that provides NFS daemon contains several key module parameters:

$ modinfo nfsd | grep parm
  parm:                 nfsd:Number of nfsd threads to start (unsigned int)
  parm:                 nfssvc_workers:Number of threads to use for serving NFS requests (unsigned int)
  parm:                 nfsv4_disable_idmapping:Disable NFSv4 idmapping feature (bool)

To load nfsd with 8 worker threads instead of the default 4:

modprobe nfsd nfsd=8 nfssvc_workers=8

The ability to tweak module settings is invaluable for administrators. Kmod enables viewing available options using modinfo before loading modules with custom configs.

Emerging Use Cases of Kernel Modules

While traditional drivers and filesystems still dominate, new classes like sandboxed security modules and dynamic tracers are emerging.

1. Seccomp BPF Sandboxing

The Seccomp BPF framework leverages the Linux BPF virtual machine to restrict process actions. Sandboxed environments improve security and resource isolation:

• Chrome browser uses it to isolate sites from OS.
• Docker sandboxing depends on it.
• Kubernetes gVisor runsc implements a seccomp profile.

Seccomp is implemented as a standard loadable module in the kernel. No custom kernel builds needed!

2. eBPF Tracing and Observability

Enhanced BPF (eBPF) allows running sandboxed programs in the kernel to trace activity and collect telemetry – without changes to application code.

It powers sophisticated observability for Linux monitoring, security, networking use cases:

• Detect endpoint threats by tracing kernel activity.
• Low-overhead monitoring through custom BPF programs.
• Inspect networking stacks via tc, XDP, cilium.

And the bpf subsystem enabling it is modular – dynamically loadable as bpf.ko and bpf-jit.ko.

eBPF adoption is accelerating. Kmod allows painless inserts to enable radical monitoring abilities!

So kernel modules continue to evolve providing benefits through novel capabilities.

Recommendations for Module Developers

For developers building custom kernel modules, keeping certain best practices in mind will facilitate easier debugging and maintenance:

• Maintain debug symbols in modules to enable tracebacks on failures.
• Specify module license info properly for compliance.
• Set versioning carefully based on kernel changes.
• Use symbol namespaces to avoid clashes.
• Add help text for non-obvious module params.
• Try to restrict external dependencies for custom drivers.
• Run static linters to catch issues early.

Following Linux kernel coding style and programming practices will result in robust dynamic extensions.

Leveraging utilities like kmod then simplifies loading these modules reliably.

Conclusion

Kmod brings kernel modules into the realm of easy extensibility by handling cumbersome dependencies and management tasks.

Whether building flexible systems using virtualization and sandboxing or diagnosing issues with GPU drivers – kmod will have your back!

With emerging trends like eBPF acceleration, the relevance of dynamic inserts only continues to grow. So while the core Linux kernel remains monolithic, the edges stay flexible and modular.

Kmod thereby perfectly balances both worlds, enabling enterprises to reap benefits without the downsides.