DEV Community: Pierre Gradot

User-Defined Deduction Guides for Class Templates in C++

Pierre Gradot — Mon, 25 Aug 2025 13:56:35 +0000

I remember when I first got serious about modern C++. It was in 2017, my company was starting a new project from scratch, and we chose the latest standard: C++17, which had just been released. Diving into all the modern features of C++, I remember one feature that left me particularly puzzled: user-defined deduction guides. They are used to influence CTAD (Class Template Argument Deduction), which was a major feature of C++17. I was puzzled because I couldn't understand why anyone would ever need to write such a guide by hand...

Last year, after 7 years of C++17 and above, I finally needed to write my very first own template deduction guide! 🤯

I know, it took me 15 months to write an article on this topic, but here it is!

Context

I was working on a plugin for QEMU's ARM system emulator to record ("trace") the execution of an ARM system running Android. I wrote a generic trace generator, and then 2 specializations for ARM 32-bit or 64-bit variants, all in the same plugin. The code avoided virtual functions as much as possible because the application was highly performance-sensitive and I didn't want to pay for the cost of vtables and virtual calls.

Genericity was mainly achieved by using templates. A lot of templates. Most of these templates worked well with the automatic template deduction rules. Most of them, but not all of them. This is the context where I needed my own deduction guide: writing templates to trace the execution of a CPU, with all the modifications to its registers and to the system memory.

In the following, I will of course not show the real code, but a minimal dummy code to illustrate the situation. It will really be a dummy code: don't expect well-designed classes, just plausible code where a "user-defined deduction guide" is necessary. However, I think that somehow realistic code samples will be more instructive than Foo-Bar examples.

An Example Where CTAD Fails

Base class template

Let's start with the base class template for a generic trace generator:

#include <iostream>
#include <tuple>

template <typename Registers, typename... MemoryReaders>
class TraceGenerator {
public:
    explicit TraceGenerator(std::tuple<MemoryReaders...>&& readers) :
        registers_m{},
        readers_m{readers} {}

    virtual ~TraceGenerator() = default;

    void generate() {
        std::cout << __PRETTY_FUNCTION__ << '\n';
        some_architecture_specific_function();
        // ...
        // Let's omit the code that uses registers_m and readers_m, for the sake of simplicity
    }

private:
    Registers registers_m;
    std::tuple<MemoryReaders...> readers_m;

    virtual void some_architecture_specific_function() = 0;
};

The template has 2 parameters:

a type Registers, with the registers of the target architecture / CPU.
a list of types MemoryReaders, to read the memories of the target machine.

There is one pure virtual function so this "class" must be inherited to implement the function. Yes, I said that I avoided virtual as much as possible, not at all 😉

Specialization for ARM

Android runs on ARM processors. We need something for Android.

struct ArmRegisters {};

template <typename... MemoryReaders>
class ArmTraceGenerator : public TraceGenerator<ArmRegisters, MemoryReaders...> {
public:
    using TraceGenerator<ArmRegisters, MemoryReaders...>::TraceGenerator;

private:
    void some_architecture_specific_function() override {
        std::cout << __PRETTY_FUNCTION__ << '\n';
    }
};

We implement the pure virtual function, we set the Registers template parameter, but we still have a class template: the MemoryReaders pack still have to be set to create an actual class. Because constructors are not automatically inherited in C++, we must add this using declaration to expose the base class constructors (see section "Inheriting constructors" from "Using-declaration" on cppreference). And yes: because the content of Registers is not actually used here, an empty structure is enough for our example.

Finally, we can write a simple main() that uses this class template to create an actual generator for our target. Let's say this machine has 2 memories, one RAM, one FLASH. We create two reader classes, a tuple with an instance of one of each these classes, and we can finally create the trace generator:

struct RamReader {};

struct FlashReader {};

int main() {
    auto generator = ArmTraceGenerator(std::tuple{RamReader{}, FlashReader{}});
    generator.generate();
}

We create our object with auto generator = ArmTraceGenerator(...) without explicitly specifying the types in the parameter pack. This means we are asking the compiler to deduce the MemoryReaders parameter pack and instantiate the appropriate ArmTraceGenerator class.

However, as beautiful and valid as this code seems, it doesn't compile (*):

main.cpp: In function ‘int main()’:
main.cpp:46:79: error: class template argument deduction failed:
   46 |         auto generator = ArmTraceGenerator(std::tuple{RamReader{}, FlashReader{}});
      |                                                                               ^

(... followed by many other template errors, as always with templates)

Why? Because CTAD is not applicable in this situation, as explained in P2582R1 :

In C++17, deduction guides (implicit and explicit) are not inherited when constructors are inherited

P1021R6 already suggested to change this behavior for C++20, but it "was not finalized in time". P2582R1 also proposed a similar change, and it was accepted for C++23. However, the compiler support is still very poor for P2582R1 "Class template argument deduction from inherited constructors": GCC implements it since version 14, but Clang and MSVC don't.

(*) = so "it doesn't compile" on my machine because I am compiling this example with GCC 12.2 and -std=c++20.

For this code to compile, we have 4 solutions:

Use GCC 14+ and -std=c++23.
Don't rely on CTAD: specify the types explicitly with auto generator = ArmTraceGenerator<RamReader, FlashReader>(...). This is fine here, as there are only 2 types in the parameter pack. Nevertheless, listing all the actual types can become a nightmare. In my real code, I had a parameter pack with more than 1000 elements, so CTAD was essential.
Remove the using TraceGenerator<ArmRegisters, MemoryReaders...>::TraceGenerator to stop inheriting constructors and provide a compatible constructor directly in the derived class:
```
explicit ArmTraceGenerator(std::tuple<MemoryReaders...>&& readers)
    : TraceGenerator<ArmRegisters, MemoryReaders...>{std::move(readers)} {}
```
Write a user-defined deduction guide.

Let's explore this last possibility (as it's the topic of this blog post 😂).

User-Defined Deduction Guide

Since the compiler cannot deduce the template arguments, we can write a deduction guide to help it. This is surprisingly simple:

template <typename... MemoryReaders>
ArmTraceGenerator(std::tuple<MemoryReaders...>) -> ArmTraceGenerator<MemoryReaders...>;

Basically, this guide explains that creating an ArmTraceGenerator object from a tuple<MemoryReaders...> in fact creates an ArmTraceGenerator<MemoryReaders...>.

As with everything in C++, user-defined deduction guides must be known when they are to be used, so this guide must be before the main() function.

The code now compiles and prints:

void TraceGenerator<Registers, MemoryReaders>::generate() [with Registers = ArmRegisters; MemoryReaders = {RamReader, FlashReader}]
void ArmTraceGenerator<MemoryReaders>::some_architecture_specific_function() [with MemoryReaders = {RamReader, FlashReader}]

Other Examples

You might be thinking:

OK, fine. You've shown an example where CTAD fails, but C++23 solved the issue and a user-defined guide was not even required, as they were workarounds even in C++17. Do you have any other (more solid) examples?

Yes 🥳

Example Where CTAD Fails Without a Workaround (*)

(*) = At least, I haven't found any.

Let's consider another class template to handle chunks of memory. It encapsulates an std::span over caller-owned data:

template <typename T>
    requires std::is_integral_v<T>
class MemoryChunk {
public:
    explicit MemoryChunk(T* data, std::size_t size) :
        chunk_m(data, size) {
        std::cout << __PRETTY_FUNCTION__ << '\n';
    }

    void print_info() const {
        std::cout << __PRETTY_FUNCTION__ << '\n';
        const auto bytes = chunk_m.size() * sizeof(T);
        std::cout << "Memory size = " << bytes << " bytes" << '\n';
    }

private:
    std::span<T> chunk_m;
};

We can use it:

int main() {
    auto data = std::array<int, 16>{};
    auto chunk = MemoryChunk{data.data(), data.size()};
    chunk.print_info();
}

This works perfectly fine, CTAD is possible and T is deduced to be int, as shown in the program output:

MemoryChunk<T>::MemoryChunk(T*, std::size_t) [with T = int; std::size_t = long unsigned int]
void MemoryChunk<T>::print_info() const [with T = int]
Memory size = 64 bytes

Now, let's say we want construct MemoryChunks from various types of iterators. We can add a templated constructor:

template <std::contiguous_iterator Iterator>
MemoryChunk(Iterator begin, Iterator end) :
    chunk_m(begin, std::distance(begin, end)) {
    std::cout << __PRETTY_FUNCTION__ << '\n';
}

However, we clearly see that CTAD won't work here: T doesn't appear so the compiler cannot deduce it and will require a user-defined deduction guide.

We must find a way to deduce T from Iterator. Our first attempt could be:

template <typename Iterator>
MemoryChunk(Iterator begin, Iterator end) -> MemoryChunk<decltype(*begin)>;

We can try this new constructor in our main():

int main() {
    auto data = std::array<int, 16>{};
    auto chunk = MemoryChunk{data.begin(), data.end()};
    chunk.print_info();
}

Sadly, this code doesn't compile... For once, the last template errors are the most informative (in general, we just read the first one and ignore everything after it 😅):

main.cpp:34:1: error: template constraint failure for ‘template<class T>  requires  is_integral_v<T> class MemoryChunk’
main.cpp:34:1: note: constraints not satisfied
main.cpp: In substitution of ‘template<class T>  requires  is_integral_v<T> class MemoryChunk [with T = int&]’:
main.cpp:34:1:   required by substitution of ‘template<class Iterator> MemoryChunk(Iterator, Iterator)-> MemoryChunk<decltype (* begin)> [with Iterator = int*]’

The last line shows that the compiler is trying to use our guide
The first line tells that the requires clause on T is not satisfied.
The third line shows that T is int&.

T is deduced as int&, not just int. Indeed, *data.begin() returns a reference, not a copy of the first element. An assertion like static_assert(std::is_same_v<decltype(*data.begin()), int&>); doesn't fail.

We just have to modify the rule to remove the reference and the code works fine:

template <typename Iterator>
MemoryChunk(Iterator begin, Iterator end) -> MemoryChunk<std::remove_reference_t<decltype(*begin)>>;

Note that guides are SFINAE friendly. We can have both rules and the code works.

You could even add an obviously incorrect rule like the one below, and the code would still work:

template <typename Iterator>
MemoryChunk(Iterator begin, Iterator end) -> MemoryChunk<typename Iterator::dummy_static_field_that_doesnt_exist>;

Example to Overwrite the Deduced Type

For this last example, let's imagine a super basic Metadata class template:

template <typename T>
struct Metadata{
    T name;
};

We can use it with a string literal:

int main() {
    auto metadata = Metadata{.name = "name of something"};
    static_assert(std::is_same_v<decltype(metadata), Metadata<const char*>>);
}

The assertion shows that CTAD has deduced const char*, which is the correct type for a string literal.

But let's imagine that we want to modify the name in the metadata afterward. metadata.name[0] = 'N'; doesn't compile: error: assignment of read-only location ‘* metadata.Metadata<const char*>::name’.

We can influence CTAD by writing a deduction guide that forces T to be std::string instead of const char*. Unlike our previous guides, it's not templated:

MemoryMetadata(const char*) -> MemoryMetadata<std::string>;

The previous assertion fails and must be replaced with static_assert(std::is_same_v<decltype(metadata), Metadata<std::string>>);. And as desired, metadata.name[0] = 'N'; now compiles.

Conclusion

Class Template Argument Deduction (CTAD) was one of the most powerful features of C++17. It greatly simplifies the use of class templates by eliminating the need to explicitly specify template arguments. However, there are situations in which CTAD fails.

User-defined deduction guides are the solution in these cases. They are fairly simple to write and allow you to restore deduction behavior, or even influence it when the default rules do not meet your requirements. You may never need them, but now you know how to write them and in which situations they can be useful.

How to Fix "Low Disk Space on /boot" on Linux

Pierre Gradot — Tue, 17 Jun 2025 12:56:11 +0000

One day, you power up your computer, wait for Linux to boot, log in... and this message pops up:

You might decide to ignore it, or you might think this is an issue worth looking into. And you’d be right!

However, clicking "Examine" doesn't really tell you how to fix the issue. It "just" shows you the disk usage analysis of the /boot partition:

This might look a little cryptic if you're not familiar with Linux internals...

This article breaks down what this message means and how to clean up the partition safely.

What is `/boot`?

The /boot directory is a special partition that contains the files needed to start the operating system. It typically resides on its own partition.

When you click "Examine" the Disk Analyzer tool displays files from this partition:

grub: the directory with the files for the GRUB bootloader, the first program that runs when you power on your computer.
initrd.img-*: the initial RAM disk, a small temporary filesystem loaded into memory before the actual filesystem can be mounted.
vmlinuz-*: the compressed Linux kernel, which is the core of the operating system.
System.map-*: a symbol table that maps kernel functions to memory addresses (mostly useful for debugging).
config-*: the kernel configuration file used to build each kernel.

Each kernel version comes with its own set of these files. That’s why you might see several groups with version numbers: 6.12.27, 6.12.22, 6.10.11, etc. The latest one is probably the kernel currently in use, while the others were left behind after system updates.

Over time, these accumulate and can fill up the partition. This is exactly what the message is telling us: the partition /boot is almost full.

You probably got it: the files in this partition are critical for the system, and it's important to have space available for new kernel updates, as they often include security fixes.

In the next section, we’ll identify which files are still needed, and which ones can be safely removed.

How Resolve The Issue?

1: Expand The Partition

If a partition is almost full, one possible solution is increase its size. I've written a comprehensive guide on resizing partitions on Linux.

However, I don't believe it's a good solution here. New kernel are regularly installed, so you'll likely run out of space again. Furthermore, old kernel are generally no longer needed and can be removed.

2: Use Synaptic Package Manager

The easiest solution is to manually select and remove old kernel packages using Synaptic Package Manager.

Open it with root privileges, click the "Search" button, and search for "linux-image":

Sort the packages by installed version in descending order (the latest versions will appear at the top of the list):

It's wise to keep 2 or 3 kernels, just in case a newer kernel causes issues. Yes, a new kernel can break something on your computer. For example, read this story where I describe how my Wi-Fi stop working after an update from 6.10.11 to 6.11.10 and how I installed a new driver manually to restore connectivity.

Select the old kernels, right-click on them, and select "Mark for complete removal":

Here, I decided to keep 6.10.11 (where my Wi-Fi was still working out-of-the-box) and the 2 latest versions.

Finally, click on the apply button to remove them.

3: Use The Command Line

If you prefer command lines over GUIs, you can manually search for and remove the packages manually from a terminal.

First, list the installed packages:

$ apt list --installed | grep linux-image

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

linux-image-6.10.11+bpo-amd64/now 6.10.11-1~bpo12+1 amd64 [installed,local]
linux-image-6.12.22+bpo-amd64/stable-backports,now 6.12.22-1~bpo12+1 amd64 [installed,automatic]
linux-image-6.12.27+bpo-amd64/stable-backports,now 6.12.27-1~bpo12+1 amd64 [installed,automatic]
linux-image-amd64/stable-backports,now 6.12.27-1~bpo12+1 amd64 [installed]

I'm running this command on my computer after removing old kernels with Synaptic Package Manager, and we can see that 6.10.5 and 6.5.0 are no longer present.

Second, delete the packages you want with sudo apt remove linux-image-<version>.

4: The Automatic Way

Another solution is to run sudo apt autoremove --purge from a terminal.

This command removes unused packages, typically dependencies that were automatically installed and are now unused. The --purge flag goes one step further by removing their configuration files as well.

However, this approach is somewhat indirect: you’re not specifically targeting old kernels, you just hope that the package manager considers them as unused and will remove them automatically. In my case, it wasn't enough: the free space went from 11 to 19 MB, and the message reappeared at the next boot.

If I run the command now on my computer, here's what I get:

 % sudo apt autoremove --purge

[sudo] password for pierre: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  linux-headers-6.10.6+bpo-common* linux-kbuild-6.10.6+bpo*
0 upgraded, 0 newly installed, 2 to remove and 11 not upgraded.
After this operation, 63.6 MB disk space will be freed.
Do you want to continue? [Y/n]

Note that no linux-image packages are selected for removal, not even 6.10.11 which is relatively old (it was released in September 2024, and I'm writing these lines in June 2025).

Still, it's a good idea to run this command after using one of the previous solutions, to be sure to clean up any leftover packages related to an uninstalled kernel (like this linux-headers-6.10.6+bpo-common here).

Conclusion

After cleanup, we can check the results with the Disk Usage Analyzer:

Initially, 430 MB were used. Now, it's down to 273 MB. The df command shows the percentage of free space:

$ df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  456M  260M  171M  61% /boot

Problem solved! Until /boot is full again 😉

clang-tidy, const & (N)RVO

Pierre Gradot — Fri, 06 Jun 2025 06:47:23 +0000

Almost one year ago, clang-tidy threw an unexpected and unfamiliar message in my face: warning: constness of 'str' prevents automatic move [performance-no-automatic-move] (documented here). The incriminated code looked something like this:

auto build_path(const std::string& basedir, int index) {
    const auto str = fmt::format("{}/output/{}", basedir, index);

    fmt::print("{}\n", str);
    // ... do some work with 'str'...

    return str; // <-- the warning was here
}

I didn’t have time to investigate, so I wrote down a note, swore I would look into it later, and just forgot... Until today. I thought it would be simple. It was not. Because, as always, C++ is full of surprises.

Let me tell you the story of clang-tidy, const and (N)RVO.

Almost Always Const Auto (AACA)

Almost Always Auto (or AAA) is a very common practice in modern C++. The acronym was coined by Herb Sutter in episode 94 of Guru Of The Week (GotW) series.

Another common practice is to use const (almost) by default. In Scott Meyers's Effective C++, item 3 is "use const whenever possible" and another episode of GotW begins with "always use const as much as possible, but no more".

In his book C++ Best Practices, Jason Turner includes 3 rules about auto and const(expr):

Rule 12: "const Everything That’s Not constexpr"
Rule 13: "constexpr Everything Known at Compile Time"
Rule 14: "Prefer auto in Many Cases"

Combine these two principles, and we have coined another acronym: Almost Always Const Auto (AACA). I have been using this coding style for a few years now, especially after discovering Rust. In Rust, variables are constant by default, and we must use the mut modifier to make them mutable, which the opposite of C++ with const.

This is why in the code sample from the introduction, I declare str with const auto.

auto is not relevant when decoding clang-tidy's warning message. const is. Apparently, I should not use const here, as it prevents the value from being moved when it's returned from the function.

Have I been wrong to use const so much during all these years? I'm not the only developer following the advice of the C++ legends mentioned above. For instance, see here, here, here or there.

Complete Example

Let’s write a complete example that we can run and analyze. I’ve chosen to avoid auto because I want to focus on const, and you don’t have the type overlays from my IDE showing the deduced types 😉 (of course, in real life, I would use auto).

main.cpp:

#include <fmt/std.h>

std::string build_path(const std::string& basedir, int index) {
    const std::string str = fmt::format("{}/output/{}", basedir, index);

    fmt::print("{}\n", str);
    // ... do some work with 'str'...

    return str;
}

int main() {
    const std::string dir = build_path("my/base/dir", 0);
}

This code compiles perfectly and prints my/base/dir/output/0.

We can run clang-tidy on this file:

$ clang-tidy -p cmake-build-debug --checks="clang-*,performance-*" main.cpp
106 warnings generated.
/home/pierre/cpp/main.cpp:9:9: warning: constness of 'str' prevents automatic move [performance-no-automatic-move]
        return str;
               ^
Suppressed 105 warnings (105 in non-user code).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.

According to clang-tidy, the string cannot be moved. It means it is copied, which is slower.

Because we use std::string, it's not easy to check what the compiler is actually doing. The simplest solution is to replace std::string with our own type so that we can print messages in its constructors.

With A Custom Class

Let's introduce a new class called Path.

#include <fmt/std.h>

class Path {
public:
    explicit Path(std::string path) :
        value_m{std::move(path)} {
        fmt::println("Constructor called");
    }

    Path(const Path& other) :
        value_m{other.value_m} {
        fmt::println("Copy constructor called");
    }

    Path(Path&& other) noexcept :

        value_m{std::move(other.value_m)} {
        fmt::println("Move constructor called");
    }

    [[nodiscard]] const std::string& value() const {
        return value_m;
    }

private:
    std::string value_m;
};

Path build_path(const std::string& basedir, int index) {
    const Path path = Path(fmt::format("{}/output/{}", basedir, index));

    fmt::print("{}\n", path.value());
    // ... do some work with 'path'...

    return path;
}

int main() {
    const Path dir = build_path("my/base/dir", 0);
}

This enhanced code compiles perfectly too and produces the following output:

Constructor called
my/base/dir/output/0

Great: path is not copied; it's not even moved. Otherwise, we would see the text printed from their bodies.

And yet, clang-tidy still emits the same warning on return path;.

This feels a little awkward. On the one hand, our static analyzer warns that we might hit a performance penalty; on the other hand, our compiler generates optimal code. We can use Compiler Explorer to check whether this behavior is consistent across several compilers and versions:

clang	gcc
20.1.0 ✅	15.1 ✅
8.0.0 ✅	8.1 ✅

As I write these lines, these are the latest versions and fairly old versions.

To understand what’s going on here, it’s time to introduce the third character of this story: (N)RVO.

(Named) Return Value Optimization

RVO (Return Value Optimization) and NRVO (Named Return Value Optimization) are two common optimizations that the compiler may use when returning a local variable from a function. Since C++17, RVO is even mandatory. PVS-Studio has a great post explaining how these optimizations work and what the difference is between them. People tend to use both terms interchangeably, but in some cases (like our case today), the difference is relevant.

Our code is an exact match for NRVO: we have a named local variable whose type exactly matches the return type of the function, and we return this variable. The compiler optimizes the code by neither copying nor moving the variable path. Instead, it constructs the object directly in place, into dir in the main function. Hence the program output: we don't see any messages from the copy or move constructors, because they are not called.

Note that we can use a slightly modified version of our code (removing the use of the
fmt library) with cppinsights. It tells us that path is indeed an "NRVO variable":

Can we say that clang-tidy's warning is a false positive? In this specific case, we can probably assume so.

Can we blindly disable the warning by adding -performance-no-automatic-move to our command-line or .clang-tidy file? Probably not... But we need to look at more complex cases to understand why.

NRVO is optional

This is an important point that I want to stress again: NRVO is optional. We have no guarantee that the future versions of clang or gcc will still optimize our code the same way. That's why I used the word "assume" in the previous section.

How easy it to fall into situations where the optimization is no longer applied? Let's experiment by modifying build_path() and observing how the program's output changes. Each variation can then be tested using Compiler Explorer with the same compilers as before. For each variation, I indicate whether NRVO is applied or whether a constructor (copy or move) is called.

Variation 1

We just add an early return (for which RVO will/must be applied). Try it here.

Path build_path(const std::string& basedir, int index) {
    if (index < 0) {
        return Path(basedir);
    }

    const Path path = Path(fmt::format("{}/output/{}", basedir, index));

    fmt::print("{}\n", path.value());
    // ... do some work with 'path'...

    return path;
}

clang	gcc
20.1.0 => ✅ NRVO	15.1 => ❌ copy
8.0.0 => ❌ copy	8.1 => ❌ copy

Variation 2

We don't name the variable and return temporary in both branches. Try it here.

Path build_path(const std::string& basedir, int index) {
    if (index < 0) {
        return Path(basedir);
    } else {
        return Path(fmt::format("{}/output/{}", basedir, index));
    }
}

clang	gcc
20.1.0 => ✅ NRVO	15.1 => ✅ NRVO
8.0.0 => ✅ NRVO	8.1 => ✅ NRVO

This is in fact RVO, not NRVO. This behavior is guaranteed since C++17. However, adding -std=c++17 doesn't change the outputs in Compiler Explorer. It's very likely that every decent compiler use RVO in this case, even before C++17.

Variation 3

This is the same as variation 1, except that the variable is not used locally, it's just returned. Try it here.

Path build_path(const std::string& basedir, int index) {
    if (index < 0) {
        return Path(basedir);
    } else {
        const auto path = Path(fmt::format("{}/output/{}", basedir, index));
        return path;
    }
}

clang	gcc
20.1.0 => ✅ NRVO	15.1 => ❌ copy
8.0.0 => ✅ NRVO	8.1 => ❌ copy

Variation 4

This is the same as variation 3, but path is not const . Try it here.

Path build_path(const std::string& basedir, int index) {
    if (index < 0) {
        return Path(basedir);
    } else {
        auto path = Path(fmt::format("{}/output/{}", basedir, index));
        return path;
    }
}

clang	gcc
20.1.0 => ✅ NRVO	15.1 => ⚠️ move
8.0.0 => ✅ NRVO	8.1 => ⚠️ move

Summary

clang applies NRVO more aggressively and consistently than gcc. Minor alterations of the code may prevent from applying NRVO, and we switch for an optimal situation to either a probably cheap move operation or a maybe-no-so-cheap copy operation.

The Warning Is Not About NRVO

Does clang-tidy emit a warning for all variations? No:

Variation	Warning
1	Yes ⚠️
2	No ✅
3	Yes ⚠️
4	No ✅

Maybe I blurred the lines earlier, so let me clarify: the warning is not about NRVO failing to apply. It's about the fact that a move operation is inhibited, and that a copy will be made instead. Hence its name: performance-no-automatic-move 😉

This is exactly what the previous table illustrates:

When clang-tidy emits a warning, gcc performs a copy.
When clang-tidy is silent, gcc performs a move.

This behavior is consistent with what PVS-Studio explains:

The C++11 standard says that if a compiler cannot apply an optional optimization, it must do the following. First, it must apply the move constructor. Then apply the copy constructor for local variables or formal function parameters.

const objects cannot be moved. When a variable is declared const and NRVO is not applied, moving it is impossible, so the compiler has no choice but to copy. That’s exactly what we observe in variation 3: gcc uses the copy constructor because the object is const, while it performs a move in variation 4.

(You might think that I am shaming gcc here, but clang was also doing a copy for variation 1 before version 15.0...)

How To Handle The Warning

clang-tidy cannot possibly know whether NRVO will be applied or not, so it must assume it won't. If a move operation is impossible because of const (as in variation 3), it emits a warning; otherwise, it remains silent (as in variation 4).

Let's get back to our question:

Can we blindly disable the warning by adding -performance-no-automatic-move to our command-line or .clang-tidy file?

The answer is "no, we can't". If NRVO is actually performed, it's a false positive. Otherwise, it's not, and we may want to change our code to avoid a copy in favor of a move.

In my opinion, there is no universal solution. It depends entirely on the context. Here are a few points to help decide:

If the entire project does not have strong performance requirements, and we receive many warnings, we might disable it entirely.
If the function is not performance-critical, we can just suppress the warning locally by adding // NOLINT(performance-no-automatic-move) on the same line as the return. For example, I originally discovered this warning on a function that builds the output path of the application. This function is called once at startup, and the process then runs several minutes or hours. Copying a string was acceptable (NRVO was applied anyway).
If we target specific compilers, we can manually analyze the code and/or experiment with Compiler Explorer to determine if NRVO is likely to be applied (as we did earlier in this post). If the answer is "yes", we can add // NOLINT(performance-no-automatic-move) too.
If this specific compiler is gcc 14 or newer, we can enable the -Wnrvo flag to help decide (see next section).
If performance is important, and we target any compiler, we might prefer to remove const. It's better to add a comment explaining why, otherwise someone will probably reintroduce const afterwards. This is especially true for libraries, since we have no idea how people will use them.

The great Bartłomiej Filipek from C++ Stories wrote a post "Const, Move and RVO". I could not agree more with his last piece of advice:

Still, if you’re unsure and you’re working with some larger objects (that have move enabled), it’s best to measure measure measure.

Be Warned When NRVO Is Not Possible

Released in May 2024, gcc 14 introduced a new compiler flag: -Wnrvo. From the documentation:

Warn if the compiler does not elide the copy from a local variable to the return value of a function in a context where it is allowed by [class.copy.elision]. This elision is commonly known as the Named Return Value Optimization. For instance, in the example below the compiler cannot elide copies from both v1 and v2, so it elides neither.

Jason Turner has dedicated episode 434 of his C++Weekly series to this feature.

Let's try recompiling every version of our code with this flag:

Version	Warning
Original	No ✅
Variation1	Yes ⚠️
Variation2	No ✅ => this is not NRVO anyway, it is RVO
Variation3	Yes ⚠️
Variation4	Yes ⚠️

The compiler's output looks like this:

<source>: In function 'Path build_path(const std::string&, int)':
<source>:33:24: warning: not eliding copy on return in 'Path build_path(const std::string&, int)' [-Wnrvo]
   33 |                 return path;
      |

This is great to handle clang-tidy's warning:

If both gcc and clang-tidy complains, we know we are paying for a copy instead of a move.
If gcc is silent but clang-tidy complains, adding // NOLINT(performance-no-automatic-move) is perfectly fine.
If gcc complains but clang-tidy is silent, it probably means the type is not moveable. Since performance-no-automatic-move is not applicable, clang-tidy has nothing to report. But because NRVO is not applied, a copy will be made.

Unfortunately, as of June 2025, clang does not offer an equivalent flag in any released version. However, the in-progress changelog for upcoming version (21.0.0) says:

New option -Wnrvo added and disabled by default to warn about missed NRVO opportunities.

Stay tuned.

Disabling (N)RVO with `-fno-elide-constructors`

If you are curious, you can disable (N)RVO entirely.

gcc and clang provide a flag to prevent constructor elision when it is allowed (but not required) by the standard: -fno-elide-constructors. According to gcc's documentation:

The C++ standard allows an implementation to omit creating a temporary that is only used to initialize another object of the same type. Specifying this option disables that optimization, and forces G++ to call the copy constructor in all cases. This option also causes G++ to call trivial member functions which otherwise would be expanded inline.

In C++17, the compiler is required to omit these temporaries, but this option still affects trivial member functions.

Let’s examine its effect on our original code, and a slightly modified version where path is not const, for both C++14 and C++17.

C++14

Original code:

Constructor called
Move constructor called
my/base/dir/output/0
Copy constructor called <-- There is a copy (just like `clang-tidy` warns)
Move constructor called

Original code without the `const`:

Constructor called
Move constructor called
my/base/dir/output/0
Move constructor called  <-- The copy is now a move
Move constructor called

C++17

Original code:

Constructor called
my/base/dir/output/0
Copy constructor called <-- Same copy as before

Original code without the `const`:

Constructor called
my/base/dir/output/0
Move constructor called <-- Same move as before

Apart from educational purpose or curiosity, this flag can be useful is to experiment with "the worst case scenario" of a compiler that (almost) never performs constructor elision. See this discussion on stackoverflow.

Conclusion

Whew! You're still here! 😅 This article was a real challenge to write, and probably not the easiest to read either. Thanks & kudos for making it through.

We’ve explored how NRVO behaves, how it’s affected by const, and what clang-tidy's performance-no-automatic-move] warning really means. Because NRVO (and RVO before C++17) is optional, small changes to a function can unexpectedly inhibit optimizations and introduce costly copies. This is also why having automated benchmarks in your CI is a good practice.

The warning by clang-tidy might be a false positive. Well, it's not really a false positive: actually, the code does inhibit a move; it's just that the compiler might just invalidate the warning by applying NRVO. Depending on the situation:

The warning may be irrelevant in practice (because we can ensure that NRVO is reliably applied), and we can safely suppress it locally.
We can decide that we don't want to change the code, that the performance penalty is acceptable, and we suppress it locally as well.
We change the code to guarantee optimal performance.

With this knowledge, you’re now better equipped to ensure your code behaves as expected and performs at its best. Happy optimized coding! 👋

Manually Install WiFi Drivers on Linux

Pierre Gradot — Mon, 24 Mar 2025 11:19:41 +0000

On December 24th, 2024, Debian had a gift for me: it installed Linux kernel 6.11.10, and suddenly the WiFi module of my laptop stopped working. I quickly found a workaround (selecting an older kernel with GRUB during boot) and hoped that the problem would be fixed by the next kernel update or by a firmware update from my laptop's manufacturer. Sadly, kernel 6.12 arrived and nothing changed. I finally took some time to really investigate the issue.

In this article, I will explain how I diagnosed and fixed the problem.

Spoiler alert: I downloaded and manually installed drivers from kernel.org.

The Symptoms

With kernel 6.11, the WiFi was simply absent from the GUI of Debian's settings. My laptop wasn't just unable to connect to the network, the WiFi was simply gone! With kernel 6.12, the entry was present in the GUI, but only to tell me that no network adapter was available.

The time to open a terminal and investigate had come.

Several commands can help determine the state of the WiFi adapter, such as lshw -C network or lspci. On my laptop, the latter showed:

$ lspci -k
...
00:14.3 Network controller: Intel Corporation Raptor Lake PCH CNVi WiFi (rev 01)
    Subsystem: Intel Corporation Raptor Lake PCH CNVi WiFi
    Kernel modules: iwlwifi
...

The -k option shows "kernel drivers handling each device and also kernel modules capable of handling it" according to the man pages. The network controller was detected, but something was clearly wrong: unlike other devices, there was no line saying "Kernel driver in use: [name-of-the-driver]".

The kernel's log provides detailed information about modules and drivers being loaded. By inspecting the log and searching for the module indicated by lspci, I noticed that the module was indeed not properly loaded:

dmesg | grep -i iwlwifi
[  264.172078] iwlwifi 0000:00:14.3: Detected crf-id 0x400410, cnv-id 0x80400 wfpm id 0x80000020
[  264.172136] iwlwifi 0000:00:14.3: PCI dev 51f1/0090, rev=0x370, rfid=0x2010d000
[  264.172137] iwlwifi 0000:00:14.3: Detected Intel(R) Wi-Fi 6E AX211 160MHz
[  264.172742] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-89.ucode (-2)
[  264.172747] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-89.ucode (-2)
[  264.172748] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-89.ucode failed with error -2
[  264.172754] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-88.ucode (-2)
[  264.172757] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-88.ucode (-2)
[  264.172758] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-88.ucode failed with error -2
[  264.172761] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-87.ucode (-2)
[  264.172764] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-87.ucode (-2)
[  264.172765] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-87.ucode failed with error -2
[  264.172769] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-86.ucode (-2)
[  264.172771] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-86.ucode (-2)
[  264.172772] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-86.ucode failed with error -2
[  264.172775] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-85.ucode (-2)
[  264.172778] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-85.ucode (-2)
[  264.172779] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-85.ucode failed with error -2
[  264.172782] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-84.ucode (-2)
[  264.172785] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-84.ucode (-2)
[  264.172786] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-84.ucode failed with error -2
[  264.172789] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-83.ucode (-2)
[  264.172792] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-83.ucode (-2)
[  264.172792] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-83.ucode failed with error -2
[  264.172796] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-82.ucode (-2)
[  264.172799] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-82.ucode (-2)
[  264.172800] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-82.ucode failed with error -2
[  264.172803] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-81.ucode (-2)
[  264.172806] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-81.ucode (-2)
[  264.172807] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-81.ucode failed with error -2
[  264.172810] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-80.ucode (-2)
[  264.172813] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-80.ucode (-2)
[  264.172814] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-80.ucode failed with error -2
[  264.172817] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-79.ucode (-2)
[  264.172820] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-79.ucode (-2)
[  264.172821] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-79.ucode failed with error -2
[  264.172825] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-78.ucode (-2)
[  264.172827] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-78.ucode (-2)
[  264.172828] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-78.ucode failed with error -2
[  264.172832] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-77.ucode (-2)
[  264.172834] iwlwifi 0000:00:14.3: firmware: failed to load iwlwifi-so-a0-gf-a0-77.ucode (-2)
[  264.172835] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-so-a0-gf-a0-77.ucode failed with error -2
[  264.172836] iwlwifi 0000:00:14.3: no suitable firmware found!
[  264.172838] iwlwifi 0000:00:14.3: minimum version required: iwlwifi-so-a0-gf-a0-77
[  264.172839] iwlwifi 0000:00:14.3: maximum version supported: iwlwifi-so-a0-gf-a0-89
[  264.172839] iwlwifi 0000:00:14.3: check git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git

This output also confirmed the model of the WiFi controller from the website of my laptop's manufacturer: it's an Intel® Wi-Fi 6E AX211.

As mentioned before, the WiFi controller was still working fine with kernel 6.10. The same dmesg command was showing similar errors, but the kernel was trying to load more modules. Eventually, it succeeded in loading version 72:

[   17.290059] iwlwifi 0000:00:14.3: loaded firmware version 72.daa05125.0 so-a0-gf-a0-72.ucode op_mode iwlmvm

The Solution

Intel's website states that this controller is supported by Linux. It even mentions that it is supported by kernel 6.11+.

A discussion thread from Intel's forum was helpful: you can manually download drivers from kernel.org and install them by simply copying them to /lib/firmware.

What was already installed on my laptop? I listed the files matching the pattern from dmesg with ll /lib/firmware/iwlwifi-so-a0-gf-*.ucode and found only one result: /lib/firmware/iwlwifi-so-a0-gf-a0-72.ucode. The dmesg output had mentioned that "minimum version required: iwlwifi-so-a0-gf-a0-77" and "maximum version supported: iwlwifi-so-a0-gf-a0-89". Conclusion: the driver was failing to load not because it was invalid, but simply because it was missing a version in the required range.

I knew the files I needed, so I headed to kernel.org and downloaded both versions 77 and 89, which are the minimum required and maximum supported versions for the kernel I was trying to load. In reality, only version 89 would have been sufficient.

I simply copied these files to /lib/firmware, rebooted my computer, and... the WiFi was back! :)

Now, dmesg confirms that a driver is correctly loaded with [ 183.837486] iwlwifi 0000:00:14.3: loaded firmware version 89.1a492d28.0 so-a0-gf-a0-89.ucode op_mode iwlmvm and the output of lspci shows that there a kernel driver in use (and yes, this is very significant difference):

$ lspci -k -s 14.3
00:14.3 Network controller: Intel Corporation Raptor Lake PCH CNVi WiFi (rev 01)
        Subsystem: Intel Corporation Raptor Lake PCH CNVi WiFi
        Kernel driver in use: iwlwifi
        Kernel modules: iwlwifi

Conclusion

Sometimes, newer kernels don't come with the required drivers. In my case, it was the WiFi driver, but it can the driver for types of other devices.

If you find yourself in a similar situation:

Identify the exact model of your device.
Check kernel logs to find error messages to determine what files are missing.
Manually download and install the required driver.
Reboot your machine and hope for the best.

PS: there is a package named firmware-iwlwifi in Debian non-free repository. Installing should provide automatic updates for Intel WiFi firmware in the future. However, this package seems clearly outdated (which is not a big surprise, as Debian is sometimes slow to update its packages):

 % sudo apt install firmware-iwlwifi
[...]]
firmware-iwlwifi is already the newest version (20230210-5).
0 upgraded, 0 newly installed, 0 to remove and 86 not upgraded.

Oh, by the way: iwlwifi probably stands for "Intel Wire*L*ess WiFI".

Resizing Partitions on Linux

Pierre Gradot — Thu, 29 Aug 2024 09:33:25 +0000

Sometimes, you need to resize your partitions on Linux. And this may not be as easy as you think...

When I installed Debian on my work computer a few months ago, the wizard suggested creating 2 separate partitions: one for root (with the OS itself), another for home (with my personal data). Alas! The suggested sizes (that I naively accepted) were highly unsuitable and a month later, the root partition (only 40 Go) was full. I had to resize them. It turns out that now, 5 months later, the home is full (I really generate a lot of data for my current project). It's time to resize my partitions again but this time I will keep track of the entire procedure in this article. Who knows when I will have to resize them one more time...

Current State and Goal

My computer as a single encrypted SSD disk:

 % lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
nvme0n1                 259:0    0  3.6T  0 disk 
├─nvme0n1p1             259:1    0  512M  0 part  /boot/efi
├─nvme0n1p2             259:2    0  488M  0 part  /boot
└─nvme0n1p3             259:3    0  3.6T  0 part 
  └─nvme0n1p3_crypt     253:0    0  3.6T  0 crypt 
    ├─debian--vg-root   253:1    0  2.6T  0 lvm   /
    ├─debian--vg-swap_1 253:2    0  976M  0 lvm   [SWAP]
    └─debian--vg-home   253:3    0    1T  0 lvm   /home

fdisk -l shows similar information.

As I said, my home partition is full. However, the root partition is almost empty:

 % df -h | grep debian
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/debian--vg-root  2.6T  202G  2.3T   8% /
/dev/mapper/debian--vg-home 1007G  937G   19G  99% /home

My goal is to resize the partitions so that root is 1T and home occupies the rest of the disk. This will be a 2-step process: first, shrink root ; second, extend home.

Apparently, I can't do this from the graphical disk utility (note "Edit Partition..." and "Resize..." being greyed out):

It's time to open a terminal and input some weird commands!

Shrinking Root

Because I want to resize the root partition, it seems that I can't directly work from my installed Debian. I put my hands back on my bootable USB drive (the one I used to install Debian in the first place) and I reboot my computer on it. It doesn't provide a live Debian mode, so I have to use the rescue mode to get a shell:

Yes, I took pictures on my laptop's screen.

Once I have configured my language and keyboard layout, and skipped the network configuration, I can finally enter the rescue mode for real:

I have to enter my passphrase because my disk is encrypted, and I now have a shell. Hooray!

Resizing a "partition" is not a single-step operation. Indeed, there are in fact a logical volume and a filesystem on top of it. Both need to be resized in the correct order.

First, reduce the filesystem with the resize2fs command. In this first step, I use a temporary size, not the desired size. It must be big enough to hold all the files but smaller that the desired size. The output df -h above shows that 202G are used on root. The desired size is 1T. Hence, 300G seems fine.

resize2fs /dev/mapper/debian--vg-root 300G

I am asked to check the filesystem first with e2fsck -f /dev/mapper/debian--vg-home. Once the check and possible fixes are done, I can call resize2fs again.

Second, reduce the volume to the desired size (1T) with the lvreduce command:

lvreduce -L 1T /dev/vgroot/lvroot

Third (and last), extend the filesystem to use the entire volume. This is done by passing only the volume to resize2fs because if the "size parameter is not specified, it will default to the size of the partition".

resize2fs /dev/vgroot/lvroot

This is how it looks like in real life:

Note that step 3 may be performed directly in step 2, because lvreduce has an option that seems to do it:

-r, --resizefs
Resize underlying filesystem together with the logical volume using fsadm.

To be honest, I wasn't really keen to try on my work computer, I can't afford any data loss. So I stuck a procedure that I knew to work, but I encourage you to test it 😅

Checking Root

Now, I shut down my computer, I remove the USB drive, I reboot to my actual system, and I'm glad to see that it still works 😄

I can check the state of my disk:

 % df -h | grep mapper
/dev/mapper/debian--vg-root 1008G  202G  756G  22% /
/dev/mapper/debian--vg-home 1007G  937G   19G  99% /home

 % lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
nvme0n1                 259:0    0  3.6T  0 disk  
├─nvme0n1p1             259:1    0  512M  0 part  /boot/efi
├─nvme0n1p2             259:2    0  488M  0 part  /boot
└─nvme0n1p3             259:3    0  3.6T  0 part  
  └─nvme0n1p3_crypt     253:0    0  3.6T  0 crypt 
    ├─debian--vg-root   253:1    0    1T  0 lvm   /
    ├─debian--vg-swap_1 253:2    0  976M  0 lvm   [SWAP]
    └─debian--vg-home   253:3    0    1T  0 lvm   /home

The root partition is fine. Of course, the home partition is unchanged. Let's resize it now.

Extending Home

This time, I don't need to boot on my USB drive.

First, extend the volume with the lvextend command. I don't have to calculate the remaining size, I just ask to add 100% of the free space of the disk to the volume:

 % sudo lvextend -l +100%FREE /dev/mapper/debian--vg-home      
  Size of logical volume debian-vg/home changed from 1.00 TiB (262144 extents) to <2.64 TiB (691219 extents).
  Logical volume debian-vg/home successfully resized.

Note that the option is -l (lowercase) not -L (uppercase), unlike with lvreduce.

Second (and last), resize the filesystem to use the entire volume (same as last operation for root):

 % sudo resize2fs /dev/mapper/debian--vg-home
resize2fs 1.47.0 (5-Feb-2023)
Filesystem at /dev/mapper/debian--vg-home is mounted on /home; on-line resizing required
old_desc_blocks = 128, new_desc_blocks = 338
The filesystem on /dev/mapper/debian--vg-home is now 707808256 (4k) blocks long.

Checking Home

Finally, I have reached my goal:

 % df -h | grep mapper                       
/dev/mapper/debian--vg-root 1008G  202G  756G  22% /
/dev/mapper/debian--vg-home  2.6T  937G  1.6T  37% /home

 % lsblk                                           
NAME                    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
nvme0n1                 259:0    0  3.6T  0 disk  
├─nvme0n1p1             259:1    0  512M  0 part  /boot/efi
├─nvme0n1p2             259:2    0  488M  0 part  /boot
└─nvme0n1p3             259:3    0  3.6T  0 part  
  └─nvme0n1p3_crypt     253:0    0  3.6T  0 crypt 
    ├─debian--vg-root   253:1    0    1T  0 lvm   /
    ├─debian--vg-swap_1 253:2    0  976M  0 lvm   [SWAP]
    └─debian--vg-home   253:3    0  2.6T  0 lvm   /home

Conclusion

I have to be honest, it was quite scary to resize partitions 😱 Of course, I backed up my important data, but I can't really back up a full system with all the installed tools and settings I have as a software developer on my machine.

At the end of the day, I left this uncomfortable situation:

Before:

 % df -h | grep mapper       
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/debian--vg-root  2.6T  202G  2.3T   8% /
/dev/mapper/debian--vg-home 1007G  937G   19G  99% /home

And I have reached with much more cozy one:

 % df -h | grep mapper                       
/dev/mapper/debian--vg-root 1008G  202G  756G  22% /
/dev/mapper/debian--vg-home  2.6T  937G  1.6T  37% /home

I hope this was the last time I had to resize my partitions on this computer! 😁

CMake on SMT32 | Episode 8: build with Docker

Pierre Gradot — Mon, 18 Mar 2024 10:57:35 +0000

In this episode, we will use Docker to set up and manage our build environment.

The embedded world doesn't always embrace the latest software development techniques. I acknowledged this in the previous episode about unit tests. Four years ago, when I started this series, I was totally unaware of Docker for instance. Since then, I have used it for non-embedded projects and I think it could have solved some issues I encountered in the past. I would like to share my experience and demonstrate that it can perfectly fit in an MCU project.

In fact, every I will say here applies to hosted C / C++ projects too!

Dependencies to build our project

Let's take a moment to look back and review all the tools required on our machine to build the project presented in this series:

cmake: obviously, this is the quintessence of this series.
make: in the first episode, we chose to use -G "MinGW Makefiles" as our generator. We may have chosen another generator, for instance -G Ninja, and have another dependency (ninja, namely).
arm-none-eabi-gcc: this is the toolchain to compile our embedded code.
gcc: we need another toolchain for the unit tests to run on the host computer.
git: CMake's FetchContent module needs Git to fetch Catch2 from GitHub.

The list may seem relatively small for the moment, but for sure it will grow in the future. We will probably want to:

write scripts with python and need pip to install packages.
debug the units tests with gdb.
analyze our code with clang-tidy and format it with clang-format.

Each developer on the project needs to install everything on their machine. Each time the team decides to upgrade a tool to a newer version or to use a new tool, everyone must update their setup. Not to mention the CI environment (if you have maintained a Jenkins machine once in your life, you know what I mean).

Should I say that everyone may have a different version of everything on their machine? Things could turn into a nightmare.

This is where Docker comes into play to help tackle this complexity. The purpose is to create a Docker image with everything the team needs to work. Instead of having to install many applications, we just need Docker (and obviously a VCS client, probably Git, to clone the central repository).

This article is not meant to be a crash course to Docker. If you don't have the basic knowledge, you should probably read a tutorial. This may be optional though, as you will probably get the concepts and the process I will describe anyway.

Dockerfile

We create a Dockerfile (yes, that the name of the file) next to our CMakeLists.txt, at the root of our project. We use the latest image of Ubuntu as the base image, and we install the packages we need. Here is the content of the file:

FROM ubuntu:latest

RUN apt-get update && apt-get install -y \
                        build-essential \
                        cmake \
                        gcc-arm-none-eabi \
                        gdb \
                        git

The build-essential package includes gcc and make. Note that I have included gdb, which isn't yet mandatory, but CLion asks for it (and we will see that this Docker image can be used seamlessly within CLion).

And... that's it! We have everything we need! We will see later how we can improve this file by using explicit versions. For the moment, let's try to build our firmware and our unit tests!

Build the Docker image

This Dockerfile describes our image, and we have to build it now. We just need to execute:

docker build -t cmake-on-stm32 .

-t is short for --tag. It gives the image a name (in fact, a tag), and this will make our life easier in the next commands.

If we list the images, we see our image and its base image:

$ docker images
REPOSITORY        TAG       IMAGE ID       CREATED          SIZE
cmake-on-stm32    latest    1ed7f1257600   48 minutes ago   3.12GB
ubuntu            latest    e34e831650c1   5 weeks ago      77.9MB

The image can now be used to run containers, which are the actual runners for this Dockerized build environment.

Using the container from the terminal

We have two options:

Get a shell inside the container and work inside the container.
Ask the container to run commands and work from the host.

Get a shell inside the container

For the first option, we just have to run the following command to get a shell inside the container:

docker run --rm -it -v $(pwd)/:/nucleo/ -w /nucleo/ cmake-on-stm32

Let's break down the options:

--rm requires Docker to remove the container when it exits.
-it is short for --interactive + --tty. This is the option that takes us straight inside the container.
-v to mount the current directory as /nucleo/ in the container. Indeed, the container is isolated from the host. If we want to share directories between the host and the container, volumes are the way to go.
-w is optional, it just sets the working directory so that the shell starts in a handy location.

NOTE: if you always use the same working directory, you can set it directly in the Dockerfile with the WORKDIR instruction.

To demonstrate the usage:

Notice how the prompt changes and how the arm-none-eabi-gcc is found or not. exit exits the shell and hence the container.

Now, we can just follow the process that has been described throughout this series: generate the project, build the firmware, build the tests, run the tests, etc. The commands are the same, we just execute them inside the container.

Execute commands from the host

We can remove the -it option and add more arguments to the command. The container will execute them. Here is an example with ls and arm-none-eabi-gcc --version:

I tend to prefer the first technique, because the commands can get very long with this second technique. Furthermore, the host can't auto-complete the commands because it doesn't know what the container supports. Nevertheless, I guess in some cases, mixing both techniques can be interesting. It's up to you to define how you want to interact with the container.

Simplify the command-line with Docker Compose

The above commands may seem a bit "heavy", mostly because of the options to mount the volume and to set the working directory. It's likely that more options will be added, making the commands even longer.

We can take advantage of Docker Compose to simplify things a little. compose is a subcommand of docker, there is no new software to install on the host machine. We can write a simple docker-compose.yml file to describe what we want:

services:
  cmake-on-stm32:
    build: .
    volumes:
      - .:/nucleo/
    working_dir: /nucleo

We can now use the docker compose run command (instead of just docker run) and omit the options that are described in docker-compose.yml:

Note that the -it option is not required to use the container interactively with docker compose.

Using the container from CLion

In episode 2, we learned how to build our CMake project for STM32 with CLion by Jetbrains. I have good news: CLion supports Docker toolchains out-of-the-box! I tried and it worked perfectly in minutes:

I added a new Docker toolchain that uses cmake-on-stm32:latest as its image.
I created 2 profiles to build the project:
1. For the units tests: I used all default values except for the toolchain where I selected my Docker toolchain.
2. For the embedded code: same except that I added -DCMAKE_TOOLCHAIN_FILE=arm-none-eabi-gcc.cmake in the CMake options field.

And voilà! We can then work within CLion without noticing that Docker is involved.

Improve user management

There is a little catch with the commands above: inside the container, the user is root. Consequence: all files and directories created from the container are owned by root. For instance when building in build-docker:

Most of the time, this is not a real issue, except when we want to delete them because we don't have the appropriate permissions. On Linux, it means we will need sudo to perform the deletion.

To get around this issue, we can add the --user option to specify the user and group we want when calling docker run. For instance on Linux, --user $(id -u):$(id -g) will use our user and group. We will then be the owner of the generated files and directories:

The user and group can also be set in Docker Compose. Read "How to set user and group in Docker Compose" by Giovanni De Mizio to learn how to do this.

Note that CLion does this in background, so the ownership of the files is correct without any further options.

Pin versions of dependencies

In our Dockerfile, we have simply chosen the latest Ubuntu version as the base image (with FROM ubuntu:latest) and have blissfully installed the packages. This is cool when testing things but there is one major drawback: we have absolutely no idea what versions of the tools we will get!

Pin the base image

As of February 29th 2024 (as I'm writing these lines), Docker Hub says that the ubuntu:latest is Jammy Jellyfish (aka 22.04, the current LTS). In April 2024 Noble Numbat (aka 24.04, the next LTS release) will be out and will probably become the new latest version. It will come (almost for sure) with new versions of the tools. Who knows if our code will still compile under Noble Numbat? We really don't want our project to suddenly stop building because a new release of Ubuntu is out.

The obvious first counter-measure is to choose a version of Ubuntu explicitly. We just have to replace ubuntu:latest with ubuntu:jammy or any other version. Pinning the base image will also (normally) pin the versions of the tools because each release of Ubuntu is quite conservative about the versions of the programs in the packages (Debian is even more). Ubuntu Packages Search website tells us the content of the packages we can install with apt-get. Thanks to this knowledge, we can decide which Ubuntu version is good for us, if we want to upgrade to the next version or not, etc. For instance with gcc:

Ubuntu	`gcc`	`gcc-arm-none-eabi`
Jammy Jellyfish	11.2	10.3
Lunar Lobster	12.2	12.2
Mantic Minotaur	13.2	12.2

Select more specific packages

On Mantic Minotaur (23.10, the latest version of Ubuntu as I write these lines), the gcc-arm-none-eabi package installs GCC 12.2 (for ARM), while the gcc package installs GCC 13.2 (for x86-64). We would like our hosted and embedded toolchains to use the same version GCC (at least the same major version). There is a solution: the gcc-12 packages installs GCC 12.3. That is a better match with the embedded toolchain. Note that there are packages for gcc-9 up to gcc-13 on Mantic Minotaur.

A better Dockerfile would probably be:

FROM ubuntu:mantic

RUN apt-get update && apt-get install -y \
                        make \
                        cmake \
                        gcc-12 \
                        gcc-arm-none-eabi \
                        gdb \
                        git

RUN ln -s -f gcc-12 /usr/bin/gcc

I should probably explain the link creation. The cmake package has gcc in its list of recommended packages. Because we haven't used the option --no-install-recommends, it will install GCC 13 and create a link /usr/bin/gcc that points to /usr/bin/gcc-13 (which is itself a link). The last line of the Dockerfile is here to overwrite this link. Oh! by the way: the gcc-12 package won't create the link anyway, so we really need this line.

Select exact versions of the packages

As I said, each release of Ubuntu is quite conservative. It's very unlikely that the version of GCC in the gcc-arm-none-eabi in the repositories will change. Unlikely doesn't mean impossible (it could be a patch version). In case we really want to be sure to stick to an exact version, we can add the version of the package in the apt-get command and use gcc-arm-none-eabi=15:12.2.rel1-1 instead of just gcc-arm-none-eabi (with Mantic Minotaur). The version string can be found in the title of package's page for each release of Ubuntu.

Manual installations

It's possible that the Ubuntu packages won't provide with the versions we want. We can them fallback to manual installations. Let's say to we want the latest release of CMake (which is 3.29.0-rc2 at the moment), we can script the installation from GitHub in the Dockerfile.

Here is our new Dockerfile:

FROM ubuntu:mantic

RUN apt-get update && apt-get install -y \
                        make \
                        gcc-12 \
                        gcc-arm-none-eabi=15:12.2.rel1-1 \
                        gdb \
                        git \
                        wget

RUN ln -s -f gcc-12 /usr/bin/gcc

RUN wget https://github.com/Kitware/CMake/releases/download/v3.29.0-rc2/cmake-3.29.0-rc2-linux-x86_64.sh
RUN chmod u+x cmake-3.29.0-rc2-linux-x86_64.sh
RUN ./cmake-3.29.0-rc2-linux-x86_64.sh --skip-license --include-subdir
ENV PATH="${PATH}:/cmake-3.29.0-rc2-linux-x86_64/bin"

We update the PATH environment variable so that cmake (and its associated binaries, such as ctest) is available from the path.

Assessing our solution

In my opinion, this Docker-centered approach has a lot of benefits:

It's much easier to set our computer when we join a new project. Install Git (and SVN or whatever we need), install Docker, clone the repository, and we're good to build!
Everyone in the team has the same environment (including Jenkins or whatever CI service we use), assuming that everyone rebuilds the Docker image regularly.
It is very easy to try a new version of a compiler (or any other tool) without smashing our current environment. We create new branch in our VCS, we update the Dockerfile to fetch the new compiler and we can work. We can then seamlessly switch back to our original branch and have our "normal" compiler back.
On the same computer, we can work on several projects that use different versions of the same tool, without conflicts (eg: project A uses GCC 12 while project B uses GCC 13) thanks to the isolation that Docker offers.
It's easier to rebuild old versions of the project. When we check out an old commit, we can rebuild a Docker image to get the tools that the project required back then, and then rebuild the project with it.

Of course, there is no magic in real life and every solution comes with some downsides:

We now have yet another tool in our environment. We have to learn it, we will run into Docker-specific issues, we have to maintain Docker code.
It may be difficult to integrate some tools in the image if their installations aren't scriptable.
We can't force people to regularly rebuild their local Docker image, so there may still be inconsistency among the team's machines.
We never know if our docker build command will work tomorrow. We rely on the Docker registry to download the Ubuntu images, on the Ubuntu repository to download the compilers and so on. It's not really a daily issue for the HEAD of an active repository. However, we have no guarantee that a given Dockerfile we will still build in X months. That's why I have written "it's easier to rebuild old versions of the project", instead of "you can rebuild any old versions of the project".

In this article, I have showed one way to use Docker. The repository contains a Dockerfile to create the build environment from scratch, making the repository somehow able to build itself. This is a nice approach because the repository is self-sufficient.

We can proceed differently to mitigate the last two downsides. For instance, we can manage the creations of the images separately from the project(s) that will use them.

Treat Docker images as a separate project

Creating build environments can be considered as a separate project.

On the one hand, we create images with revisions. Instead of always tagging the images as "latest" with docker build -t cmake-on-stm32 ., we use explicit version numbers with docker build -t cmake-on-stm32:0.1 ., docker build -t cmake-on-stm32:0.2 ., etc. Each version of the image corresponds to a set of tools at known versions. If we need a new version of CMake (for instance), we update the Dockerfile and release image 0.3.

On the other hand, we fetch the exact Docker image we need in our STM32-based project. We can create build scripts to wrap the commands shown in this article to force people to use the correct Docker image at any revision of the repository.

Sharing the Docker images can be manual, using docker image save + docker image load + a shared drive. The images can also be pushed to the DockerHub or self-hosted registry with docker image push.

Conclusion

In this episode, we have seen how we can leverage Docker to seamlessly set up and manage our build environment. Instead of having to install several applications on our computer, we now only need Docker to build the project. We have also seen there are several ways to include Docker in the process. It's up to you to find how you want to use it in your project. I really wished I could go back in time and try Docker on some of my previous projects! Tell me in the comments how you use/would like to use Docker for your projects!

CMake on SMT32 | Episode 7: unit tests

Pierre Gradot — Thu, 15 Feb 2024 12:00:00 +0000

It's been almost four years, and it's about time to revive this series on CMake and STM32 (or MCUs in general, STM32 just serves as a tangible example).

Funny enough, I'm not working with microcontrollers anymore.... This is a very recent change in my professional career, even if I think I will probably work again with MCUs sooner or later. However, I still use CMake to build C++ projects! The real reason to resume this series is my desire to write an article Docker for builds. In 2020, I had never used Docker at all. Today, I use it very often, and I'm convinced it could have solved some build and environment issues back then.

But hey, you might be wondering why I'm talking about Docker. Isn't this article supposed to be about unit tests? Yep, it is. I originally planned to cover unit tests back in 2020, but things didn't pan out. It seems logical to write about this topic first. It will give us even more reasons to use Docker (subscribe so you don't miss the next episode)!

Why unit tests?

I don't want to get into a lengthy debate about pros & cons about unit tests. Sure, they can be misused (so badly), but I believe they are a very powerful tool to improve software quality. There are tons of great articles on this topic out there if you want to if you want to delve into this topic. If you speak (or at least understand) French, I was invited on a podcast to talk about unit tests, and I explained why I believe they are a crucial part of software development.

After 13+ years as a software developer, mostly working projects with MCUs (and especially STM32), I must say that unit tests aren't very common in the embedded world. Embedded projects sometimes feel more like electronics-with-some-code projects than modern software projects. Hardware can be a real constraint too, so we tend to think that unit tests aren't for us.

In this article, I want to show that it doesn't have to be necessarily that way. With CMake, we can easily build unit tests for our embedded projects.

Running tests on board

Since we're dealing with electronic devices, our first thought is naturally to run tests directly on the board.

Running unit tests on board is great, because the same compiler and processor are used for both the tests and the real application. The tests are very close to the real usage of the code.

To do this, we will need to create an additional firmware dedicated solely to running tests and generating a test report. This means we have to add another add_executable() in your CMakeLists.txt. Similar to the real application's target, we have call target_compile_definitions(), target_include_directories(), target_link_options(), and so on, to generate a suitable firmware for the board. This process is not different from what we covered in the previous episodes.

However, running tests on board presents several challenges:

A test framework that compiles for the target must be found.
Tests may need to be split into several executables so that each one fits within the flash memory of the MCU.
The standard output of your board must be captured to get the test report.
Running tests takes time, simply because you have to program the board.
Integration in CI pipeline may be problematic because it may not be feasible to connect a board to the server.

While these challenges are not insurmountable, the following section will explore the advantages of running tests on your computer, eliminating these constraints and offering additional benefits.

If you want to learn more about embedded tests, the great MCU on Eclipse blog has recently published an article on this topic.

Running tests on your computer

Instead of running the tests directly on board, an alternative approach is to compile them using a compiler for PC and execute them on our computer. This eliminates the constraints mentioned earlier.

You may argue that we are using different compiler, and this won't accurately reflect of the real usage of the code. And in some ways, you're absolutely right. Indeed, even something as "basic" as the size of an integer could differ between arm-none-eabi-gcc and regular gcc.

But that's actually the silver lining! Compiling our project with another compiler may raise new warnings, highlighting non-portable constructs. Running on a different CPU may reveal undefined behaviors in our code. Finally, being able to compile a significant part of the project for another target demonstrates that the abstraction from the hardware layers are well-designed. Naturally, this also implies that we won't be able to test the parts that are tightly coupled to hardware features.

In the remainder of this article, I will demonstrate how to use CMake to build an embedded firmware and PC-based tests within the same CMakeLists.txt.

Project structure

The structure of the project is as follows:

Most elements are simply the result of the previous episodes. I have just added a directory (tests) and two files (compute.hpp and test_compute.cpp), because... we need something to test!

A simple test case

The purpose here is to demonstrate some CMake features, don't expect amazing code.

First, we need something to test. For instance a simple quadratic function implementation. This is a good candidate for PC-based tests, because it doesn't have any dependencies to the BSP. The code goes into compute.hpp:

namespace compute {

template <typename T>
T quadratic(T a, T b, T c, T x) {
    return a * x * x + b * x + c;
}

}

The tests are written in a separate file, test_compute.cpp. For this example, I have decided a use Catch2, a great unit testing framework for C++ that I have been using for the last 5 years.

#include <catch2/catch_test_macros.hpp>

#include "compute.hpp"

TEST_CASE("Compute quadratic function") {
    int a = 2;
    int b = 1;
    int c = 10;

    CHECK(compute::quadratic(a, b, c, 0) == 10);
    CHECK(compute::quadratic(a, b, c, 1) == 13);
    CHECK(compute::quadratic(a, b, c, 2) == 20);
    CHECK(compute::quadratic(a, b, c, -3) == 25);
}

True story: I found that my function had a bug when I wrote these tests. I had mistakenly written a * a * x instead of a * x * x in the implementation of compute::quadratic().

Create the executable for the tests

Similar to adding tests for on-board execution, we have to add another add_executable() in our CMakeLists.txt. However, things are sightly different: we need to handle two different compilers now. Fortunately, CMake makes it straightforward to determine whether we are cross-compiling (to create our STM32-based firmware) or not (to create our PC-based program), thanks to the variable CMAKE_CROSSCOMPILING. We will see in the next section that this variable is set when we provide CMake with a toolchain file.

The structure of our CMakeLists.txt will look like this:

cmake_minimum_required(VERSION 3.15.3)

project(nucleo-f413zh)

enable_language(C CXX ASM)
set(CMAKE_C_STANDARD 99)
set(CMAKE_C_STANDARD_REQUIRED ON)
set(CMAKE_C_EXTENSIONS OFF)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

if(CMAKE_CROSSCOMPILING)
        message(STATUS "Cross compiling for board...")

        # Here goes the code from the previous episodes

else()
        message(STATUS "Compiling for PC...")

        # Here goes the code to build the unit tests
        # See below :)
endif()

Catch2 is available on GitHub. You can get a copy of its source code and integrate it into your project source tree. You can also use FetchContent module to get it from GitHub directly during the build process, as suggested in Catch2's official documentation. We will use this technique here.

Let's complete the else() clause from the previous snippet to create our executable:

message(STATUS "Compiling for PC...")

# Get Catch2
include(FetchContent)

FetchContent_Declare(
        Catch2
        GIT_REPOSITORY https://github.com/catchorg/Catch2.git
        GIT_TAG v3.5.2
)

FetchContent_MakeAvailable(Catch2)

# Create executable
add_executable(tests
        tests/test_compute.cpp)

target_include_directories(tests PRIVATE
        source)

target_link_libraries(tests PRIVATE
        Catch2::Catch2WithMain)

Build and run the tests

As mentioned before, passing a toolchain file to CMake from the command-line automatically set CMAKE_CROSSCOMPILING. We will hence have to generate two "projects" from a single CMakeLists.txt.

On one side, we generate a project for the arm-none-eabi-gcc toolchain, just like in the first episode of this series. From the nucleo directory:

$ cmake -B build-embedded -DCMAKE_BUILD_TYPE=Debug -DCMAKE_TOOLCHAIN_FILE=arm-none-eabi-gcc.cmake
-- The C compiler identification is GNU 10.3.1
-- The CXX compiler identification is GNU 10.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/arm-none-eabi-gcc - skipped
...
-- Cross compiling for board...
...

$ cmake --build build-embedded/
Scanning dependencies of target nucleo-f413zh.out
[  3%] Building C object CMakeFiles/nucleo-f413zh.out.dir/BSP/Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_tim.c.obj
[  7%] Building C object CMakeFiles/nucleo-f413zh.out.dir/BSP/Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_tim_ex.c.obj
[ 11%] Building C object CMakeFiles/nucleo-f413zh.out.dir/BSP/Drivers/STM32F4xx_HAL_Driver/Src/stm32f4xx_hal_uart.c.obj
...
[100%] Linking CXX executable nucleo-f413zh.out
   text    data     bss     dec     hex filename
   7868      20    2892   10780    2a1c nucleo-f413zh.out
[100%] Built target nucleo-f413zh.out

On the other side, we generate another project to build with our default system compiler:

$ cmake -B build-tests -DCMAKE_BUILD_TYPE=Release                                             
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
...
-- Compiling for PC...
...

$ cmake --build build-tests/
[  0%] Building CXX object _deps/catch2-build/src/CMakeFiles/Catch2.dir/catch2/benchmark/catch_chronometer.cpp.o
[  1%] Building CXX object _deps/catch2-build/src/CMakeFiles/Catch2.dir/catch2/benchmark/detail/catch_analyse.cpp.o
[  2%] Building CXX object _deps/catch2-build/src/CMakeFiles/Catch2.dir/catch2/benchmark/detail/catch_benchmark_function.cpp.o
[  3%] Building CXX object _deps/catch2-build/src/CMakeFiles/Catch2.dir/catch2/benchmark/detail/catch_run_for_at_least.cpp.o
...
 96%] Built target Catch2
[ 97%] Building CXX object _deps/catch2-build/src/CMakeFiles/Catch2WithMain.dir/catch2/internal/catch_main.cpp.o
[ 98%] Linking CXX static library libCatch2Main.a
[ 98%] Built target Catch2WithMain
[ 99%] Building CXX object CMakeFiles/tests.dir/tests/test_compute.cpp.o
[100%] Linking CXX executable tests
[100%] Built target tests

Because our executable depends on Catch2, CMake builds the library before.

Our tests are now ready to run:

$ ls build-tests/
CMakeCache.txt  CMakeFiles  Makefile  _deps  cmake_install.cmake  tests

$ ./build-tests/tests 
Randomness seeded to: 179865564
===============================================================================
All tests passed (4 assertions in 1 test case)

Hurray! The tests passed!

Conclusion

In this episode, we have discussed unit tests for embedded projects.

We have seen that running tests on board is great, but there are some difficulties to overcome. Running tests on PC don't have these difficulties and may even offer additional benefits. If you have the chance to do both, do it!

We've seen how to manage both an STM32-based firmware and a PC-based program within the same CMakeLists.txt. We generate two separate build directories, one for each compiler and hence one for each executable.

I hope that this episode convinced you to add unit tests to your STM32 (or any other MCU) projects. Happy tests!

vtable under the surface | Episode 3 - How virtual functions are actually called

Pierre Gradot — Tue, 23 Jan 2024 15:57:43 +0000

In this episode, we will see how invoking a virtual function in C++ translates into assembly instructions. We will see how our class instance is constructed and how it relates to the vtable. Then, we will see how this vtable is used to call the appropriate function.

In you have actually built the project and analyzed the binary in the previous episode, don't forget to remove the -fno-rtti option and to rebuild the project. I will use this binary as a reference here.

Execute the Program with `gdb`

To understand how vtables are used in assembly, disassembling the binary with objdump --dissassemble could have been an option, but instead, I decided to use gdb to actually execute the program:

$ gdb a.out

By default, gdb uses AT&T syntax to disassemble the code, but I prefer Intel flavor:

(gdb) set disassembly-flavor intel
(gdb) show disassembly-flavor
The disassembly flavor is "intel".

Enabling name demangling will make it easier to understand the symbols being manipulated:

(gdb) set print asm-demangle
(gdb) show print asm-demangle
Demangling of C++/ObjC names in disassembly listings is on.

Run to the main:

(gdb) break main
Breakpoint 1 at 0x1139: file /home/pierre/CLionProjects/untitled/main.cpp, line 3.
(gdb) run
Starting program: /home/pierre/CLionProjects/untitled/cmake-build-debug/a.out 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main () at /home/pierre/CLionProjects/untitled/main.cpp:3
3       int main() {

We can now disassemble the code:

(gdb) disas
Dump of assembler code for function main():
=> 0x0000555555555139 <+0>:     sub    rsp,0x18
   0x000055555555513d <+4>:     mov    DWORD PTR [rsp+0x8],0x0
   0x0000555555555145 <+12>:    mov    DWORD PTR [rsp+0xc],0x0
   0x000055555555514d <+20>:    lea    rax,[rip+0x2c44]        # 0x555555557d98 <vtable for Derived+16>
   0x0000555555555154 <+27>:    mov    QWORD PTR [rsp],rax
   0x0000555555555158 <+31>:    mov    rdi,rsp
   0x000055555555515b <+34>:    call   0x5555555551c1 <use(Base const&)>
   0x0000555555555160 <+39>:    mov    eax,0x0
   0x0000555555555165 <+44>:    add    rsp,0x18
   0x0000555555555169 <+48>:    ret
End of assembler dump.

We can clearly see the call to use(). The immediately preceding line prepares the first and only argument of this function (it puts it in the rdi register). The other lines above are the initialisation of obj, our instance of Derived. We can't understand how use() is called and uses the vtable to call Derived::foo() if we don't understand how obj is initialized and how it is connected to the vtable.

Clarification About Addresses

Before we dive in a line-by-line assembly analysis, I want to clarify why the address of use() in the call instruction is 0x005555555551c1 and not 0x00000000000011c1 (as in the symbol table from the previous episode, or in the disassembly produced by obdjump --disassemble).

When the binary is loaded in memory and executed, it's not placed at address 0x0000000000000000 but somewhere else. The addresses in the ELF file are relative from this "somewhere else".

We can compute this offset with a simple subtraction: 0x005555555551c1 - 0x00000000000011c1 = 0x00555555554000. The same translation can be applied to any other symbols. We can verify this offset with the address of main. We need the address of main in the binary:

$  objdump --syms a.out | grep 'main' | grep '.text'
0000000000001139 g     F .text  0000000000000031              main

If we add the offset to this address, the result is the first address in the disassembly above: 0x0000000000001139 + 0x00555555554000 = 0x0000555555555139.

Construction of the Object

In main(), obj is initialized by these instructions (note that => indicates the line where gdb is paused):

=> 0x0000555555555139 <+0>:     sub    rsp,0x18
   0x000055555555513d <+4>:     mov    DWORD PTR [rsp+0x8],0x0
   0x0000555555555145 <+12>:    mov    DWORD PTR [rsp+0xc],0x0
   0x000055555555514d <+20>:    lea    rax,[rip+0x2c44]        # 0x555555557d98 <vtable for Derived+16>
   0x0000555555555154 <+27>:    mov    QWORD PTR [rsp],rax

The first line moves the stack pointer, allocating space to store obj. We can execute a few commands to verify this:

(gdb) info reg rsp
rsp            0x7fffffffddd8      0x7fffffffddd8
(gdb) nexti
4           auto obj = Derived();
(gdb) info reg rsp
rsp            0x7fffffffddc0      0x7fffffffddc0
(gdb) print &obj
$2 = (Derived *) 0x7fffffffddc0

After these commands, gdb is paused on the second line, at the address 0x000055555555513d. The address of obj is the same as the value of the rsp register. At this moment, the object is created but not initialized:

(gdb) print obj
$3 = {<Base> = {_vptr.Base = 0x0, dummy_base = 1431671456}, dummy_derived = 21845}

It's not surprising to see dummy_base and dummy_derived, as they are the member data of Derived. But what is _vptr.Base? This is the "virtual pointer". It's automatically inserted by the compiler to reference the vtable. It doesn't point to (the beginning of) the vtable actually, but to a location that the ABI's specification refers to as the "virtual table address point", inside the vtable. This point corresponds to the first function pointer in the vtable. We will get back to this pointer later.

The next instructions are here to initialize obj.

First, dummy_base and dummy_derived are set to 0 by the two mov instructions (yes, even though our code doesn't require to initialize them). A C++ equivalent of mov DWORD PTR [rsp+0x8],0x0 would ressemble something like *(rsp + 0x8) = 0x0. At this point, the value of rsp is 0x007fffffffddc0, confirming that rsp+0x8 and rsp+0xc are indeed the addresses of the data members:

(gdb) print &obj.dummy_base
$5 = (int *) 0x7fffffffddc8
(gdb) print &obj.dummy_derived
$6 = (int *) 0x7fffffffddcc

The initialization of the virtual pointer is slightly more complex:

   0x000055555555514d <+20>:    lea    rax,[rip+0x2c44]        # 0x555555557d98 <vtable for Derived+16>
   0x0000555555555154 <+27>:    mov    QWORD PTR [rsp],rax

The lea instruction with an operand that is relative to the rip register is classic compiler technique to get the address of something that is inside the binary being executed. gdb kindly shows the result of the computation. It even tells us that this is the address of vtable for Derived+16. In the previous episode, we noted that the first function address in the vtable comes after 8 bytes for the offset and 8 bytes for the pointer to the RTTI, hence a total of 16 bytes. You should understand now why the virtual pointer is set to vtable for Derived+16: this is the "virtual table address point".

This address is temporarily stored into rax before being moved on the stack to complete the initialization of obj.

We can execute these instructions with gdb to run the initialization:

(gdb) ni
(gdb) ni
(gdb) ni
(gdb) ni
(gdb) print obj
$7 = {<Base> = {_vptr.Base = 0x555555557d98 <vtable for Derived+16>, dummy_base = 0}, dummy_derived = 0}

If we want to verify what is being pointed to by the virtual pointer, we have 2 options.

The first solution is the manual, tedious one. We can dump the memory located at this address and compare the dumped values to the addresses of the member functions of Derived. We need to dump 2 sets of 8 bytes each (since a function pointer is 8-byte wide):

(gdb) x/2gx 0x555555557d98
0x555555557d98 <vtable for Derived+16>: 0x0000555555555196      0x00005555555551ac
(gdb) print 'Derived::foo'
$8 = {void (const Derived * const)} 0x555555555196 <Derived::foo() const>
(gdb) print 'Derived::bar'
$9 = {void (const Derived * const)} 0x5555555551ac <Derived::bar() const>

The second solution is just using the dedicated command provided by gdb:

(gdb) info vtbl obj
vtable for 'Derived' @ 0x555555557d98 (subobject @ 0x7fffffffddc0):
[0]: 0x555555555196 <Derived::foo() const>
[1]: 0x5555555551ac <Derived::bar() const>

Calling `use()` from `main()`

The object is now ready, and gdb is paused here:

(gdb) disas
(...)
=> 0x0000555555555158 <+31>:    mov    rdi,rsp
   0x000055555555515b <+34>:    call   0x5555555551c1 <use(Base const&)>
(...)
End of assembler dump.

On Linux-x64, the calling convention for functions uses the rdi register as the first parameter. We've seen in the previous section that rsp holds the address of obj at this point. This address is moved in the rdi register and the function is ready to be called. Indeed, a reference in C++ is often just an address in assembly.

How the Virtual Function is Actually Called

We can reach the beginning of use() with ni (to execute the mov) and si (to step into the call). We can disassemble the function:

(gdb) disas
Dump of assembler code for function _Z3useRK4Base:
=> 0x00005555555551c1 <+0>:     sub    rsp,0x8
   0x00005555555551c5 <+4>:     mov    rax,QWORD PTR [rdi]
   0x00005555555551c8 <+7>:     call   QWORD PTR [rax]
   0x00005555555551ca <+9>:     add    rsp,0x8
   0x00005555555551ce <+13>:    ret
End of assembler dump.

The virtual call is right here!

We have seen that the value of rdi is the virtual pointer and that the first entry in the array pointed to by this virtual pointer is the address of Derived::foo(). The value pointed to by rdi is moved into rax, and then the value pointed to by rax is called. The function Derived::foo() executes and prints Derived => foo()!

You can try to change the code to call bar() instead of foo(). The call instruction will be different. You will have call QWORD PTR [rax+0x8] instead, to get the next pointer in the array of function pointers.

Indirect Call

In use(), the instruction call QWORD PTR [rax] is an indirect call because the value of a register is used. The instruction call 0x5555555551c1 to call use() from main() is a direct call. The mnemonic is the same (call) but the opcodes (the binary values used to translate the mnemonic) are different.

We can examine the memory with gdb to see the difference:

(gdb) x /1bx use+7
0x5555555551c8 <use(Base const&)+7>:    0xff
(gdb) x /1bx main+34
0x55555555515b <main()+34>:     0xe8

Indirect calls are typically used to implement function pointer.

Conclusion

In this episode, we discovered that the compiler adds a hidden field for each instance of a class with virtual functions. This field is the "virtual pointer" (or "vptr" in short). Thanks to this pointer, we can get a function pointer inside the vtable, and the desired function.

I will publish the episodes as I write them. Subscribe to be notified! 💃🏽

vtables under the surface | Episode 2 - ELF files

Pierre Gradot — Fri, 05 Jan 2024 09:07:47 +0000

In this episode, we will explore what vtables mean in terms of bytes within ELF files.

Build Output

On Linux, GCC produces ELF files as the result of the compilation process. In our project, the file is a.out, and we can use the file command to get details about it:

$ file a.out 
a.out: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=a87e1cb2356338a14f1a9aa2fef85fb7036bee65, for GNU/Linux 3.2.0, not stripped

If the compiler has generated vtables for Base and Derived, there must be corresponding symbols and bytes in the binary.

Inspect the Symbols

Let's identify the symbols related to these classes. We can use objdump to get the symbol table, employ sort to order the symbols by addresses, and filter the output to retain only what relates to Base and Derived:

$ objdump --syms --demangle a.out | sort | egrep "Base|Derived"
000000000000116a g     F .text  0000000000000015              Base::foo() const
0000000000001180 g     F .text  0000000000000015              Base::bar() const
0000000000001196 g     F .text  0000000000000015              Derived::foo() const
00000000000011ac g     F .text  0000000000000015              Derived::bar() const
00000000000011c1 g     F .text  000000000000000e              use(Base const&)
0000000000002042  w    O .rodata        0000000000000006              typeinfo name for Base
0000000000002048  w    O .rodata        0000000000000009              typeinfo name for Derived
0000000000003d68  w    O .data.rel.ro   0000000000000020              vtable for Base
0000000000003d88  w    O .data.rel.ro   0000000000000020              vtable for Derived
0000000000003da8  w    O .data.rel.ro   0000000000000010              typeinfo for Base
0000000000003db8  w    O .data.rel.ro   0000000000000018              typeinfo for Derived

If you are not familiar with the output format of objdump --syms, you can refer to the man pages of the command. The help of the --syms options provide with detailed information about each column.

We see the addresses of our 4 member functions, as well as use(). These functions are stored in the .text section, which is the dedicated section for code. We also see several objects (notice the O in the third column) for the information about the types and... the virtual tables! Let's dig into the contents of these tables.

Alternatives to `objdump`

Before we move to the hard values, I just want to briefly mention alternatives to objdump. Unfortunately, neither display the sections along with the symbols (please leave a comment if I have missed the magic options!).

Alternative 1 = nm

$ nm --demangle --print-size a.out | sort | egrep "Base|Derived"
000000000000116a 0000000000000015 T Base::foo() const
0000000000001180 0000000000000015 T Base::bar() const
0000000000001196 0000000000000015 T Derived::foo() const
00000000000011ac 0000000000000015 T Derived::bar() const
00000000000011c1 000000000000000e T use(Base const&)
0000000000002042 0000000000000006 V typeinfo name for Base
0000000000002048 0000000000000009 V typeinfo name for Derived
0000000000003d68 0000000000000020 V vtable for Base
0000000000003d88 0000000000000020 V vtable for Derived
0000000000003da8 0000000000000010 V typeinfo for Base
0000000000003db8 0000000000000018 V typeinfo for Derived

Alternative 2 : readelf

$ readelf --syms --demangle a.out | cut -d: -f2- | sort | egrep "Base|Derived"
 000000000000116a    21 FUNC    GLOBAL DEFAULT   15 Base::foo() const
 0000000000001180    21 FUNC    GLOBAL DEFAULT   15 Base::bar() const
 0000000000001196    21 FUNC    GLOBAL DEFAULT   15 Derived::foo() const
 00000000000011ac    21 FUNC    GLOBAL DEFAULT   15 Derived::bar() const
 00000000000011c1    14 FUNC    GLOBAL DEFAULT   15 use(Base const&)
 0000000000003d68    32 OBJECT  WEAK   DEFAULT   22 vtable for Base
 0000000000003d88    32 OBJECT  WEAK   DEFAULT   22 vtable for Derived
 0000000000003da8    16 OBJECT  WEAK   DEFAULT   22 typeinfo for Base
 0000000000003db8    24 OBJECT  WEAK   DEFAULT   22 typeinfo for Derived

To sort by addresses, we must remove the symbol number at the beginning of each line. Indeed, a line is formatted as follows:

22: 0000000000001180    21 FUNC    GLOBAL DEFAULT   15 Base::bar() const

Note that typeinfo name for Base and Derived are missing here (again, if you know why, leave a comment below!).

The Bytes in Vtables

Vtables and typeinfo symbols reside in the .data.rel.ro section. We need to examine the raw data of this section to understand their content. readelf can produce a hexdump of a section:

$ readelf --hex-dump .data.rel.ro a.out

Hex dump of section '.data.rel.ro':
  0x00003d68 00000000 00000000 a83d0000 00000000 .........=......
  0x00003d78 6a110000 00000000 80110000 00000000 j...............
  0x00003d88 00000000 00000000 b83d0000 00000000 .........=......
  0x00003d98 96110000 00000000 ac110000 00000000 ................
  0x00003da8 00000000 00000000 42200000 00000000 ........B ......
  0x00003db8 00000000 00000000 48200000 00000000 ........H ......
  0x00003dc8 a83d0000 00000000                   .=......

Note that:

These are little-endian hexadecimal data. Hence, a83d0000 00000000 on the last line is in fact 0x0000000000003da8.
The same dump can be created with objdump --section=.data.rel.ro --full-contents a.out.

Let's inspect these bytes.

How? By navigating through the symbol table to obtain the addresses and sizes of the symbols, and utilizing hexdumps to find out the bytes at these addresses. We will then gain a better understanding of these symbols.

Stay calm and take a deep breath, there will be a lot a hexadecimal numbers.

Vtables

According to the symbol table, both vtables are 32-byte wide (this is 0x20 in hexadecimal) and their addresses are:

0x0000000000003d68 for the vtable for Base
0x0000000000003d88 for the vtable for Derived

Since each line in the hexdump holds 16 bytes, we can deduce that, in the hexdump:

Lines 1 and 2 show the vtable for Base.
Lines 3 and 4 show the vtable for Derived.

The ABI's specification (or this presentation on slide 5) explains the content of the vtable. Let's apply this knowledge to analyze the vtables, and match the dumped bytes to addresses in the symbol tables.

For Base:

Dump	Value	Type	Symbol
00000000 00000000	0x0000000000000000	`ptrdiff_t`
a83d0000 00000000	0x0000000000003da8	pointer to struct	typeinfo for `Base`
6a110000 00000000	0x000000000000116a	function pointer	`Base::foo const`
80110000 00000000	0x0000000000001180	function pointer	`Base::bar const`

For Derived:

Dump	Value	Type	Symbol
00000000 00000000	0x0000000000000000	`ptrdiff_t`
b83d0000 00000000	0x0000000000003db8	pointer to struct	typeinfo for `Derived`
96110000 00000000	0x0000000000001196	function pointer	`Derived::foo() const`
ac110000 00000000	0x00000000000011ac	function pointer	`Derived::bar() const`

As you may have guessed, if there were more virtual functions, the vtables would contain more function pointers.

Typeinfo

The vtables point to the typeinfo objects, let's have a closer look at their contents.

From the symbol table, we know that:

Symbol	Address	Size (hexa)	Size (dec)
typeinfo for `Base`	0x0000000000003da8	0x10	16
typeinfo for `Derived`	0x0000000000003db8	0x18	24

In the hexdump of the section, we can deduce that:

Line 5 shows the typeinfo for Base.
Lines 6 and 7 show the typeinfo for Derived.

Once again, we can match the dumped bytes with the data from the symbol table.

For Base:

Dump	Value	Symbol
00000000 00000000	0x0000000000000000
42200000 00000000	0x0000000000002042	typeinfo name for `Base`

For Derived:

Dump	Value	Symbol
00000000 00000000	0x0000000000000000
48200000 00000000	0x0000000000002048	typeinfo name for `Derived`
a83d0000 00000000	0x0000000000003da8	typeinfo for `Base`

As you have probably guessed, the objects are implementations of std::type_info. The ABI's specification includes a dedicated section for RTTI. For in-depth details, you may explore (the somewhat esoteric) part about RTTI layout. You will see that std::type_info has several subtypes, and that __si_class_type_info is (in all likelihood) used here.

Typeinfo Names

The remaining data we haven't explored yet are the names residing in the .rodata section. This section is dedicated to store the data from our code.

$ readelf --hex-dump .rodata a.out

Hex dump of section '.rodata':
  0x00002000 01000200 42617365 203d3e20 666f6f28 ....Base => foo(
  0x00002010 29004261 7365203d 3e206261 72282900 ).Base => bar().
  0x00002020 44657269 76656420 3d3e2066 6f6f2829 Derived => foo()
  0x00002030 00446572 69766564 203d3e20 62617228 .Derived => bar(
  0x00002040 29003442 61736500 37446572 69766564 ).4Base.7Derived
  0x00002050 00                                  .

On the right side of the dump, we find the ASCII interpretation of the hexadecimal values.

Here, we see the strings printed in each function and the mangled names of the classes. Why do the mangled names contain 4 and 7? This simply corresponds to the length of the subsequent identifiers.

Disable RTTI

As a bonus, we will disable RTTI and observe the impact on the vtables.

Modify your CMakeLists.txt to add the appropriate option:

target_compile_options(a.out PRIVATE
        -O1
        -fno-rtti
)

Once the project has been recompiled, we can examine the symbol table again:

$ objdump --syms --demangle a.out | sort | egrep "Base|Derived"
000000000000116a g     F .text  0000000000000015              Base::foo() const
0000000000001180 g     F .text  0000000000000015              Base::bar() const
0000000000001196 g     F .text  0000000000000015              Derived::foo() const
00000000000011ac g     F .text  0000000000000015              Derived::bar() const
00000000000011c1 g     F .text  000000000000000e              use(Base const&)
0000000000003da0  w    O .data.rel.ro   0000000000000020              vtable for Base
0000000000003dc0  w    O .data.rel.ro   0000000000000020              vtable for Derived

The size of the vtables remains unchanged, but all symbols related to type information are now absent.

We can also examine the hexdump of the .data.rel.ro section:

$ readelf --hex-dump .data.rel.ro a.out

Hex dump of section '.data.rel.ro':
  0x00003da0 00000000 00000000 00000000 00000000 ................
  0x00003db0 6a110000 00000000 80110000 00000000 j...............
  0x00003dc0 00000000 00000000 00000000 00000000 ................
  0x00003dd0 96110000 00000000 ac110000 00000000 ................

The second double word in each vtables is now zero, to represent a null pointer, since there is no RTTI to point to.

If you dump the .rodata section, you will see that the type names are absent as well.

Conclusion

In this episode, we delved into the bytes stored in the ELF file for the vtables. Their content is mostly dictated by the Itanium ABI followed by GCC. Vtables hold a pointer to RTTI (if enabled) along with pointers to all virtual functions. This was a basic case, with a single level of simple inheritance. I may cover more advanced cases in a further episode.

In the next episode, we will explore how vtables are used at assembly level.

I will publish the episodes as I write them. Subscribe to be notified! 🥳

vtables under the surface | Episode 1 - Concepts

Pierre Gradot — Fri, 05 Jan 2024 09:04:49 +0000

If you've been around C++ for a while, you've likely come across the terms "vtable", "virtual table" or "virtual method table". Vtables are not part of the C++ standard, even though this concept pops up almost immediately when you try to understand how virtual functions actually work in C++. Indeed, vtables are the most common implementation of polymorphism in C++.

You may also have already encountered a cryptic compilation error like "undefined reference to vtable for MyClass". This may have left you perplexed because you didn't explicitly create anything named "vtable" in your code. This error signals that, under the hood, the compiler generates vtables to handle virtual functions. If you're curious about this error, here is a good discussion on stackoverflow (the second answer is particularly enlightening).

In this series, we're going to the lion's den and explore how a vtable works at the byte and assembly levels.

Disclaimer

As mentioned in the introduction, we'll be discussing implementation details. By definition, these details may vary from one compiler to another or from one OS to another. They may even change in the next version of the same compiler for the same OS.

However, most major compilers (except MSVC) follow the Itanium C++ ABI:

The Itanium C++ ABI is an ABI for C++. As an ABI, it gives precise rules for implementing the language, ensuring that separately-compiled parts of a program can successfully interoperate. Although it was initially developed for the Itanium architecture, it is not platform-specific and can be layered portably on top of an arbitrary C ABI. Accordingly, it is used as the standard C++ ABI for many major operating systems on all major architectures, and is implemented in many major C++ compilers, including GCC and Clang.

The ABI has a section about vtables, so using any compiler following this ABI should yield similar implementation details.

In this series, I will use GCC 12.2.0 on Debian BookWorm (unless stated otherwise).

The Code

Here is a code that will serve as the foundation for the discussion in the series (at least for the first episodes). It is split into 3 different files:

code.hpp:

#pragma once

struct Base {
    virtual void foo() const;
    virtual void bar() const;
private:
    int dummy_base;
};

struct Derived : Base {
    void foo() const override;
    void bar() const override;
private:
    int dummy_derived;
};

void use(const Base&);

code.cpp:

#include "code.hpp"

#include <cstdio>

void Base::foo() const {
    std::puts("Base => foo()");
}

void Base::bar() const {
    std::puts("Base => bar()");
}

void Derived::foo() const {
    std::puts("Derived => foo()");
}

void Derived::bar() const {
    std::puts("Derived => bar()");
}

void use(const Base& obj) {
    obj.foo();
}

main.cpp:

#include "code.hpp"

int main() {
    auto obj = Derived();
    use(obj);
}

This code is a typical example of polymorphism using virtual functions. use() receives a reference to a Base object. At runtime, dynamic dispatch selects and calls Derived::foo() and the program prints Derived => foo(). Looking at this code, you don't see any vtables. We will analyze the generated binary and find them later in the series.

Build and Run

You can easily build the program with CMake:

cmake_minimum_required(VERSION 3.25)
project(vtables)

add_executable(a.out
        main.cpp
        code.cpp
        code.hpp
)

target_compile_options(a.out PRIVATE
        -O1
)

From the root directory of the project:

$ cmake -B build -S .
$ cmake --build build
$ ./build/a.out 
Derived => foo()

Notes:

Separated source files avoid possible compiler optimizations.
-O1 generates an assembly code that is much easier to read and understand than -O0.

We're Ready

We are now ready. Let's dive into vtables!

I will publish the episodes as I write them. Subscribe to be notified! 🎉

Just in case: Debian Bookworm comes with a buggy GCC

Pierre Gradot — Fri, 15 Dec 2023 11:27:09 +0000

Last month, I embarked on a new project. I set up a new computer with the latest Debian version, installed my favorites tools, and was all set to code. My first task was to migrate all the repositories from C++14 to C++20. While it might seem as straightforward as updating all the CMakeLists.txt to replace set (CMAKE_CXX_STANDARD 14) with set (CMAKE_CXX_STANDARD 20) reality proved otherwise (and I knew it would).

Among the funny things I encountered (such as std::experimental::basic_string_view::to_string() being missing in std::basic_string_view), the most cryptic error was something like that:

/usr/include/c++/12/bits/char_traits.h:431:56: warning: ‘void* __builtin_memcpy(void*, const void*, long unsigned int)’ accessing 9223372036854775810 or more bytes at offsets -4611686018427387902 and [-4611686018427387903, 4611686018427387904] may overlap up to 9223372036854775813 bytes at offset -3 [-Wrestrict]

  431 |         return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n));

This is a known bug in GCC and it affects only the 12.x branch. It's (Bug 105329 - [12/13/14 Regression] Bogus restrict warning when assigning 1-char string literal to std::string since r12-3347-g8af8abfbbace49e6).

Encountering this bug might not be an everyday occurrence, but it's quite easy to trigger it. Unfortunately, as you have guessed, Debian Bookworm ships with this problematic version of GCC. Bookworm is the latest stable version of Debian, released in June this year. Previous stable versions, Buster and Bulleye, came with GCC 8 and GCC 10 respectively. Since Debian releases a new stable version every two years, if you are using Debian like me, we may have to deal with this bug for an extended period.

In this article, I will explain how you can trigger this bug and how you can avoid it. It's also an opportunity to revisit various techniques for circumventing spurious warnings.

Conditions to Trigger the Bug

Beyond from the source code itself, specific build conditions are necessary to trigger the bug.

You must use GCC 12, obviously. GCC 11 wasn't affected and the bug is fixed in GCC 13.
Optimizations must be enabled. You must use either -O2 or -O3. -O0, -O1 and -Os don't seem to concerned. If you set CMAKE_BUILD_TYPE to Release, CMake will add -O3. Your release builds will then be susceptible, unlike debug builds.
You must use C++20 and C++23. C++17 is unaffected.
You must pass the -Wrestrict flag since it causes the warning. It is included in -Wall.

This means your build may break simply by:

Updating GCC (it happened to GoogleTest).
Changing from debug to release mode (but I guess your CI always builds in release mode, right?).
Switching for C++17 or below to C++20 or above (and I encourage you to regularly upgrade to the new standard version).
Enabling warnings (no, I can't believe you had lived without -Wall so far).

I use the word "break" because most release builds also use -Werror to turn warnings into errors, ensuring they are absolutely noticeable.

The Code Itself

Oh, poor boyz and girlz... The code is so simple.

Here is a CMake CMakeLists.txt:

cmake_minimum_required(VERSION 3.27)
project(buggy)

add_executable(buggy main.cpp)
target_compile_options(buggy PRIVATE -Wall)
target_compile_features(buggy PRIVATE cxx_std_20)

And here a main.cpp that will raise the dreaded warning:

#include <string>

std::string s;

int main() {
    s = "a"; // oopsi...
}

Here is another case that emits the warning:

#include <string>

std::string s;

int main() {
    s = "a" + std::string("b"); // oopsi...
}

The string literal must be single byte and must be the left-hand-side operand. The following lines are fine:

s = "ab";
s = "ab" + std::string("c");
s = std::string("a") + "b";

Circumvention Measures

The discussion for bug 105329 clearly indicates, in comments 26 and 27, than the 12 branch won't be fixed. The solution is hence to move to GCC 13. Alas! Debian will probably not change the major version of GCC in Bookworm. The next release, Trixie, is currently in testing mode and comes with GCC 13, so perhaps one day it will be available in the backports.

Fortunately, there are easy workarounds.

Disable `-Wrestrict` globally

As a temporary solution, you may add -Wno-restrict to your build:

target_compile_options(buggy PRIVATE -Wall -Wno-restrict)

This is extreme and I highly discourage pushing such a change to a develop or a master/main branch.

Disable `-Wrestrict` for some files

Another solution I also consider temporary is to disable the warning but only for a particular set of files.

You can use CMake for this, especially if you can't change these sources files:

set_source_files_properties(
        main.cpp
        PROPERTIES
        COMPILE_FLAGS -Wno-restrict)

Alternatively, you can use a pragma at the beginning of each file:

#pragma GCC diagnostic ignored "-Wrestrict"
#include <string>

std::string s;

int main() {
    s = "a";
}

Disable `-Wrestrict` locally

Disabling the warning only for a couple of lines is a more acceptable solution:

#include <string>

std::string s;

int main() {
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wrestrict"
    s = "a";
#pragma GCC diagnostic pop
}

I had to use this technique around a #include <boost/test/data/test_case.hpp> because BOOST_DATA_TEST_CASE triggered the bug. This solution is very verbose and reduce your code's readability. Think twice before choosing it.

Note about Clang

Clang does support #pragma GCC but it doesn't support -Wrestrict... It means that the previous codes will raise a warning or error:

$ clang++ main.cpp -Wall -Werror
main.cpp:7:32: error: unknown warning group '-Wrestrict', ignored [-Werror,-Wunknown-warning-option]
#pragma GCC diagnostic ignored "-Wrestrict"

If you want your code to compile both with GCC and Clang, you will need some extra preprocessor tricks:

#ifndef __clang__
#pragma GCC diagnostic ignored "-Wrestrict"
#endif

See Clang's builtin-macros.

The preferred solution: change the code

The solution with set_source_files_properties() may be acceptable if you plan to switch to GCC 13 soon. Otherwise, if you plan to use GCC 12 for a long time, just change your code.

Remember the two faulty lines:

 s = "a";
 s = "a" + std::string("b");

Simply change them to:

s = std::string("a");
s = std::string("a") + std::string("b");

This simply makes implicit conversions explicit.

Conclusion

At the end of the day, the warning message is quite scary, but the code can easily be changed to avoid the bug. Apart from changing the code to avoid this particular issue with GCC 12, we also discussed several techniques you may use to mitigate spurious warnings in your code.

This is not the first time I run into a compiler's bug. It's always so much fun (sarcasm indented) 😆

Des enumérations encore plus puissantes avec Python 3.11

Pierre Gradot — Thu, 08 Jun 2023 14:27:10 +0000

Python 3.11 est sorti à la fin de l'année dernière. Comme souvent, il y a beaucoup de nouveautés. Une section a plus attiré mon œil que les autres : il y a eu beaucoup de changements et d'ajouts sur les enums ! Bon, la vérité, c'est que je cherchais comment faire quelque chose d'assez spécifique et j'ai vu que Python 3.11 apportait justement cette fonctionnalité... J'ai bien sûr immédiatement mis à jour mon interpréteur pour tester ça !

Dans cet article, je vous présente les nouveautés qui me semblent les plus prometteuses.

3.11 : une version importante pour le module `enum`

Le module enum est très stable depuis son apparition en version 3.4 et l'implémentation de la PEP 435.

Au début de la documentation, on voit :

New in version 3.6: Flag, IntFlag, auto

New in version 3.11: StrEnum, EnumCheck, ReprEnum, FlagBoundary, property, member, nonmember, global_enum, show_flag_values

La version 3.11 est donc une version qui apporte beaucoup de nouveautés. 9 sont listées au début de la documentation, mais il y en a une 10ᵉ qu'on trouve plus bas : verify(). En vrai, la documentation est loin d'être parfaite, mais on s'en sort.

Streets of Rage

Pour le fun, j'ai décidé d'utiliser des exemples basés sur Streets of Rage. Quoi ?! Tu connais pas Streets of Rage ?! Mais fonce vite agrandir ta culture pop !

Mon épisode préféré est clairement le 2, mais j'utiliserai un peu le 1 aussi !

Ce qu'on pouvait déjà faire avant Python 3.11

Si on souhaite lister les niveaux de Streets of Rage 2, il est possible de faire une énumération comme celle-ci depuis Python 3.4 :

class Stages(Enum):
    DOWNTOWN = 1
    BRIDGE_CONSTRUCTION = 2
    AMUSEMENT_PARK = 3
    STADIUM = 4
    # ... et plusieurs autres encore !

Elles sont très permissives et on peut par exemple faire quelque chose comme :

class Stages(Enum):
    DOWNTOWN = 1
    BRIDGE_CONSTRUCTION = 2
    AMUSEMENT_PARK = 'three'
    STADIUM = [4]

Il est donc possible d'avoir des valeurs de types différents. Ça peut être pratique dans certains cas, mais on souhaite en général imposer le type des valeurs, comme dans notre exemple dans lequel chaque niveau correspond à un numéro. C'est pour cette raison que IntEnum a été introduite en Python 3.6 :

class Stages(IntEnum):
    DOWNTOWN = 1
    BRIDGE_CONSTRUCTION = 2
    AMUSEMENT_PARK = 3
    STADIUM = 'four'

On obtient une exception à l'exécution : ValueError: invalid literal for int() with base 10: 'four'.

Notez que si on a STADIUM = '4' (notez les simple quotes autour du 4), le code fonctionne. En effet, comme l'indique l'exception, IntEnum utilise int() pour obtenir la valeur et il s'avère que int('4') == 4. On peut ainsi utiliser comme initializer une instance d'une classe qui fournit une méthode def __int__(self) -> int.

IntEnum est en fait une "mixed enum". Le principe des mixed enums est de faire un héritage (multiple) d'un type souhaité T et d'enum. Ce n'est pas très bien documenté à mon goût, mais on trouve des explications dans le "Enum HOWTO" (ici et un peu là). On obtient ainsi une énumération dont les valeurs sont obligatoirement du même type T.

Après ces rappels, on va s'attarder dans la suite aux changements apportés par la version 3.11.

`ReprEnum`

Si on hérite ReprEnum plutôt que Enum, on créé une énumération pour laquelle la conversion en string de ses valeurs sera alors la même que la conversion du mixed-in type. La documentation précise :

ReprEnum uses the repr() of Enum, but the str() of the mixed-in data type.

(...)

Inherit from ReprEnum to keep the str() / format() of the mixed-in data type instead of using the Enum-default str().

L'affichage des `IntEnum`s change à cause de `ReprEnum`

What’s New In Python 3.11 nous dit :

Changed IntEnum (...) to now inherit from ReprEnum, so their str() output now matches format() (both str(AnIntEnum.ONE) and format(AnIntEnum.ONE) return '1', whereas before str(AnIntEnum.ONE) returned 'AnIntEnum.ONE'.

Regardons ce que ça donne avec notre énumération Stages(IntEnum) :

print('member\t', Stages.DOWNTOWN)
print('name\t', Stages.DOWNTOWN.name)
print('value\t', Stages.DOWNTOWN.value)
print('str()\t', str(Stages.DOWNTOWN))
print('repr()\t', repr(Stages.DOWNTOWN))
print('f-str\t', f'{Stages.DOWNTOWN}')

En 3.10 :

member   Stages.DOWNTOWN
name     DOWNTOWN
value    1
str()    Stages.DOWNTOWN
repr()   <Stages.DOWNTOWN: 1>
f-str    1

Affichage modifié en 3.11 :

member   1
name     DOWNTOWN
value    1
str()    1
repr()   <Stages.DOWNTOWN: 1>
f-str    1

Personnellement, je trouve ça plus logique, mais ce changement peut avoir des conséquences sur un code existant !

`StrEnum`, pour faire comme `IntEnum` mais avec des strings

On a souvent besoin de faire une énumération qui ne contient que des strings, par exemple pour lister les personnages du jeu :

class Characters(StrEnum):
    AXEL = 'Axel Stone'
    BLAZE = 'Blaze Fielding'
    MAX = 'Max Thunder'
    SKATE = 'Eddie "Skate" Hunter'

StrEnum hérite de ReprEnum, ce qui implique que print(str(Characters.BLAZE)) et print(f'{Characters.BLAZE}') affichent Blaze Fielding. Si on avait fait Characters(Enum), l'affichage aurait donné Characters.BLAZE. Comme pour IntEnum, je trouve cet affichage logique.

On peut utiliser auto() avec StrEnum :

class Characters(StrEnum):
    # ...
    SKATE = auto()

str(Characters.SKATE)) sera alors skate.

Il était déjà possible de faire un équivalent de StrEnum avant Python 3.11, avec une simple enum mais le typage était moins fort. On pouvait par exemple faire :

class Characters(str, Enum):
    AXEL = 'Axel Stone'
    BLAZE = 'Blaze Fielding'
    MAX = 'Max Thunder'
    SKATE = 8

Et ça passait crème. En effet, il est possible de construire une string à partir de 8 avec str(8). Ce n'est pas dit dans la doc, mais on peut regarder l'implémentation de StrEnum dans enum.py et on voit que le constructeur est redéfini et vérifie explicitement le typage avec des isinstance(..., str). Ce n'est pas le cas de IntEnum.

Plus de vérifications avec le décorateur `@verify`

@unique est présent depuis le début du module enum et permet de s'assurer que chaque membre a une valeur... unique ! 😂

C'est très bien pour définir les niveaux du jeu et s'assurer qu'ils ont tous un numéro différent. Exemple :

@unique
class Stages(IntEnum):
    DOWNTOWN = 1
    BRIDGE_CONSTRUCTION = 2
    AMUSEMENT_PARK = 3
    STADIUM = 3

Ce code génère une exception : ValueError: duplicate values found in <enum 'Stages'>: STADIUM -> AMUSEMENT_PARK.

Un nouveau décorateur, @verify, est apparu en 3.11 :

A class decorator specifically for enumerations. Members from EnumCheck are used to specify which constraints should be checked on the decorated enumeration.

Il prend donc en paramètres des EnumChecks :

EnumCheck contains the options used by the verify() decorator to ensure various constraints; failed constraints result in a ValueError.

Seuls UNIQUE, CONTINUOUS et NAMED_FLAGS sont disponibles pour le moment. @verify(UNIQUE) est équivalent à @unique.

On peut passer plusieurs flags en paramètres, ce qui est parfait pour notre exemple :

@verify(UNIQUE, CONTINUOUS)
class Stages(IntEnum):
    DOWNTOWN = 1
    BRIDGE_CONSTRUCTION = 2
    AMUSEMENT_PARK = 3
    STADIUM = 5

Une exception nous prévient qu'il manque une valeur : ValueError: invalid enum 'Stages': missing values 4.

Rendre les membres accessibles dans le namespace global

Pour accéder à un membre, il faut normalement y accéder via la classe : Stages.STADIUM.

Dans certains cas (et avec les éventuels risques de name clashes qui vont avec), vous pourriez souhaitez utiliser directement STADIUM. C'est possible à partir de Python 3.11, en annotant votre classe avec @global_enum.

Contrôler ce qui est membre et ce qui n'est pas membre

Deux nouveaux décorateurs permettent de contrôler explicitement ce qui est membre de l'énumération et ce qui ne l'est pas :

@enum.member
A decorator for use in enums: its target will become a member.

@enum.nonmember
A decorator for use in enums: its target will not become a member.

Quand on parle de membres d'une énumération, on parle de ses différentes valeurs possibles.

Ce décorateur @member est très pratique pour définir une énumération dont les valeurs sont des fonctions.

Pour Streets of Rage 2, il nous faut par exemple une énumération des 3 actions de base que peut faire un personnage. Une fonction est un bon type pour représenter une action. Par défaut, une fonction définie dans une classe dérivant de Enum est une static method. Ainsi, le code suivant ne fait pas ce qu'on souhaiterait, car il crée une énumération sans valeur :

class Controls(Enum):
    def special_move():
        print('Special move, massive damage!')

    def attack():
        print('Attack? OK! Punch!')

    def jump():
        print('The floor is lava! Jump!')

print(list(Controls))
Controls.attack()

Ce code affiche :

[]
Attack? OK! Punch!

Pour corriger ça, il suffit d'annoter les fonctions :

class Controls(Enum):
    @member
    def special_move():
        print('Special move, massive damage!')

    @member
    def attack():
        print('Attack? OK! Punch!')

    @member
    def jump():
        print('The floor is lava! Jump!')


print(list(Controls))
Controls.attack.value()

On obtient cette fois :

[<Controls.special_move: <function Controls.special_move at 0x0000015B0080AD40>>,
        <Controls.attack: <function Controls.attack at 0x0000015B00778680>>,
        <Controls.jump: <function Controls.jump at 0x0000015B00822200>>]
Attack? OK! Punch!

Parfait ! Notez bien que Controls.attack n'est pas callable (car c'est le membre de l'énumération) et qu'il faut utiliser .value pour accéder réellement à la fonction.

À l'inverse, si vous voulez qu'une donnée soit statique à la classe, il faut utiliser @nonmember(). La syntaxe est un peu surprenante (je trouve) et la documentation officielle n'en donne aucun exemple. En voici donc un petit pour la route :

class Characters(StrEnum):
    playable = nonmember(True)
    AXEL = 'Axel Stone'
    BLAZE = 'Blaze Fielding'
    MAX = 'Max Thunder'
    SKATE = 'Eddie "Skate" Hunter'


print(Characters.playable)

Comme toujours en Python, un champ de la classe est accessible via ses membres, donc on peut utiliser Characters.SKATE.playable.

Conclusion

Il y a beaucoup de nouveautés intéressantes dans cette version de Python 3.11 ! Quand ton langage principal est C++, où les enumérations sont vraiment très basiques, tu es comme un enfant dans un magasin de bonbons ! Je regrette quand même une documentation pas ouf (certaines features sont très mal, voire pas documentées) et des trucs trop bizarres (comme show_flag_values() qui n'est pas ajouté à __all__ et dont l'utilisabilité est vraiment mauvaise). Gageons que ça s'améliorera dans les prochaines versions et profitons dès maintenant de cette puissance supplémentaire dans le package enum !

DEV Community: Pierre Gradot

User-Defined Deduction Guides for Class Templates in C++

Context

An Example Where CTAD Fails

Base class template

Specialization for ARM

User-Defined Deduction Guide

Other Examples

Example Where CTAD Fails Without a Workaround (*)

Example to Overwrite the Deduced Type

Conclusion

How to Fix "Low Disk Space on /boot" on Linux

What is /boot?

How Resolve The Issue?

1: Expand The Partition

2: Use Synaptic Package Manager

3: Use The Command Line

4: The Automatic Way

Conclusion

clang-tidy, const & (N)RVO

Almost Always Const Auto (AACA)

Complete Example

With A Custom Class

(Named) Return Value Optimization

NRVO is optional

Variation 1

Variation 2

Variation 3

Variation 4

Summary

The Warning Is Not About NRVO

How To Handle The Warning

Be Warned When NRVO Is Not Possible

Disabling (N)RVO with -fno-elide-constructors

C++14

Original code:

Original code without the const:

C++17

Original code:

Original code without the const:

Conclusion

Manually Install WiFi Drivers on Linux

The Symptoms

The Solution

Conclusion

Resizing Partitions on Linux

Current State and Goal

Shrinking Root

Checking Root

Extending Home

Checking Home

Conclusion

CMake on SMT32 | Episode 8: build with Docker

Dependencies to build our project

Dockerfile

Build the Docker image

Using the container from the terminal

Get a shell inside the container

Execute commands from the host

Simplify the command-line with Docker Compose

Using the container from CLion

Improve user management

Pin versions of dependencies

Pin the base image

Select more specific packages

Select exact versions of the packages

Manual installations

Assessing our solution

Treat Docker images as a separate project

Conclusion

CMake on SMT32 | Episode 7: unit tests

Why unit tests?

Running tests on board

Running tests on your computer

Project structure

A simple test case

Create the executable for the tests

Build and run the tests

Conclusion

vtable under the surface | Episode 3 - How virtual functions are actually called

What is `/boot`?

Disabling (N)RVO with `-fno-elide-constructors`

Original code without the `const`:

Original code without the `const`:

Execute the Program with `gdb`

Calling `use()` from `main()`

Alternatives to `objdump`

Disable `-Wrestrict` globally

Disable `-Wrestrict` for some files

Disable `-Wrestrict` locally

3.11 : une version importante pour le module `enum`

`ReprEnum`

L'affichage des `IntEnum`s change à cause de `ReprEnum`

`StrEnum`, pour faire comme `IntEnum` mais avec des strings

Plus de vérifications avec le décorateur `@verify`