smeso

Memory ordering and atomic operations synchronization

2024-03-04T19:00:00+01:00

Every time I need to play with atomic variables and I try to be clever and optimize them as much as possible I need to re-learn what the various memory ordering options do. It doesn't help that memory ordering is easily one of the most complex topics I ever worked with. For this reason I decided to finally write down what I (think) I know about memory ordering, making it easier for the future me to re-learn it the next time I'll need it. If you read so far, maybe it will be useful for you as well.

In this post I'll talk about C++, but everything applies identically to C and Rust (probably also to other languages). I will also completely ignore the existence of release-consume ordering: Rust never supported it, in C++ its use is discouraged since C++17 and compilers never actually made good use of it anyway. This is a pity, because the logic behind the release/consume model is actually very useful and it constitutes the basis for Linux's RCU synchronization system, but we need to be patient and wait for ISO to revise it.

What is memory ordering anyway?

Whenever you perform an atomic operation (e.g. when you use std::atomic_int) your compiler will make sure that the instructions around it will behave according to the selected memory model's rules. Both the compiler (when it generates the instructions) and the CPU (at runtime) can, in general, change the order of execution of the memory accesses around an atomic operation. Of course we don't want these optimizations to change the expected behaviour of the program, so we want them to be applied only when it's safe to do so. But how can the computer know when it is safe to re-order some instructions around an atomic operation?

It doesn't know!

For this reason, by default, the model used in C++ is the sequentially-consistent one. This model is the strictest and the most expensive, but it's also the safest. If you are building a low-latency system or you are simply affected by premature optimization, you may want to speed up your program by using more relaxed models. They may be safe to use in your use case, while being much faster than the default.

The basic building blocks: memory barriers

The way to tell the compiler and the CPU what they cannot do in respect to re-ordering memory operations is to use a memory barrier. There are at least 7 different varieties of memory barriers, but we are only interested in 2 of them:

Acquire: reads and writes that appear after this operation can not be moved before it. But anything that appears before the acquire can be moved past it. A mutex lock is an acquire operation.
Release: reads and writes that appear before this operation can not be moved after it. But anything that appears after the release can be moved before it. A mutex unlock is a release operation.

Other then the ordering restrictions imposed by the above tools, you can also always count on the fact that operations on one specific atomic variable will never be reordered in respect of each other (of course!) but accesses to 2 different atomics may be reordered if you don't use the appropriate memory fencing. Also, thread-local variables can be moved around, ignoring these barriers, because they can't cause trouble in concurrent programs.

The main memory ordering modes

The main memory orderings we are going to analyze in this post are:

Sequential consistent ordering
Release/Acquire ordering
Relaxed ordering

Sequential consistent ordering

As we said, this is the default and the safest option. It basically behaves in the way you would expect the code to behave, if you didn't know that compilers and CPUs love to re-order things around. It also works well across threads.

In details:

every read performs an acquire: every operation that appears after your read will actually be performed after it. In other words, an atomic read happens-before anything that appears after it in the code.
every write performs a release: every operation that appears before your write will actually be performed before it. In other words, anything that appears before your atomic write in the code, will happen-before the write itself.
every read-modify-write perform both acquire and release: the operation acts as a wall that prevents moving things across it from both sides.
a total single modification ordering is enforced across all threads for all atomic operations. In other words, all atomic operations will be observed in the same order by all threads.

The performance implications of the above rules are not just related to the inability to re-order instructions in the most effective way, but also to the cache coherency issues that, depending on the architecture, can force expensive cache operations on all cores to keep everything in sync.

e.g.

This example shows the acquire and release operations at work:

#include <atomic>
#include <cassert>
#include <thread>

std::atomic<bool> x = {false};
int y = 0;

void thread_1()
{
  y = 1;
  x.store(true, std::memory_order_seq_cst);
  // y = 1 always happens-before the store
}

void thread_2()
{
  // the load always happens-before the y == 1 check
  if (x.load(std::memory_order_seq_cst))
    assert(y == 1);  // This _never_ fails and there is no race
}

int main()
{
  std::thread a(thread_1);
  std::thread b(thread_2);
  a.join();
  b.join();
  return 0;
}

This example shows how multiple atomics work consistently across all threads:

#include <atomic>
#include <cassert>
#include <thread>

std::atomic<int> x = 0;
std::atomic<int> y = 0;

void thread_1()
{
  // The following stores will always happen
  // strictly in this order from the point
  // of view of every thread in the program
  y.store(2, std::memory_order_seq_cst);
  x.store(1, std::memory_order_seq_cst);
}

void thread_2()
{
  // If x was a regular variable this loop would be UB
  // and it will likely optimized away.
  // But this is not the case.
  while (x.load(std::memory_order_seq_cst) != 1)
    ;
  assert(y.load(std::memory_order_seq_cst) == 2);  // This _never_ fails
  y.store(1, std::memory_order_seq_cst);
}

void thread_3()
{
  if (y.load(std::memory_order_seq_cst) == 1)
    assert(x.load(std::memory_order_seq_cst) == 1);  // This _never_ fails
}

int main()
{
  std::thread a(thread_1);
  std::thread b(thread_2);
  std::thread c(thread_3);
  a.join();
  b.join();
  c.join();
  return 0;
}

Please note that I'm using std::memory_order_seq_cst explicitly for clarity. Being the default mode, you don't really need to specify it.

Release/Acquire ordering

This is a slightly more relaxed mode compared to the previous one. The only actual difference is that there is no total single modification ordering. You still need to perform an acquire for reads and a release for writes. The interesting thing about this mode is that it can be almost completely free. Because the acquire/release semantic is implicit on some architectures (e.g. x86) and the compiler doesn't have to produce any memory barrier instruction. Re-ordering of non-atomic (or atomic relaxed) operations still follows the same rules as per the sequential consistent ordering, it's only atomic operations on different atomics that can be observed as happening in different orders by different threads.

e.g.

This example shows the lack of total ordering:

#include <atomic>
#include <cassert>
#include <thread>

std::atomic<bool> x = {false};
std::atomic<bool> y = {false};

void thread_1()
{
  x.store(true, std::memory_order_release);
}

void thread_2()
{
  y.store(true, std::memory_order_release);
}

void thread_3()
{
  // this assert _can_ succeed
  assert(x.load(std::memory_order_acquire) &&
         !y.load(std::memory_order_acquire));
}

void thread_4()
{
  // this assert _can_ also succeed in the same run
  // because different threads can see the effects
  // of the stores in different orders
  assert(!x.load(std::memory_order_acquire) &&
         y.load(std::memory_order_acquire));
}

int main()
{
  std::thread a(thread_1);
  std::thread b(thread_2);
  std::thread c(thread_3);
  std::thread d(thread_4);
  a.join();
  b.join();
  c.join();
  d.join();
  return 0;
}

Relaxed ordering

This is the simplest kind of operation: no synchronization happens. Memory accesses around it can be re-ordered in any way. You can always count on the fact that operations on one specific atomic variable will never be reordered in respect of each other. This is useful when what you want is just an atomic variable and you don't want to use it as a synchronization mechanism. Relaxed operations are not 100% free, they may still trigger some cache flushing.

e.g.

This example shows how everything can be re-ordered around relaxed ops:

#include <atomic>
#include <cassert>
#include <thread>

int x = 0;
std::atomic<bool> y = {false};

void thread_1()
{
  x = 1;
  y.store(true, std::memory_order_relaxed);
}

void thread_2()
{
  if (y.load(std::memory_order_relaxed))
    assert(x == 1); // this _can_ fail
}

int main()
{
  std::thread a(thread_1);
  std::thread b(thread_2);
  a.join();
  b.join();
  return 0;
}

What about volatile?

Accesses to volatile glvalues (at some point I should write a post about C++ value categories) won't be re-ordered past each other (or any other observable side-effect e.g. I/O, modifying an object etc). But that's it: anything else can be re-ordered around them, different threads can see changes to different volatiles happening in different orders, and they are not atomic. There are only 2 legitimate uses for volatiles:

volatile static std::sig_atomic_t to set a flag in a signal handler
to access some memory-mapped I/O register, if you are doing some very low-level/embedded stuff.

If you are trying to use them for anything else, you have a subtle bug lurking in the shadows, waiting for the right moment to attack you.

Adults only: intermingling modes

If you didn't have your coffee yet, maybe this is the moment to go and get a hot one: things are about to get hairy.

Mixing different synchronization primitives is very complex. The risks of getting everything wrong might outweigh the performance benefits. But if you are looking for new reasons to bang your head against the screen, I'll make you happy in a moment.

A quite common use case for intermixing different modes is reference counting. When you increment the counter, you don't need to know what value the counter had before or what value it has after the increment, you just want the value to be incremented. You need the operation to happen atomically, but ordering and synchronization don't matter. When it's time to decrement it, things are bit different. Now you want to know what the resulting value is, because you need to free some resource when the counter reaches 0. This is a typical case of when certain operations on an atomic are safe to be performed in relaxed mode (i.e. the increment) while some other operation will need to use acquire/release mode (i.e. decrement).

e.g.

This example shows a naive (and slightly buggy) reference counting system:

#include <atomic>
#include <cassert>
#include <thread>

char * ptr = nullptr;
atomic_size_t counter;

void increment()
{
  // We just care about atomicity here, nothing else
  counter.fetch_add(1, std::memory_order_relaxed);
}

void decrement_and_delete()
{
  // We need to use _release_ here because we want to make sure
  // that other accesses (e.g. increment) to the counter won't be
  // moved past this point.
  if (counter.fetch_sub(1, std::memory_order_release) == 1) {
    // If we reached this it means that counter is now 0
    // So we need to free ptr, but we need to make sure
    // this won't be moved before the decrement.
    // We could have obtained the same guarantee using
    // std::memory_order_acq_rel in fetch_sub.
    // But we only need to use _acquire_ when counter reaches 0
    // so it is more efficient to do it separately and only in
    // that case.
    // Also, because we don't have any other atomic operation
    // to perform, we can use std::atomic_thread_fence that
    // simply applies the memory fencing on its own.
    std::atomic_thread_fence(std::memory_order_acquire);
    delete ptr;
  }
}

void thread_1()
{
  increment();
  // do something with ptr
  decrement_and_delete();
}

void thread_2()
{
  increment();
  // do something with ptr
  decrement_and_delete();
}

int main()
{
  ptr = new char(10);
  increment();
  std::thread a(thread_1);
  std::thread b(thread_2);
  decrement_and_delete();
  a.join();
  b.join();
  return 0;
}

Kudos to you if you can spot the bug.

Conclusion

We made it! We reached the end and, hopefully, we also learned something useful! Feel free to let me know what you think about this post and please let me know if you found any error!

ClickHouse: don't roll your own crypto

2024-03-03T15:22:00+01:00

ClickHouse is a column-oriented database management system that is designed for high-performance analytics. It is known for its speed and efficiency in processing large volumes of data, making it a popular choice for companies looking to analyze massive datasets in real-time.

(the above text was written by some AI, because I couldn't be bothered to think of anything, but I promise I'll write the rest of the post myself, maybe)

Working as a CH contributor, a while ago, I stumbled upon an issue in the way the disk encryption feature was implemented.

Some context

When you use CH's disk encryption every file that is written to disk will be encrypted using AES in CTR mode. The same key is used for every file and a different nonce is generated for each file and stored in the file header.

CTR mode

When you use a cipher in CTR mode, there is one thing you need to be very careful about: never re-use the same pair of key and nonce+counter. Wikipedia explains very well how CTR works, so I won't repeat it here. But, summarizing it, we can say that it works by generating a keystream that is XORed with the plain-text to produce the cipher-text. A keystream is simply an "infinite" sequence of random bytes that is generated from the key and nonce+counter pair. So it will always be the same, if the said pair is the same. Re-using this infamous pair with 2 different plain-text is dangerous because of some interesting property that the XOR operation has.

If you have C1 = P1 ^ K it's impossible to recover P1 from C1 unless you know K, this is essentially a one-time-pad cipher. But if you also have C2 = P2 ^ K then, because of the mathematical properties of XOR, you can recover the K simply doing K = C1 ^ C2 and now you can decrypt both C1 and C2.

Long story short, never use key and nonce twice or it will be very easy to decrypt your data.

The problem

We said before that CH usese the same key for every file (which is fine) and generates a new nonce for each file, which would be okay, if the nonce was actually generated randomly. The code looked like this:

InitVector InitVector::random()
{
    std::random_device rd;
    std::mt19937 gen{rd()};
    std::uniform_int_distribution<UInt128::base_type> dis;
    UInt128 counter;
    for (auto & i : counter.items)
        i = dis(gen);
    return InitVector{counter};
}

There are some issues with this code:

The source of entropy is std::random_device which isn't required by the C++ standard to actually return random numbers, in most cases it will and it may even return cryptographically-secure random numbers, but no portable solution should rely on it.
std::mt19937 is seeded using a 32bit integer and returns a 32 bits integer. Also it isn't a CSPRNG.

The latter point is the most worrying. In fact, with only 2^32 different initial states, will can very easily end up re-using nonces. In fact, because of the birthday attack it only takes 65536 nonce generations to have a 50% probability of re-using nonces. And 65536 files are not many files for CH.

Conclusion

I reported the issue upstream and provided a fix that was merged and backported to various versions. This happened a long time ago, so I don't expect any vulnerable CH instance to still be around. Hence why I thought it was a good idea to write about it now.

But please note that simply upgrading CH is not enough to fix the problem, because any file that was encrypted with the buggy code is still in danger. So you also need to re-encrypt everything!

Never roll your own encryption! As you saw in this post, even the best developers can make mistakes with these things!

Kudos to the upstream CH team for being very responsive and helpful, they even offered my a bounty (which I had to decline).

MIPS stacktrace: an unexpected journey

2024-03-02T14:30:00+01:00

Automatically receiving a stacktrace when your C program crashes isn't rocket science. But this time it was more difficult than I expected. This is a short recollection of the things I found out few years ago. This post assumes that the reader has some basic knowledge about functions' calling conventions, CPU registers, and assembly.

Some context

A C program running on Linux was randomly crashing on one specific embedded device deployed on the other side of the world. The device architecture was MIPS32. We needed a system to asynchronously receive reports with as much details as possible (i.e. a stacktrace). The device was using glibc, ideally we wanted a solution that could also work with other standard libraries (e.g. musl and uClibc) but even a non-portable solution was okay, at least to fix this one problem. Using external dependencies, especially if they were large, wasn't an option.

The simple solution

glibc already has backtrace(3) and backtrace_symbols_fd(3). They are easy to use and they will certainly work very well! glibc simply calls libgcc. Any code I’ll ever write will never understand the code generated by GCC better than libgcc!

Well, this may be true when you are on x86_64, but on some other architectures like MIPS32 those functions don't work at all.

Some notes on MIPS

Just in case you are not too familiar with MIPS, I wanted to add some notes about how it works with gcc on Linux.

The return address of the current function is stored in the $ra register
When entering a new function the "old" return address is pushed onto the stack and is the last thing just before the start of the stack frame of the new function.

To reconstruct the stack trace we need to recover all the return addresses in sequence. We can start from $ra and then go backwards, pulling the return addresses from the end of each stack frame.

The problem

It turns out that backtrace(3) does the same thing that too many blog posts on the Internet recommend to do: unwind the stack using the frame pointer register to figure out the position of the previous function's return address. This makes perfect sense, the frame pointer (aka $fp or $30 on MIPS) is usually a register designed to help debuggers to refer to local variables and other information stored on the stack (e.g. the previous function return address) using constant offsets. In theory, while the stack pointer (aka $sp) always points at the top of the stack, the $fp should point at the beginning of the current stack frame and should not move from there. If you want to retrieve the return address using the $sp, you need to know how much stuff you put on your stack since you entered the current stack frame, this depends on: what function you are in, how many automatic variables this functions is using, what type are those functions, and how many bytes do those types use. It would be very inconvenient to work with the $sp during debug, so we are very lucky to have the $fp! Using the $fp we can always retrieve the return address of the previous function without any complex operation! The offset between the $fp and the return address is the same constant for all functions in all programs!

... Or is it?

Well... it turns out it isn't.

In fact, when using GCC on Linux on MIPS32, the frame pointer just works exactly like another stack pointer: it's completely useless! I'm not 100% sure about the reason behind this choice, but I think it could be related to the fact that, on architectures with (relatively) small registers, it would be difficult to reference the top of the stack using a real $fp, but still it sounds like the wrong thing to do.

The funniest thing is that, backtrace(3) implementation from libgcc seems to ignore how gcc works in this context and it just returns random values.

The real solution

We can get rid of the $fp completely and just work with $sp, but how can we figure out the correct offset to use with $sp for any function?

You may not like the answer (or maybe you will) but the only way to know where the beginning of the stack frame is... is to jump into the actual code of the function and parse the opcodes to figure out how much $sp was decremented by the compiler.

Here is one way to do it:

#include <link.h>
#include <sys/ucontext.h>

static inline void my_backtrace(ucontext_t* c)
{
    unsigned long *ra;
    unsigned long *fp;
    unsigned long *sp;
    size_t ra_offset;
    size_t stack_size;
    int reached_start = 0;
    int first_time = 1;

    pc = (unsigned long*)(unsigned long)c->uc_mcontext.pc;
    ra = (unsigned long*)(unsigned long)c->uc_mcontext.gregs[31];
    fp = (unsigned long*)(unsigned long)c->uc_mcontext.gregs[30];
    sp = (unsigned long*)(unsigned long)c->uc_mcontext.gregs[29];
    print(pc - 1);
    print(ra - 1);

    while (!reached_start) {
        int using_fp = 0;
        ra_offset = 0;
        stack_size = 0;
        for (unsigned long* addr = ra;
             (ra_offset == 0 || stack_size == 0) && !reached_start;
             --addr) {
            switch (*addr & 0xffff0000) {
                case 0x27bd0000:
                    // found addiu sp, sp, stack_size
                    stack_size = abs((short)(*addr & 0xffff));
                    break;
                case 0xafbf0000:
                    // found sw ra, ra_offset(sp)
                    ra_offset = (short)(*addr & 0xffff);
                    break;
                case 0x03a00000:
                    if (0x03a0f000 == (*addr & 0xffffff00)) {
                        // found pseudo instruction move fp, sp
                        using_fp = 1;
                    }
                    break;
                case 0x03e00000:
                    if (0x03e00025 == *addr) {
                        // found move zero,ra
                        // so we found the start
                        reached_start = 1;
                    }
                    break;
                default:
                    break;
            }
        }

        if (!ra_offset) {
            break;
        }

        if (using_fp && first_time) {
            sp = fp;
        }
        first_time = 0;

        ra = *(unsigned long**)((unsigned long)sp + ra_offset);
        if (using_fp) {
            sp = *(unsigned long**)((unsigned long)sp + ra_offset - 4);
        } else {
            sp = (unsigned long*)((unsigned long)sp + stack_size);
        }

        print(ra - 1);
    }

}

Symbolizing the addresses

It would be nice to be able to translate the addresses, that we just found, to actual function names. This isn't usually the funniest thing to do, but after what we just did, it seems trivial. We can use dl_iterate_phdr(3) to look into every loaded shared object. Once we found the shared object that contains our address, we can walk through its ELF sections and look at its .symtab to find the function name.

Here is an example of how to do it:

struct file_match {
    const char *file;
    void *address;
    void *base;
};

static int
find_matching_file(struct dl_phdr_info *info,
                   size_t size,
                   void *data)
{
    struct file_match *match = data;
    long n;
    const ElfW(Phdr) *phdr;
    /* This code is modeled from Gfind_proc_info-lsb.c:callback() from libunwind */
    ElfW(Addr) load_base = info->dlpi_addr;
    phdr = info->dlpi_phdr;
    for (n = info->dlpi_phnum; --n >= 0; phdr++) {
        if (phdr->p_type == PT_LOAD) {
            ElfW(Addr) vaddr = phdr->p_vaddr + load_base;
            if (match->address >= vaddr && match->address < vaddr + phdr->p_memsz) {
                match->file = info->dlpi_name;
                match->base = info->dlpi_addr;
            }
        }
    }
    return 0;
}

static inline void
symbolize(const char* progname, void* addr)
{
    struct file_match match = { .address = addr };
    dl_iterate_phdr(find_matching_file, &match);
    if (match.file && strlen(match.file)) {
        fprintf(stderr, " %s(+%p)\n", match.file, addr - match.base);
    } else {
        fprintf(stderr, " %s\n", progname);
    }
    fflush(stderr);
}

Wouldn't it be cool if we also retrieved the source file name and the line? To do that we need to use the information provided in the .debug_line section using the DWARF format. To use as little space as possible DWARF doesn’t simply store a list of address-to-line mappings. It stores a line number program, which is a serialized finite state machine that can be used to find out line numbers and more. Parsing DWARF is left as an exercise for the reader. Alternatively we can manually invoke addr2line.

Conclusion

In the end the bug was found and fixed and everyone lived happily ever after.

I learned, once again, that features/bugs can sometime happen in your most trusted dependency and one should never refrain from doubting the correctness of the most respected software.

I also learned, once again, that the Internet is full of misleading information and blog posts written with authority by people that never actually tried to do the things that they talk about. As of today if you try to lookup information about the frame pointer on MIPS32 it will be very hard to find any mention at all of the issues outlined in this post. So, never trust a blog post! Not even this one! Your combination of compiler and libc might behave differently and have different, new, exciting issues that will ruin your day!

P.S.

This post made HN's front page and received a few comments! You can add your comments here as well.

Hello world!

2017-10-06T10:42:00+02:00

This is my first post!