It isn't correct.

We don't need acquire metalock here, it's updating tx's local meta page.

+1, i also thought the same, but u know better 👍🏽

OK, I see now, we're filling the buf basically. But then how about the writeAt below? It'll flush the buf onto the page 0 or 1 and that's exactly where (*DB) meta() reads data from.

Or that's the question of how pwrite() updates the page. Not sure about that.

iiuc bbolt uses CoW, which means the updates will be written to a new pages, and only upon commit the metadata list will be updated to point to the new pages

@ahrtr know better these details 👍🏽

But then how about the writeAt below?

Only one write TXN is allowed at a time.

Or that's the question of how pwrite() updates the page. Not sure about that.

pls refer to https://github.com/ahrtr/etcd-issues/blob/master/docs/cncf_storage_tag_etcd.md#storage-boltdb-feature

Only one write TXN is allowed at a time.

That I know, that's rwlock. But my concern here is concurrent RO and RW transactions and initialization specifically. To initialize an RO transaction we need to get the meta page and copy it, like in

#include <fcntl.h>
#include <pthread.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

size_t const pageSize = 4096;
char *mm;

void *reader(void *arg)
{
        for (;;) {
                int i;
                char first, next;

                for (i = 0; i < pageSize; i++) {
                        if (i == 0) {
                                first = mm[i];
                        } else {
                                next = mm[i];
                                if (next != first) {
                                        printf("reader mismatch, first %hhX, read %hhX at %d", first, next, i);
                                        exit(11);
                                }
                        }
                }
        }
}

int main()
{
        char b[pageSize];
        int fd, r, i;
        ssize_t written;
        pthread_t pth;

        fd = open("data", O_RDWR | O_CREAT);
        if (fd < 0) {
                printf("bad fd: %d\n", fd);
                exit(10);
        }
        r = ftruncate(fd, pageSize);
        if (r < 0) {
                printf("bad ftruncate: %d\n", r);
                exit(10);
        }
        mm = mmap(NULL, pageSize, PROT_READ, MAP_SHARED, fd, 0);
        if (mm == NULL) {
                printf("mmap failed\n");
                exit(10);
        }
        r = pthread_create(&pth, NULL, reader, NULL);
        if (r < 0) {
                printf("bad pthread_create: %d\n", r);
                exit(10);
        }
        for (i = 0; i < 1000; i++) {
                memset(b, i%256, pageSize);
                written = pwrite(fd, b, pageSize, 0);
                if (written != pageSize) {
                        printf("can't write properly\n");
                        exit(10);
                }
        }
        printf( "Done\n" );
        return 0;
}

With one reader thread doing its things via mmapped region and writer doing its job via pwrite() (which is what (*File) WriteAt() does under the hood). To me it gives things like reader mismatch, first 4, read 5 at 1478 or reader mismatch, first 6, read 7 at 1115 easily.

Is this a correct analogy or am I missing something?

Yes, it's a valid point, although it's highly unlikely in practice. We have two meta pages, usually Readonly TXN won't read the same meta page as the RW TXN at the same time; even it does, if it reads some dirty data or partially written data, then the checksum won't match, so it still fallback to the other meta page. It's exactly the reason why we never see any issue in the concurrent test

it's highly unlikely in practice

I agree with that, the amount of data is small, there is validation, yet it's still a race that can happen. Threads can be stalled at any point on a busy machine as well in which case chances can get higher. Like consider validation done successfully in meta(), but then the reader thread paused and waiting in the queue while writers change the contents before or in the middle of reader doing a Copy().

There might be some performance penalty under high concurrent readonly TXNs (especially short live TXNs)

writeAt() should be fast, it's a single page. And the next patch will make readers work much better anyway. The thing is that I want to leverage this lock in that patch now.