Reftables support by pks-gitlab · Pull Request #7117 · libgit2/libgit2

pks-gitlab · 2025-08-04T14:23:25Z

This pull request introduces support for the reftable backend. It depends on the following list of pull requests:

Introduction of the "refFormat" extension #7102 introduces the refFormat extension.
Fix handling of pseudorefs with different refdb backends #7111 fixes our handling of pseudo and root refs to align with modern Git.
Test fixes to prepare for the reftable backend #7113 adjusts our tests to not access references via the filesystem directly anymore.
cmake: disable warnings for operands with different enum types #7114 is a semi-related fix, but our CI is currently broken due to changes in the MSVC compiler version.

Other than that, all tests are passing and there aren't any memory leaks. But some polishing is still required for non-Linux platforms.

This pull request supersedes #5462.

csware · 2025-10-21T11:04:03Z

Any news on this PR?
@ethomson

pks-gitlab · 2025-10-21T11:51:57Z

Any news on this PR? @ethomson

@csware I guess the single biggest thing that these PRs need is review. If you have the time to help review I think that would help quite a bit to move this forward.

Other than that I also plan to have some of the ICs in my team at GitLab review those PRs in the next Git release cycle.

pks-gitlab · 2025-10-22T04:51:55Z

@csware Thanks for your initial feedback! One note though: I'd recommend to review the prerequisite PRs listed in the description first. This PR here is basically a merge of all of these plus the reftable backend.

In our util headers we expose the `GIT_INLINE()` macro to declare a function as inline, but we don't provide a way to access the keyword directly. Expose a new `GIT_INLINE_KEYWORD` to fill this gap. This define will be used in a subsequent commit.

Import the reftable library from commit 4fee6ff3b2 (Merge branch 'ps/reftable-portability', 2026-04-08). This is an exact copy of the reftable library. The library will be wired into libgit2 over the next couple of commits.

pks-gitlab · 2026-04-09T04:51:17Z

The patch series that makes the reftable library more portable has been merged upstream, so I've updated the pull request with the exact version that we have in upstream Git now.

knayak-gitlab · 2026-04-16T09:28:43Z

Some questions/comments from me. I mostly focused on the reftable code and not the infrastructure around it. Thanks.

knayak-gitlab · 2026-04-15T11:52:59Z

+
+#define poll(fds, fds_len, timeout) p_poll(fds, fds_len, timeout)
+
+#define REFTABLE_ALLOW_BANNED_ALLOCATORS


I couldn't understand what this header is for?

@knayak-gitlab Do you mean the header specifically, or this specific define here?

This particular define. Nvm though I didn't check the other files. All good here.

knayak-gitlab · 2026-04-15T11:56:50Z

 uint32_t reftable_rand(void)
 {
-	return git_rand(CSPRNG_BYTES_INSECURE);
+	return rand();


Shouldn't we use git_rand_next() here?

Is it because of the type? but that shouldn't matter for when converting uint64_t to uint32_t for a random number no?

@knayak-gitlab It's mostly that I missed git_rand_next(). Will adapt.

knayak-gitlab · 2026-04-15T12:48:43Z

+
+	*out = NULL;
+
+#ifdef GIT_EXPERIMENTAL_SHA256


So sha256 support itself is experimental in libgit2?

@knayak-gitlab Yes, exactly.

knayak-gitlab · 2026-04-15T13:45:55Z

+	if ((error = refdb_reftable_check_ref(data->stack, data->ref->name, data->expected_oid, data->expected_target)) < 0) {
+		data->error = error;
+		git_error_set(GIT_ERROR_REFERENCE, "old reference value does not match");
+		goto out;
+	}
+
+	if ((error = refdb_reftable_check_refname_available(data->stack, NULL, data->ref->name, data->force)) < 0) {
+		data->error = error;
+		goto out;
+	}


Both these function independently read the ref from the stack, wouldn't it be better if we supplied it?

knayak-gitlab · 2026-04-16T09:08:09Z

+
+	/* We do not (yet?) support renames across different worktree stacks. */
+	if (git_repository_is_worktree(backend->repo) &&
+	    git_reference__is_per_worktree_ref(old_name) != git_reference__is_per_worktree_ref(new_name)) {


But this doesn't validate against two different worktrees, it only ensures both the names are worktree refs.

@knayak-gitlab Hm, right. As far as I can see this can only be the case when we have two named worktree refs (refs/worktree/) that name different worktrees. So something like this should address this, right?

diff --git a/src/libgit2/refdb_reftable.c b/src/libgit2/refdb_reftable.c index 8b0219a38..e9013f739 100644 --- a/src/libgit2/refdb_reftable.c +++ b/src/libgit2/refdb_reftable.c @@ -1311,13 +1311,46 @@ static int refdb_reftable_rename(git_reference **out, data.out = out; data.error = 0; - /* We do not (yet?) support renames across different worktree stacks. */ + /* + * Are we in a worktree, but the references aren't both per-worktree + * refs? In that case they cannot be part of the same worktree, but we + * don't support cross-worktree renames yet. + */ if (git_repository_is_worktree(backend->repo) && git_reference__is_per_worktree_ref(old_name) != git_reference__is_per_worktree_ref(new_name)) { error = GIT_EINVALID; goto out; } + /* + * Otherwise, are both refs named worktree refs, but for different + * worktrees? If so, we need to bail out, as well. + */ + if (!git__prefixcmp(old_name, "refs/worktree/") && + !git__prefixcmp(new_name, "refs/worktree/")) { + const char *old_wtname_end = strchr(old_name + strlen("refs/worktree/"), '/'); + const char *new_wtname_end = strchr(new_name + strlen("refs/worktree/"), '/'); + size_t old_prefix_len, new_prefix_len; + + if (!old_wtname_end || !new_wtname_end) { + error = GIT_EINVALID; + goto out; + } + + old_prefix_len = p_strnlen(old_name, old_wtname_end - old_name); + new_prefix_len = p_strnlen(new_name, new_wtname_end - new_name); + + if (old_prefix_len != new_prefix_len) { + error = GIT_EINVALID; + goto out; + } + + if (git__strncmp(old_name, new_name, old_prefix_len)) { + error = GIT_EINVALID; + goto out; + } + } + if ((error = refdb_reftable_stack_for_refname(&data.stack, backend, old_name)) < 0) goto out;

Yeah this looks good!

knayak-gitlab · 2026-04-16T09:12:32Z

+			creation = old_log;
+			git__free(creation.refname);
+			creation.refname = git__strdup(new_name);
+			memset(&old_log, 0, sizeof(old_log));


Why do we need to memset only during rename? Why do we need to memset at all?

@knayak-gitlab The idea here is that we're passing ownership of the old_log over to the updates array. I'll restructure this a bit and add a comment to explain.

knayak-gitlab · 2026-04-16T09:23:33Z

+	if ((error = refdb_reftable_stack_for(&stack, backend, REFDB_REFTABLE_STACK_MAIN)) < 0)
+		goto out;
+
+	if ((error = reftable_stack_compact_all(stack->stack, NULL)) < 0) {


So no way to do geometric compression?

@knayak-gitlab I guess this can be added to the interfaces at a later point.

pks-gitlab · 2026-04-16T12:20:52Z

@knayak-gitlab Thanks for your review, back to you! 🏓

While the reftable library is mostly decoupled from the Git codebase, some specific functionality intentionally taps into Git subsystems. To make porting of the reftable library easier all of these are contained in "system.h" and "system.c". Reimplement those compatibility shims so that they work for libgit2.

Wire up the reftable library with CMake. At the current point in time the library is not yet plugged into libgit2 itself.

Extract the function `is_per_worktree_ref()` from the "files" backend and expose it via "refs.h". This function will be reused by the "reftable" backend.

Implement the reftable backend that is used to read and write reftables. The backend is not yet used anywhere after this commit.

Wire up the "reftable" storage format via the newly implemented reftable backend.

Generate test resources for reftables. These resources are basically the Git repositories we already have, but converted to use the "reftable" format. For most of the part, this conversion is done by executing `git refs migrate`. A couple notes: - This require a recent Git upstream version with not-yet-upstreamed patches due to a bug in `git refs migrate` with reflogs. - The migration command does not yet support repositories with worktrees. Those were converted by first removing the worktrees, migrating the refs and then recreating them. - The HEAD_TRACKER reference in testrepo.git is not recognized as a root ref and is thus not automatically migrated. - testrepo.git has an empty reflog for refs/heads/with-empty-log that does not get migrated.

Wire up reftable tests so that they can be executed by setting the `CLAR_REF_FORMAT` environment variable. This only catches tests that use `cl_git_sandbox_init()`, but that should cover most of our tests. So this infrastructure isn't perfect, but for now it's good enough. We may want to iterate on it in the future.

Wire up three new CI jobs that exercise libgit2 with the reftable backend.

knayak-gitlab · 2026-04-17T08:37:20Z

@knayak-gitlab Thanks for your review, back to you! 🏓

The changes look good now! Thanks.

jdidion · 2026-05-03T15:11:31Z

I have a project that is blocked on this PR landing.

In short - users who initialize repos with git init --ref-format=reftable hit Not a repository on startup; we emit a friendly error but cannot open the repo via git2::Repository::discover until libgit2 supports reftable.

I am happy to test my project against a branch build if that would help. I am also willing to contribute an AI code review (Opus + GPT 5.5 + Gemini working as a “crew”) if that would be of help.

Thanks to @pks-gitlab for carrying this through multiple iterations.

ethomson · 2026-05-06T21:02:28Z

Thanks @pks-gitlab for pushing on this!

## Summary - Remove the `git2` crate and its C dependencies (`libgit2-sys`, `libz-sys`), replacing all remaining usage with the git CLI and other previously used rust crates - Replace `git2::Patch` in `buffer_diff` with `imara-diff` (used elsewhere in the codebase for word diffing) - Replace `git2::Oid` with `[u8; 20]` and the `hex` crate - Replace `git2::Repository` in `RealGitRepository` with stored paths and git CLI calls via the existing `GitBinary` - Fixes a bug where linked worktree git dir events could cause the repository entry to be dropped ### Motivation libgit2 does not support the [reftable](https://github.blog/2024-04-29-highlights-from-git-2-45/#preliminary-reftable-support) storage format that was introduced in git 2.45 (see [libgit2#7117](libgit2/libgit2#7117)), meaning that git operations in the app using these (like git status, branch names, diffs, e.g.) appeared broken for any repository using `--ref-format=reftable`. [Git 3 will use reftables by default](https://www.deployhq.com/blog/git-3-0-on-the-horizon-what-git-users-need-to-know-about-the-next-major-release), so this needs to be supported eventually. Most operations used the git binary already, but there were still a handfull of stragglers using libgit2. By swapping out the remaining uses, we can remove the dependancy of libgit2 and eliminate ~30k lines of vendored C and the associated build complexity (cmake, libz, pkg-config), as well as ensure that all git operations go through similar codepaths. Closes zed-industries#46747 Closes zed-industries#45702 fixes ZED-76X fixes ZED-73N ### Notes - `reload_index` is removed from the `GitRepository` trait since both implementations were no-ops (the CLI always reads from disk) - `change_branch` is simplified to `git checkout <name>`, which natively handles local/remote branch resolution - `load_index_text` and `load_committed_text` no longer explicitly filter symlinks (the old code returned `None` for symlinked entries) ## Test plan - [x] `cargo test -p git` (34 tests) - [x] `cargo test -p buffer_diff` (14 tests) - [x] `cargo test -p worktree -- test_linked_worktree_git_dir_events_do_not_panic` - [x] `cargo test -p util` (108 tests) - [x] `cargo test -p project -- test_file_status test_repository_subfolder_git_status test_update_gitignore` - [ ] Manual testing of diff gutter, staging, branch switching, remote operations - [ ] Manual testing with a reftable repository Release Notes: - Fixed git integration not working with repositories using the reftable reference storage format --------- Co-authored-by: Anthony Eid <anthony@zed.dev>

pks-t mentioned this pull request Aug 4, 2025

WIP: Reftables support #5462

Closed

pks-gitlab force-pushed the pks/reftables-support branch 15 times, most recently from fba6f0e to b122fb0 Compare August 5, 2025 12:04

sandyarmstrong mentioned this pull request Sep 3, 2025

GitRepository: Support additional upstream extensions dotnet/sourcelink#1469

Closed

Syphdias mentioned this pull request Sep 7, 2025

Does not work with --ref-format=reftable romkatv/gitstatus#455

Open

pks-gitlab mentioned this pull request Sep 8, 2025

reftable support for libgit2 #5352

Open