Skip to content

feat: CHAR(36) UUID primary keys for collision-free Dolt federation#2575

Closed
trillium wants to merge 1 commit into
gastownhall:mainfrom
trillium:upstream-pr
Closed

feat: CHAR(36) UUID primary keys for collision-free Dolt federation#2575
trillium wants to merge 1 commit into
gastownhall:mainfrom
trillium:upstream-pr

Conversation

@trillium

@trillium trillium commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

Note: This change was necessary for my multi-clone Dolt federation setup, where AUTO_INCREMENT collisions made bd unusable. I'm happy to maintain this as a personal fork if beads doesn't want to go this direction, but thought it might be useful for anyone else running federated Dolt deployments.

Problem

Dolt's AUTO_INCREMENT counter is per-server-instance with no cross-clone reconciliation. When multiple Dolt clones independently write to the same table and then sync via dolt_push/dolt_pull, INSERT operations fail with:

Error 1062: duplicate primary key given: [4144]

This is documented behavior:

"Dolt branches and clones do not play well with AUTO_INCREMENT primary keys."

Six tables in beads use BIGINT AUTO_INCREMENT PRIMARY KEY and are all vulnerable in any multi-clone deployment.

How it happens

1. Clone A inserts events → gets IDs 4183, 4184, 4185...
2. Sync: dolt_push from Clone A → Clone B (rows arrive on Clone B)
3. Clone B's AUTO_INCREMENT counter does NOT advance past the pushed rows
4. Client writes to Clone B → AUTO_INCREMENT assigns 4183 → COLLISION

Impact

  • ALL write operations blocked: bd create, bd update, bd close fail on any node whose counter is behind
  • Every sync cycle re-introduces the gap — not a one-time fix
  • The only workaround was retrying failed INSERTs ~24 times to advance the counter past the gap, then repeating after every sync

Prior mitigation (insufficient)

beads#2133 added ALTER TABLE <tbl> AUTO_INCREMENT = MAX(id)+1 after every DOLT_PULL (resetAutoIncrements). This works when bd does the pull, but fails when external sync scripts use raw dolt_pull/dolt_push, when other tools write independently, or when two nodes write concurrently. Even within a single server, dolthub/dolt#7702 documents AUTO_INCREMENT race conditions.

Solution

Replace BIGINT AUTO_INCREMENT PRIMARY KEY with CHAR(36) NOT NULL PRIMARY KEY DEFAULT (UUID()) on all six affected tables, following DoltHub's official recommendation for multi-clone scenarios.

Why UUID v7

Where the application generates IDs explicitly (e.g., ImportIssueComment), we use UUID v7 (RFC 9562) — time-sorted UUIDs that preserve chronological ordering (ORDER BY id ≈ creation order).

Tables affected

Table Writes from
events Every create, update, close, reopen, label, rename
comments bd comment
issue_snapshots Compaction
compaction_snapshots Compaction
wisp_events Wisp lifecycle events
wisp_comments Wisp annotations

Migration strategy

Dolt cannot ALTER COLUMN to change a PK's type in place. Migration 010 uses a 4-step pattern for each table:

  1. Idempotency check — skip if already char(36) or table doesn't exist
  2. Add columnALTER TABLE ADD COLUMN uuid_id CHAR(36) NOT NULL DEFAULT (UUID())
  3. BackfillUPDATE SET uuid_id = UUID()
  4. Drop old PKMODIFY id BIGINT NOT NULL (remove AUTO_INCREMENT first — Dolt requires this before DROP PRIMARY KEY), DROP PRIMARY KEY, DROP COLUMN id
  5. Rename & promoteRENAME COLUMN uuid_id TO id, ADD PRIMARY KEY (id)

Dolt DDL quirk

ALTER TABLE ... DROP PRIMARY KEY fails on Dolt when the column has AUTO_INCREMENT. The workaround is to MODIFY the column first to remove it. This is not documented in Dolt's migration docs.

Breaking changes

  1. GetAllEventsSince signature: sinceID int64since time.Time (UUIDs aren't sequential, timestamps are the natural cursor)
  2. Type changes: Event.ID and Comment.ID changed from int64 to string
  3. resetAutoIncrements removed — root cause fixed, band-aid unnecessary
  4. Schema version: 6 → 7

Testing

This change passes the existing test suite — test modifications are minimal (signature updates and int64string assertions):

  • dolt_test.go: comment1.ID == 0comment1.ID == "" (string zero-value check)
  • tracker_test.go: GetAllEventsSince mock updated to accept time.Time
  • Migration is idempotent and was validated against 18 live databases across 3 nodes
  • E2E federation test: create on Clone A → sync → write on Clone B → no Error 1062

Scope of this PR and remaining work

This PR is the minimal core change — the schema migration, type updates, and interface changes needed to make UUID primary keys work. It compiles, passes tests, and functions correctly.

However, a complete adoption would benefit from updating additional files that still reference or assume the old AUTO_INCREMENT behavior. I'm pausing here to see if this direction is desired before doing more work. The files below were updated in my full working branch but are not included in this PR to keep the diff reviewable:

Files touched in the full change (58 additional files, click to expand)

Deleted — dead code after UUID migration:

  • internal/storage/issueops/create.go — inlined into DoltStore.CreateIssue
  • internal/storage/issueops/helpers.go — inlined into DoltStore
  • internal/storage/dolt/credentials.go — credential CLI routing (removed)
  • internal/storage/dolt/credentials_test.go
  • internal/storage/dolt/federation_test.go
  • internal/storage/dolt/git_remote_test.go
  • internal/storage/embeddeddolt/blocked.go — embedded backend simplified to stubs
  • internal/storage/embeddeddolt/config_metadata.go
  • internal/storage/embeddeddolt/create_issue.go
  • internal/storage/embeddeddolt/create_issue_test.go
  • internal/storage/embeddeddolt/flock.go
  • internal/storage/embeddeddolt/flock_stub.go
  • internal/storage/embeddeddolt/statistics.go
  • internal/storage/embeddeddolt/version_control.go
  • cmd/bd/backup_export_git.go — git export backup removed
  • cmd/bd/backup_export_git_test.go
  • cmd/bd/doctor/tracked_runtime.go
  • cmd/bd/doctor/tracked_runtime_test.go
  • cmd/bd/init_embedded_test.go
  • cmd/bd/simple_helpers_test.go
  • cmd/bd/prime_test.go
  • docs/GIT_INTEGRATION.md

Modified — stale references, inlined logic, cleanup:

  • internal/storage/dolt/issues.goCreateIssue inlined from issueops
  • internal/storage/dolt/wisps.gocreateWisp inlined, UUID PKs
  • internal/storage/dolt/federation.go — credential CLI routing removed
  • internal/storage/dolt/adaptive_length.go — new: birthday-paradox ID length scaling
  • internal/storage/embeddeddolt/store.go — simplified to stubs
  • internal/storage/embeddeddolt/schema.go
  • internal/config/config.go
  • internal/beads/beads.go
  • internal/beads/beads_test.go
  • cmd/bd/init.go, cmd/bd/init_contributor.go, cmd/bd/init_team.go, cmd/bd/init_git_hooks.go
  • cmd/bd/backup.go, cmd/bd/backup_git.go
  • cmd/bd/doctor.go, cmd/bd/doctor_fix.go, cmd/bd/doctor_test.go
  • cmd/bd/dolt.go, cmd/bd/import_shared.go, cmd/bd/list.go, cmd/bd/prime.go
  • cmd/bd/store_factory.go, cmd/bd/store_factory_embedded.go
  • Various test files: rename_test.go, schema_parity_test.go, wisp_gc_test.go, wisp_validation_test.go, backup_auto_test.go

Of particular note: issueops/create.go has a comment still referencing "auto-increment PK" (line 278) and PersistComments works by coincidence — it omits the id column from INSERT, so DEFAULT(UUID()) handles it. Correct behavior, but the code isn't aware of the change.

Request for review

This is a significant schema change and we want to make sure we haven't missed anything. We'd especially appreciate a second set of eyes on:

  • Any callers or code paths that still assume integer event/comment IDs
  • Edge cases in the migration (partial failures, large tables, concurrent access during migration)
  • Whether the time.Time high-water mark in GetAllEventsSince handles all the scenarios the old int64 cursor did
  • Any other tables or columns we may have overlooked that could hit the same federation collision

@trillium trillium changed the title feat: UUID primary keys for federation-safe events feat: CHAR(36) UUID primary keys for collision-free Dolt federation Mar 13, 2026
Replace BIGINT AUTO_INCREMENT with CHAR(36) UUID primary keys on 6 tables
(events, comments, issue_snapshots, compaction_snapshots, wisp_events,
wisp_comments) to eliminate duplicate PK collisions during Dolt federation.

- Change Event.ID and Comment.ID from int64 to string
- Change GetAllEventsSince to use time.Time instead of int64 high-water marks
- Add migration 010 to convert existing BIGINT PKs to UUID
- Remove resetAutoIncrements band-aid (no longer needed)
- Generate UUIDs in Go before INSERT for comments (ImportIssueComment)
- Schema version bumped from 6 to 7
@steveyegge

Copy link
Copy Markdown
Contributor

Cherry-picked to main — thanks @trillium! Exceptional work on this PR. The AUTO_INCREMENT collision problem is real and well-documented, and UUID v7 keys follow DoltHub's own recommendation for multi-clone scenarios. The migration is clean and idempotent. Welcome contribution for the Wasteland federation story.

If you want to continue with the remaining 58 files from your full branch, we'd welcome a follow-up PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants