feat: replicate control requests through WAL via state machine by mattisonchao · Pull Request #882 · oxia-db/oxia

mattisonchao · 2026-02-09T03:25:46Z

Motivation

Enable the leader to replicate control commands (e.g., feature enablement) through the WAL alongside regular writes, so that features negotiated between replicas are applied consistently across all nodes via the replication log rather than in-memory state.

Modification

Add ControlRequest and FeatureEnableRequest protobuf messages to the LogEntryValue oneof, allowing control commands to be serialized into WAL entries alongside write requests.
Introduce a statemachine package with a Proposal interface (WriteProposal, ControlProposal) where each proposal type knows how to serialize itself to a log entry and apply itself to the DB.
Add ApplyLogEntry function for the follower/replay path that deserializes and applies WAL entries.
Refactor leader controller: rename write to propose to reflect the generalized proposal flow, split BecomeLeader to propose feature enablement outside the lock, and delegate to proposal.Apply() after quorum ack.
Refactor follower controller: replace inline write-processing with statemachine.ApplyLogEntry, hold read lock during apply, and expose IsFeatureEnabled/Checksum methods.
Make DB checksum computation opt-in via EnableFeature(FEATURE_DB_CHECKSUM) — checksum is only computed when the feature is enabled or a non-zero checksum already exists.
Use read locks instead of exclusive locks for read-only operations (Checksum, ApplyLogEntry on follower, IsFeatureEnabled on leader).
Rename FEATURE_FINGERPRINT to FEATURE_DB_CHECKSUM.
Add unit tests for state machine and proposal logic, and an integration test verifying feature enablement replication and checksum consistency across 3 replicas.

common/proto/storage_vtproto.pb.go

Copilot

Pull request overview

This PR extends Oxia’s replication/WAL format to carry control-plane commands (feature enablement) and introduces a shared state-machine layer to apply both write and control log entries consistently across leader and follower paths, enabling negotiated activation of the DB checksum feature.

Changes:

Add ControlRequest{FeatureEnableRequest} support to LogEntryValue and implement application logic via a new controller/statemachine package.
Wire leaders/followers to apply WAL entries through the state machine; leaders can now propose control entries to enable features post-election.
Rename negotiated feature FEATURE_FINGERPRINT to FEATURE_DB_CHECKSUM and expose checksum/feature state via controllers and tests.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
`tests/control/control_request_test.go`	Adds an integration-style test for feature enablement and checksum consistency across replicas.
`oxiad/dataserver/server.go`	Exposes `GetShardDirector()` for test/introspection access to shard controllers.
`oxiad/dataserver/database/db.go`	Adds feature enable tracking and conditional checksum persistence/read APIs on the DB.
`oxiad/dataserver/controller/statemachine/state_machine.go`	New shared apply-layer for proposals and WAL log entries (write + control).
`oxiad/dataserver/controller/statemachine/state_machine_test.go`	Unit tests for applying proposals and WAL entries (write + feature enable).
`oxiad/dataserver/controller/statemachine/proposal.go`	Proposal abstraction for write and control entries and WAL serialization.
`oxiad/dataserver/controller/statemachine/proposal_test.go`	Unit tests for proposal creation and WAL encoding.
`oxiad/dataserver/controller/lead/leader_controller.go`	Refactors write path to `propose()`, replays WAL via state machine, and proposes feature-enable control entries post-election; exposes checksum.
`oxiad/dataserver/controller/lead/leader_controller_test.go`	Updates feature negotiation tests for the renamed DB checksum feature.
`oxiad/dataserver/controller/follow/follower_controller.go`	Applies committed WAL entries via state machine; exposes checksum and feature-enabled state.
`oxiad/coordinator/controller/shard_controller_test.go`	Updates coordinator tests to expect negotiated DB checksum feature.
`oxiad/coordinator/controller/shard_controller_election_test.go`	Updates feature negotiation unit tests for renamed feature.
`oxiad/common/feature/feature.go`	Updates supported feature list to `FEATURE_DB_CHECKSUM`.
`oxiad/common/feature/feature_test.go`	Removes prior supported-features tests (file deleted).
`oxiad/common/crc/crc.go`	Adds `Checksum.IsZero()` helper.
`common/proto/storage.proto`	Adds `control_request` to `LogEntryValue` and defines `ControlRequest/FeatureEnableRequest`.
`common/proto/storage.pb.go`	Regenerated Go bindings for `storage.proto`.
`common/proto/storage_vtproto.pb.go`	Regenerated vtproto bindings to support new messages/oneofs.
`common/proto/replication.proto`	Renames feature enum value to `FEATURE_DB_CHECKSUM`.
`common/proto/replication.pb.go`	Regenerated Go bindings for `replication.proto`.

Comments suppressed due to low confidence (2)

oxiad/dataserver/controller/lead/leader_controller.go:849

Timer started here is never finished on the early-return path (non-leader). This will skew latency metrics and may leak resources depending on the histogram implementation. Ensure timer.Done()/DoneCtx is called before every return.

func (lc *leaderController) propose(ctx context.Context, proposalSupplier func(offset int64) statemachine.Proposal, cb concurrent.Callback[statemachine.ApplyResponse]) {
	timer := lc.writeLatencyHisto.Timer()
	lc.Lock()
	if err := checkStatusIsLeader(lc.status); err != nil {
		lc.Unlock()
		cb.OnCompleteError(err)
		return
	}

oxiad/dataserver/controller/lead/leader_controller.go:868

If MarshalVT() fails, propose() returns without calling timer.Done/DoneCtx, leaving the latency timer open. Make sure to finish the timer on this error path as well.

	value, err := entryValue.MarshalVT()
	if err != nil {
		lc.Unlock()
		cb.OnCompleteError(err)
		return
	}

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/control/control_request_test.go

oxiad/dataserver/controller/lead/leader_controller.go

oxiad/dataserver/controller/statemachine/state_machine.go

oxiad/dataserver/database/db.go

tests/control/control_request_test.go

oxiad/dataserver/controller/lead/leader_controller_test.go

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

oxiad/dataserver/controller/lead/leader_controller.go:869

propose() starts a latency timer, but on early-return paths (eg not leader at lines 846-850, or MarshalVT error at 865-869) the timer is never completed. This can skew latency metrics and potentially leak timer resources depending on the implementation. Suggest calling timer.DoneCtx(ctx) (or timer.Done()) before each early return.

func (lc *leaderController) propose(ctx context.Context, proposalSupplier func(offset int64) statemachine.Proposal, cb concurrent.Callback[statemachine.ApplyResponse]) {
	timer := lc.writeLatencyHisto.Timer()
	lc.Lock()
	if err := checkStatusIsLeader(lc.status); err != nil {
		lc.Unlock()
		cb.OnCompleteError(err)
		return
	}
	newOffset := lc.quorumAckTracker.NextOffset()
	walLog := lc.wal
	tracker := lc.quorumAckTracker
	term := lc.term
	proposal := proposalSupplier(newOffset)

	lc.log.Debug("propose a proposal", slog.Any("proposal", proposal))

	entryValue := proto.LogEntryValueFromVTPool()
	defer entryValue.ReturnToVTPool()

	proposal.ToLogEntry(entryValue)

	value, err := entryValue.MarshalVT()
	if err != nil {
		lc.Unlock()
		cb.OnCompleteError(err)
		return
	}

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

oxiad/dataserver/controller/statemachine/state_machine.go

oxiad/dataserver/controller/lead/leader_controller.go

oxiad/dataserver/controller/lead/leader_controller_test.go

…anges Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Each proposal type now knows how to apply itself via an Apply method, eliminating the type-switch in ApplyProposal and the applyWriteRequest wrapper. Callers invoke proposal.Apply() directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ApplyResponse belongs with the Proposal interface that returns it. applyControlRequest was a 5-line helper used in only two places — inline it into ControlProposal.Apply and ApplyLogEntry. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…test The feature-enable proposal is fire-and-forget after BecomeLeader, so followers may not have applied it yet when the assertion runs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

ReadChecksum reads an atomic.Pointer, so a read lock is sufficient. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Replace vague "propose a proposal" with "Appending proposal to WAL" and log useful fields (offset, timestamp) instead of unexported struct. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mattisonchao self-assigned this Feb 9, 2026

mattisonchao requested review from RobertIndie, coderzc and merlimat as code owners February 9, 2026 03:25

coderzc requested a review from Copilot February 9, 2026 03:31

Copilot started reviewing on behalf of coderzc February 9, 2026 03:32 View session

github-code-quality bot found potential problems Feb 9, 2026

View reviewed changes

common/proto/storage_vtproto.pb.go Dismissed Show dismissed Hide dismissed

common/proto/storage_vtproto.pb.go Dismissed Show dismissed Hide dismissed

Copilot AI reviewed Feb 9, 2026

View reviewed changes

coderzc requested a review from Copilot February 9, 2026 03:57

Copilot started reviewing on behalf of coderzc February 9, 2026 03:57 View session

Copilot AI reviewed Feb 9, 2026

View reviewed changes

mattisonchao added 12 commits February 22, 2026 23:51

feat: support control request for state machine

8758f34

add some changes

df0cb9d

feat: add tests for it

167ff05

remove useless changes

65f436d

fix lint

780a45e

fix license

b2ec4b2

fix the test

5e6c007

fix the race condition

b593777

add empty check

a5d50a2

fix the flaky test

d6c1cee

fix dirty state

ac934dd

add lock to avoid race condition

226e6d0

mattisonchao force-pushed the feat.control.request branch from fa596b5 to f681fc5 Compare February 22, 2026 16:24

mattisonchao changed the title ~~feat: support control request for shard~~ feat: support control request in replication state machine Feb 22, 2026

fix: restore follower controller to main with minimal statemachine ch…

ed7a78c

…anges Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mattisonchao force-pushed the feat.control.request branch from f681fc5 to ed7a78c Compare February 22, 2026 16:38

mattisonchao and others added 3 commits February 23, 2026 00:44

fix: hold rwMutex.RLock during ApplyLogEntry in follower

f1240f9

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mattisonchao and others added 3 commits February 23, 2026 01:23

fix: use assert.Eventually for feature-enabled checks in integration …

45644f0

…test The feature-enable proposal is fire-and-forget after BecomeLeader, so followers may not have applied it yet when the assertion runs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix: use RLock for read-only Checksum() in leader controller

aa7c528

ReadChecksum reads an atomic.Pointer, so a read lock is sufficient. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix: improve debug log message in propose method

a14cb61

Replace vague "propose a proposal" with "Appending proposal to WAL" and log useful fields (offset, timestamp) instead of unexported struct. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mattisonchao changed the title ~~feat: support control request in replication state machine~~ feat: replicate control requests through WAL via state machine Feb 22, 2026

fix: remove trailing newline in state_machine.go

77b7ce7

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mattisonchao added this to the 0.16 milestone Feb 22, 2026

mattisonchao merged commit baa33eb into main Feb 22, 2026
11 of 12 checks passed

mattisonchao deleted the feat.control.request branch February 22, 2026 17:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: replicate control requests through WAL via state machine#882

feat: replicate control requests through WAL via state machine#882
mattisonchao merged 20 commits intomainfrom
feat.control.request

mattisonchao commented Feb 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mattisonchao commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modification

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mattisonchao commented Feb 9, 2026 •

edited

Loading