Add support for system-level mutation mode for zero-downtime migration by eazyhozy · Pull Request #42 · kakao/actionbase

eazyhozy · 2026-01-14T09:03:03Z

Summary

This PR implements support for a system-mutation-mode configuration. This allows operators to enforce a specific mutation mode (SYNC or ASYNC) across the entire graph instance, overriding individual label settings.

A force query parameter on the existing /sync endpoint allows internal systems (e.g., Async Processor) to bypass the system override and maintain synchronous processing guarantees.

Closes #41

Design

Before (Request > Table)

mode = request ?: table

Request	Table	Mode	Queue
SYNC	SYNC	SYNC	false
SYNC	ASYNC	SYNC	false
SYNC	IGNORE	—	Invalid
ASYNC	*	ASYNC	true
IGNORE	*	IGNORE	true
N/A	SYNC	SYNC	false
N/A	ASYNC	ASYNC	true
N/A	IGNORE	IGNORE	true

After (Request(force) > System > Request > Table)

mode = when {
    force        -> request     // force requires non-null request
    system!=null -> system
    request!=null -> request
    else         -> table
}

Force(Request)	System	Request	Table	Mode	Queue
true	*	SYNC	*	SYNC	false
true	*	ASYNC	*	ASYNC	true
true	*	N/A	*	—	Invalid
false	SYNC	*	*	SYNC	false
false	ASYNC	*	*	ASYNC	true
false	IGNORE	*	*	IGNORE	true
false	N/A	SYNC	SYNC	SYNC	false
false	N/A	SYNC	ASYNC	SYNC	false
false	N/A	SYNC	IGNORE	—	Invalid
false	N/A	ASYNC	*	ASYNC	true
false	N/A	IGNORE	*	IGNORE	true
false	N/A	N/A	SYNC	SYNC	false
false	N/A	N/A	ASYNC	ASYNC	true
false	N/A	N/A	IGNORE	IGNORE	true

Design Background

The /sync endpoint is currently used by both internal systems (Async Processor) and external services. A simple System > Request > Table hierarchy would silently override external /sync requests when System=ASYNC is set. The force parameter solves this by allowing callers to explicitly opt out of the system override when needed.

Use Case: Zero-Downtime HBase Cluster Migration

Set system-mutation-mode=ASYNC → All requests are queued to Kafka
Async Processor continues writing via /sync?force=true (bypasses System)
Stop Async Processor, redeploy App Server to new cluster
Resume Async Processor → Zero-downtime migration complete

Changes

Configuration: Renamed globalMutationMode to systemMutationMode in GraphProperties and GraphConfig.
API Consolidation:
- Removed separate internalMutate methods from Graph and MutationService.
- Single mutate method with forceSyncMode: Boolean parameter.
- Removed /internal/sync endpoints; added force query parameter to existing /sync endpoints.
MutationModeContext:
- Replaced global/internal parameters with system/force.
- Primary constructor is private; creation enforced via .of() factory method.
- Precedence: request(force=true) > system > request(force=false) > table.
DML Isolation: This setting applies only to DML. DDL operations use forceSyncMode=true internally to maintain synchronous guarantees.

Files Changed

Engine (main)

MutationEngine.kt — Renamed globalMutationMode to systemMutationMode
MutationModeContext.kt (engine + v2) — Replaced global/internal with system/force; private constructor + .of() factory
MutationService.kt — Consolidated mutate/internalMutate into single mutate with forceSyncMode
Graph.kt — Renamed property; consolidated mutate APIs
GraphConfig.kt — Renamed config property and builder method
DdlService.kt — Changed from internalMutate to mutate(..., forceSyncMode=true)
KafkaProducer.kt, WalLog.kt, V2BackedEngine.kt, V2BackedMessageBinding.kt — Minor adjustments

Server (main)

GraphProperties.kt — Renamed to systemMutationMode
GraphConfiguration.kt — Wired systemMutationMode into GraphConfig.Builder
EdgeMutationController.kt — Removed /internal/sync; added force param to /sync
MultiEdgeMutationController.kt — Same

Tests

MutationModeContextTest.kt — Full precedence matrix coverage (25 cases)
MutationServiceSystemAsyncSpec.kt — system=ASYNC override + force=true bypass tests
EdgeMutationForceSyncTest.kt — E2E tests for /sync?force=true
DdlServiceSpec.kt — Verified systemMutationMode does not affect DDL
StartUpTest.kt — Verified server initialization with system mutation mode

How to Test

Configuration

Add the following to your application.yaml. This configuration is optional and can be omitted to continue using label-specific settings.

kc:
  graph:
    system-mutation-mode: async # optional: sync | async

Verification

During local execution, check the initialization logs:

$ ./gradlew server:bootRun

... com.kakao.actionbase.v2.engine.Graph : graph initialized with config: GraphConfig(..., systemMutationMode=ASYNC)

Force Sync

The Async Processor uses the force parameter to bypass the system override:

POST /graph/v3/databases/{database}/tables/{table}/edges/sync?force=true
POST /graph/v3/databases/{database}/tables/{table}/multi-edges/sync?force=true

DDL operations are also unaffected — they invoke mutate(..., forceSyncMode=true) directly.

Further TODO

Extend mutation controllers to support async request endpoint (e.g., /edges/async or syncMode query parameter) so callers can explicitly request ASYNC processing.

Introduce `globalMutationMode` in `GraphConfig` to enable configuration of mutation processing modes. Updated tests to verify ASYNC mutation mode behavior.

Introduce `MutationMode` to handle synchronous and asynchronous mutation processing. Extend test fixtures and services to validate mutation behavior in both modes.

Extend test fixtures to include multi-edge schema creation and DDL operations for both sync and async modes. Update tests to validate multi-edge mutation behavior.

Introduce `globalMutationMode` in `Graph` to globally define mutation processing mode. Update mutation services to respect `globalMutationMode` and fallback to sync when unspecified. Revise tests to validate behavior for ASYNC and sync scenarios.

Remove redundant type arguments in readValue calls and simplify property initialization by eliminating explicit nullable type declaration.

Add unit tests to validate behavior when globalMutationMode is set to SYNC, ensuring synchronous processing overrides label-specific async configurations for both single-edge and multi-edge mutations.

Introduce `globalMutationMode` property to `GraphProperties` for configuring mutation processing modes. Update `GraphConfiguration` to apply global mutation mode settings when provided.

eazyhozy · 2026-01-15T08:51:16Z

@em3s please review this PR!

em3s · 2026-01-15T09:03:16Z

@eazyhozy nice work!

Quick review:

This change also needs to be applied to the v2 engine (Graph.kt).
Please describe explicitly what was changed and where.
For example:
- applied to the v3 engine
- applied to the v2 engine

eazyhozy · 2026-01-15T09:29:50Z

@em3s Thanks for a quick feedback. I updated the description above.

Logic:

(✅ Done) Updated the v3 engine (V3MutationService) so that the global-mutation-mode overrides the table-level mutation mode for all v3 mutation requests.

(🔄 WIP) Updating the v2 engine (Graph.mutate()) to ensure v2 mutation requests also respect the global configuration.

em3s · 2026-01-19T11:20:21Z

Reformatted per #62 via gh by Claude Code.

eazyhozy · 2026-01-21T02:12:27Z

@em3s

The primary goal of this PR (#41) is to introduce a new configuration that can override table-level mutationMode settings.

Key Principles

DML Only: This setting applies strictly to DML. It must not affect DDL operations to ensure server stability and prevent startup failures.
Precedence Logic: The hierarchy is defined as Request (r) > Global (g) > Table (t).

Current vs. Proposed Behavior

Currently, the behavior is defined as follows(according to MutationModeContext):

label (t)	request (r)	queue
ASYNC	N/A	true
ASYNC	SYNC	false
SYNC	N/A	false
SYNC	ASYNC	true

By introducing globalMutationMode, I propose the following expanded precedence matrix:

Table (t)	Global (g)	Request (r)	queue
ASYNC	N/A	N/A	true
ASYNC	SYNC	N/A	false
ASYNC	ASYNC	N/A	true
SYNC	ASYNC	N/A	true
SYNC	SYNC	ASYNC	true
(Any)	(Any)	SYNC	false

(Note: Request level remains the highest priority to ensure synchronous processing for dedicated paths like external async processors.)

Feedback Requested

Directional Review: Does this Request > Global > Table precedence logic align with our architectural goals? I want to ensure that providing a middle-tier override between the request and table metadata is the correct approach for operational flexibility.
Naming Convention: The prefix global usually suggests a fallback (default value). However, in this PR, it acts as an override with higher priority than table settings. Do you think globalMutationMode is clear enough, or would something like overrideMutationMode or forcedMutationMode be more appropriate?
(added) Behavior of IGNORE: I am uncertain how the IGNORE mode should behave within this new hierarchy. Should a global IGNORE setting take precedence over all table settings, or should it only apply if the table itself is not set to IGNORE? I would appreciate your thoughts on the most consistent design for this case.

…riority Update `MutationModeContext` to handle global mutation modes along with request-specific overrides, ensuring priority order is request > global > label. Add comprehensive test coverage for various combinations of mutation modes and scenarios.

Update method signatures in mutation services to rename the `sync` parameter to `mode` for improved clarity and consistency. Adjust related controller and test references accordingly.

…vice Simplify variable naming for improved readability and consistency within mutation services. Adjust related references accordingly.

Eliminate multi-edge schema and related DDL operations from test fixtures as they are no longer used. Simplify tests and improve maintainability.

Simplify method signature by providing a default value for the `configBuilder` parameter in the `create` method. Improves usability and reduces redundancy.

Eliminate the unused `MutationMode` parameter in test fixture methods and related calls. Simplifies method signatures and improves code clarity.

eazyhozy · 2026-01-22T00:06:56Z

@dosu plz summarize the file changes.

dosubot · 2026-01-22T00:07:55Z

Created changelog: https://app.dosu.dev/documents/2e3e62c0-3fd1-40d5-bbaf-1f1fb41d04c2

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

…obalAsyncSpec

…ncerns Rewrite V3MutationServiceGlobalAsyncSpec with 6 tests covering 3 scenarios (global overrides table, global overrides request, internal overrides global) x 2 table types (EDGE, MULTI_EDGE). Simplify verifyWal to only check queue flag, since MutationModeContext logic is fully covered by MutationModeContextSpec. Remove internal mode tests from V3MutationServiceSpec and V3MutationServiceAsyncSpec as they now belong in GlobalAsyncSpec.

# Conflicts: # engine/src/main/kotlin/com/kakao/actionbase/v2/engine/v3/V3MutationService.kt

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

eazyhozy · 2026-02-23T02:23:44Z

@em3s

Resolved merge conflicts with the latest main (conflict in V3MutationService.kt due to #199 refactor) and applied spotless formatting. CI is now passing.

Re-requesting your review!

Port globalMutationMode and internal mode concepts to the new MutationEngine/MutationService architecture introduced in main. - Extend engine MutationModeContext with global/internal fields - Add globalMutationMode to MutationEngine interface - Add internalMutate() API to MutationService - Add /internal/sync endpoints to Edge/MultiEdge controllers - Migrate GlobalAsyncSpec tests to new MutationService API Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Move global/internal precedence matrix tests from v2 package (MutationModeContextSpec) to engine package (MutationModeContextTest) to align with main's refactored structure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Adds of(table, request) convenience overload that delegates to the 4-arg factory with global=null and internal=null, matching the engine package's MutationModeContext API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…e coverage Remove comprehensive test and merge overlapping cases into focused tests. Reorganize OfWithSystemAndForceTest from 24 to 13 cases while maintaining full matrix coverage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ontext Rename globalMutationMode to systemMutationMode and replace internal MutationMode parameter with boolean force flag. This changes the priority model from "internal > global > request > label" to "request(force=true) > system > request(force=false) > label". Remove separate internalMutate methods from Graph and MutationService, consolidating into a single mutate method with a force parameter. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

engine/src/main/kotlin/com/kakao/actionbase/engine/service/MutationService.kt

engine/src/main/kotlin/com/kakao/actionbase/v2/engine/metadata/MutationModeContext.kt

engine/src/main/kotlin/com/kakao/actionbase/engine/metadata/MutationModeContext.kt

engine/src/main/kotlin/com/kakao/actionbase/v2/engine/metadata/MutationModeContext.kt

engine/src/main/kotlin/com/kakao/actionbase/engine/metadata/MutationModeContext.kt

engine/src/test/kotlin/com/kakao/actionbase/engine/metadata/MutationModeContextTest.kt

Reorder KDoc to place queue formula after constraints, rename isSyncOnIgnoreTable to isSyncRequestOnIgnoreTable for clarity, and reorder test parameters to match (label, request, system, queue). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Remove dedicated /internal/sync endpoints from EdgeMutationController and MultiEdgeMutationController. Add force query parameter to existing /sync endpoints to achieve the same behavior. Rename MutationService.force to forceSyncMode for clarity. Add E2E tests verifying /sync?force=true overrides system=ASYNC. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The priority rule and matrix table already convey the resolution logic. Remove redundant mode = when { ... } block per self-review. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…nModeContextTest Reorder YAML fields from (label, system, request, queue) to (label, request, system, queue) to match function parameter order. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ypass The merge from main dropped the force parameter from Graph.mutate(), causing DdlService compilation failures. Restore the parameter and pass systemMutationMode and force to MutationModeContext.of(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ing merge The merge ec86cd0 reverted #209 change that replaced the hardcoded sufficientFetchSize=1000 with the configurable metadataFetchLimit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…Engine Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

em3s · 2026-03-03T02:59:27Z

LGTM 👍

wip: add support for globalMutationMode configuration (test fails yet)

975922c

Introduce `globalMutationMode` in `GraphConfig` to enable configuration of mutation processing modes. Updated tests to verify ASYNC mutation mode behavior.

eazyhozy self-assigned this Jan 14, 2026

eazyhozy added 6 commits January 15, 2026 11:03

test: add support for sync and async mutation modes

e9d32bc

Introduce `MutationMode` to handle synchronous and asynchronous mutation processing. Extend test fixtures and services to validate mutation behavior in both modes.

test: add multi-edge mutation support in test fixtures (test fails yet)

910e049

Extend test fixtures to include multi-edge schema creation and DDL operations for both sync and async modes. Update tests to validate multi-edge mutation behavior.

refactor: simplify property initialization and type casting adjustments

7cffff9

Remove redundant type arguments in readValue calls and simplify property initialization by eliminating explicit nullable type declaration.

test: add tests for globalMutationMode=SYNC overriding label mode=ASYNC

5e9e312

Add unit tests to validate behavior when globalMutationMode is set to SYNC, ensuring synchronous processing overrides label-specific async configurations for both single-edge and multi-edge mutations.

feat: add globalMutationMode support in GraphProperties

24bae7f

Introduce `globalMutationMode` property to `GraphProperties` for configuring mutation processing modes. Update `GraphConfiguration` to apply global mutation mode settings when provided.

eazyhozy requested a review from zipdoki January 15, 2026 08:33

eazyhozy marked this pull request as ready for review January 15, 2026 08:34

eazyhozy requested a review from em3s as a code owner January 15, 2026 08:34

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Jan 15, 2026

em3s removed request for em3s and zipdoki January 15, 2026 09:32

em3s changed the title ~~feat: add support for globalMutationMode configuration~~ feat(engine): add support for globalMutationMode configuration Jan 19, 2026

eazyhozy added 7 commits January 21, 2026 11:47

refactor: rename sync parameter to mode in mutation services

9483c23

Update method signatures in mutation services to rename the `sync` parameter to `mode` for improved clarity and consistency. Adjust related controller and test references accordingly.

refactor: rename mutationModeContext to mutationMode in V3MutationSer…

1a21601

…vice Simplify variable naming for improved readability and consistency within mutation services. Adjust related references accordingly.

refactor: remove unused multi-edge DDL and schema in test fixtures

1e90fa5

Eliminate multi-edge schema and related DDL operations from test fixtures as they are no longer used. Simplify tests and improve maintainability.

refactor: set default value for GraphConfig.Builder in create method

7855ea9

Simplify method signature by providing a default value for the `configBuilder` parameter in the `create` method. Improves usability and reduces redundancy.

refactor: remove MutationMode parameter from test fixtures

fd304e3

Eliminate the unused `MutationMode` parameter in test fixture methods and related calls. Simplifies method signatures and improves code clarity.

style: remove unnecessary change

80feff6

eazyhozy added 3 commits February 9, 2026 16:14

refactor: extract global mutation mode tests into V3MutationServiceGl…

32502d3

…obalAsyncSpec

style: use positional arguments for requestContext to minimize diff

8b29f45

em3s added scope:main and removed module:engine labels Feb 12, 2026

em3s changed the title ~~feat(engine): add support for globalMutationMode configuration~~ Add support for globalMutationMode configuration Feb 12, 2026

eazyhozy and others added 2 commits February 19, 2026 19:53

Merge remote-tracking branch 'origin/main' into init/global-async-mode

a7a84dd

# Conflicts: # engine/src/main/kotlin/com/kakao/actionbase/v2/engine/v3/V3MutationService.kt

style: apply spotless formatting to V3MutationService

8f203e2

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

eazyhozy and others added 5 commits February 25, 2026 10:51

eazyhozy commented Feb 25, 2026

View reviewed changes

eazyhozy and others added 2 commits February 25, 2026 18:39

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Feb 25, 2026

eazyhozy and others added 6 commits February 26, 2026 18:10

refactor: remove mode pseudo-code block from MutationModeContext KDoc

0f7af05

The priority rule and matrix table already convey the resolution logic. Remove redundant mode = when { ... } block per self-review. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

refactor: align YAML data field order with parameter order in Mutatio…

79fe955

…nModeContextTest Reorder YAML fields from (label, system, request, queue) to (label, request, system, queue) to match function parameter order. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into init/global-async-mode

ec86cd0

fix(engine): restore metadataFetchLimit in getOnlineMetadata lost dur…

f77e714

…ing merge The merge ec86cd0 reverted #209 change that replaced the hardcoded sufficientFetchSize=1000 with the configurable metadataFetchLimit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

refactor(engine): extract V2MutationMode.toV3() extension in V2Backed…

7eee1f7

…Engine Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

em3s force-pushed the init/global-async-mode branch from c2c10b9 to 7eee1f7 Compare March 3, 2026 02:54

em3s changed the title ~~Add support for globalMutationMode configuration~~ Add support for system-level mutation mode for zero-downtime migration Mar 3, 2026

em3s merged commit dfc86b0 into main Mar 3, 2026
7 checks passed

Conversation

eazyhozy commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design

Before (Request > Table)

After (Request(force) > System > Request > Table)

Design Background

Use Case: Zero-Downtime HBase Cluster Migration

Changes

Files Changed

How to Test

Configuration

Verification

Force Sync

Further TODO

Uh oh!

eazyhozy commented Jan 15, 2026

Uh oh!

em3s commented Jan 15, 2026

Uh oh!

eazyhozy commented Jan 15, 2026

Uh oh!

em3s commented Jan 19, 2026

Uh oh!

eazyhozy commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Principles

Current vs. Proposed Behavior

Feedback Requested

Uh oh!

eazyhozy commented Jan 22, 2026

Uh oh!

dosubot bot commented Jan 22, 2026

Uh oh!

eazyhozy commented Feb 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

em3s commented Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

eazyhozy commented Jan 14, 2026 •

edited

Loading

eazyhozy commented Jan 21, 2026 •

edited

Loading