Skip to content

fix: Add Master readiness check in mini-cluster to prevent startup race conditions in test environment#162

Merged
szbr9486 merged 1 commit intoCurvineIO:mainfrom
jlon:mini_cluster
Aug 11, 2025
Merged

fix: Add Master readiness check in mini-cluster to prevent startup race conditions in test environment#162
szbr9486 merged 1 commit intoCurvineIO:mainfrom
jlon:mini_cluster

Conversation

@jlon
Copy link
Copy Markdown
Contributor

@jlon jlon commented Aug 10, 2025

This PR fixes a race condition in the mini-cluster test environment where Workers would attempt to connect to the Master before its RPC service was fully initialized, causing "Failed to connect" warnings and test instability.

Changes:

  • Added wait_master_ready() function to ensure Master is fully operational before starting Workers
  • Implemented two-phase readiness check: Raft leader election + RPC service availability
  • Added proper timeout and retry mechanisms with detailed logging

Impact:

  • Eliminates connection refused errors during cluster startup
  • Improves test reliability and stability
  • Provides better visibility into cluster initialization process

@jlon jlon changed the title fix: Add Master readiness check in mini-cluster to prevent startup race conditions in test environment (#161) fix: Add Master readiness check in mini-cluster to prevent startup race conditions in test environment Aug 10, 2025

while LocalTime::mills() <= wait_time {
retry_count += 1;
let info = fs.get_master_info().await?;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change info to master_info

@szbr9486 szbr9486 merged commit ffc775b into CurvineIO:main Aug 11, 2025
3 checks passed
@jlon jlon deleted the mini_cluster branch September 2, 2025 08:30
@lzjqsdd lzjqsdd added the bug Something isn't working label Sep 22, 2025
Coderlxl pushed a commit to Coderlxl/curvine that referenced this pull request Dec 25, 2025
…ce conditions in test environment (CurvineIO#161) (CurvineIO#162)

Co-authored-by: jianglong <771989*Jl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants