fix(channel): split intermediate and init readiness channels by uran0sH · Pull Request #3504 · youki-dev/youki

uran0sH · 2026-04-17T02:46:28Z

Description

Under high concurrency, the init process can send InitReady before the
intermediate process sends IntermediateReady. The main process used to wait
for both messages on the same receiver, so wait_for_intermediate_ready()
could receive InitReady first and fail with an unexpected message error.

Add separate main-facing receivers for intermediate-owned and init-owned
readiness messages. IntermediateReady is read from the intermediate receiver,
while InitReady is forwarded through the init receiver and remains available
until wait_for_init_ready() consumes it.

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Refactoring (no functional changes)
Performance improvement
Test updates
CI/CD related changes
Other (please describe):

Testing

Added new unit tests
Added new integration tests
Ran existing test suite
Tested manually (please provide steps): 1. clone boxlite 2. apply this patch and change the youki's version in the boxlite 3. run make test.

Related Issues

ContainerBuilder::build() fails intermittently with:
received unexpected message: InitReady, expected: IntermediateReady(0)
The error comes from libcontainer::process::channel::MainReceiver::wait_for_intermediate_ready().

Normal message flow
During container creation, libcontainer spawns two child processes:

Intermediate process — sets up namespaces, cgroups, etc.
Init process — the actual container payload.
They communicate with the parent via a SOCK_SEQPACKET socket:
Intermediate sends IntermediateReady(pid).
Parent receives it in wait_for_intermediate_ready().
Init sends InitReady.
Parent receives it in wait_for_init_ready().

Root cause: parallel scheduling race
The init process is forked by the intermediate process using clone3(CLONE_PARENT). The two processes run in parallel. Under CPU pressure, the scheduler may execute the init process faster than the intermediate process. Consequently, InitReady can arrive at the parent before IntermediateReady.
wait_for_intermediate_ready() only calls recv() once. When it sees InitReady, it treats it as an unexpected message and aborts the build

Additional Context

saku3 · 2026-04-18T01:46:33Z

Thank you for the fix.

Would it be possible to share the actual reproduction steps?

I understand that the intermediate process and the init process run concurrently, but once the init process has been cloned, the intermediate process seems to have very little left to do before sending IntermediateReady, whereas the init process still appears to have a fair amount of work remaining, such as namespace setup and mount-related operations.

So I’d like to better understand under what kind of real-world scenario or environment this was observed.

As for the fix itself, I’d also like to see a unit test added for this scenario.

uran0sH · 2026-04-19T08:35:45Z

Thank you for the fix.

Would it be possible to share the actual reproduction steps?

I understand that the intermediate process and the init process run concurrently, but once the init process has been cloned, the intermediate process seems to have very little left to do before sending IntermediateReady, whereas the init process still appears to have a fair amount of work remaining, such as namespace setup and mount-related operations.

So I’d like to better understand under what kind of real-world scenario or environment this was observed.

As for the fix itself, I’d also like to see a unit test added for this scenario.

I run make test:integration in boxlite(https://github.com/boxlite-ai/boxlite). It will fail:

FAIL [  21.353s] boxlite::zygote_integration test_zygote_concurrent_stderr_isolation
FAIL [  20.259s] boxlite::zygote_integration test_zygote_concurrent_workdir_isolation
FAIL [  19.162s] boxlite::zygote_integration test_zygote_repeated_bursts
FAIL [  20.012s] boxlite::zygote_integration test_zygote_sequential_then_concurrent

And the reason is:

thread 'test_concurrent_exec_high_concurrency' (1656451) panicked at src/boxlite/tests/execution_shutdown.rs:1466:46:
    called `Result::unwrap()` on an `Err` value: Internal("spawn_failed: internal error: build failed: failed to create container: received unexpected message: InitReady, expected: IntermediateReady(0)")

I suspect it's because it's running inside a VM, which increases the probability of this happening.

nayuta723 · 2026-04-24T16:17:22Z

Is this an issue that can also be reproduced in runc?

nayuta723 · 2026-04-24T16:20:58Z

My understanding is that this issue does not occur in runc because it uses separate pipes for communication between the main process and the intermediate process, and between the main process and the init process.

In contrast, in youki’s implementation, the main receiver handles messages from both the intermediate and init processes over the same channel, which I believe is what leads to this issue.

saku3 · 2026-04-25T07:43:11Z

Thank you for the comment.

I think this PR may be acceptable as an immediate fix, assuming the tests pass.

However, I think the same ordering issue could also happen with messages other than InitReady, such as HookRequest, SetupNetworkDeviceReady, or SeccompNotify.

As nayuta723 mentioned, youki currently receives messages from both the intermediate process and the init process through the shared MainReceiver. To fix this more permanently, I think we should split the communication channels, for example by using separate intermediate-to-main and init-to-main channels.

uran0sH · 2026-04-25T07:48:34Z

Thank you for the comment.

I think this PR may be acceptable as an immediate fix, assuming the tests pass.

However, I think the same ordering issue could also happen with messages other than InitReady, such as HookRequest, SetupNetworkDeviceReady, or SeccompNotify.

As nayuta723 mentioned, youki currently receives messages from both the intermediate process and the init process through the shared MainReceiver. To fix this more permanently, I think we should split the communication channels, for example by using separate intermediate-to-main and init-to-main channels.

I agree with you. I can also provide a permanent fix.

uran0sH · 2026-05-15T09:22:35Z

@saku3 @nayuta723 Hi, I've submitted a new version that splits the channels. I've verified it in Boxlite's tests, but I haven't added the corresponding tests to the Youki repository yet. I thought it is difficult to design a test to reproduce this problem. Do you have any good suggestions?

nayuta723 · 2026-05-17T02:27:22Z

Thank you! In my opinion, we don't need to add tests for asynchronous edge cases like CPU pressure, etc. I believe that for this PR, ensuring all existing tests pass and sharing the manual verification results on Boxlite under 'Tested manually (please provide steps)' should be sufficient.

@saku3
What do you think about that?

saku3 · 2026-05-17T05:51:30Z

I agree with nayuta723’s suggestion.

saku3

Thanks, that makes sense. If you have a chance, could you update both the PR title and the commit message to reflect the current implementation?

Under high concurrency, the init process can send InitReady before the intermediate process sends IntermediateReady. The main process used to wait for both messages on the same receiver, so wait_for_intermediate_ready() could receive InitReady first and fail with an unexpected message error. Add separate main-facing receivers for intermediate-owned and init-owned readiness messages. IntermediateReady is read from the intermediate receiver, while InitReady is forwarded through the init receiver and remains available until wait_for_init_ready() consumes it. Signed-off-by: Wenyu Huang <huangwenyuu@outlook.com>

nayuta723

LGTM

Thanks!

nayuta723 added the kind/bug label Apr 17, 2026

uran0sH force-pushed the fix-initready-race branch from 3d0d8d4 to dc061a1 Compare April 19, 2026 10:21

DorianZheng mentioned this pull request May 1, 2026

fix(guest): apply youki PR #3504 to fix InitReady/IntermediateReady race under concurrent exec boxlite-ai/boxlite#455

Open

uran0sH force-pushed the fix-initready-race branch 2 times, most recently from 183de45 to 2cae7e7 Compare May 14, 2026 17:07

saku3 approved these changes May 17, 2026

View reviewed changes

saku3 requested a review from nayuta723 May 17, 2026 10:33

uran0sH force-pushed the fix-initready-race branch from 2cae7e7 to 519c459 Compare May 17, 2026 12:46

uran0sH changed the title ~~fix(channel): cache InitReady when it arrives before IntermediateReady~~ fix(channel): split intermediate and init readiness channels May 17, 2026

nayuta723 approved these changes May 18, 2026

View reviewed changes

saku3 merged commit 4b2f0e0 into youki-dev:main May 18, 2026
28 checks passed

github-actions Bot mentioned this pull request May 18, 2026

Release for v0.6.1 #3440

Open

G4614 mentioned this pull request May 28, 2026

fix(libcontainer): bump to youki rev — InitReady race + per-exec mount leak boxlite-ai/boxlite#611

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(channel): split intermediate and init readiness channels#3504

fix(channel): split intermediate and init readiness channels#3504
saku3 merged 1 commit into
youki-dev:mainfrom
uran0sH:fix-initready-race

uran0sH commented Apr 17, 2026 •

edited

Loading

Uh oh!

saku3 commented Apr 18, 2026

Uh oh!

uran0sH commented Apr 19, 2026 •

edited

Loading

Uh oh!

nayuta723 commented Apr 24, 2026

Uh oh!

nayuta723 commented Apr 24, 2026

Uh oh!

saku3 commented Apr 25, 2026

Uh oh!

uran0sH commented Apr 25, 2026 •

edited

Loading

Uh oh!

uran0sH commented May 15, 2026

Uh oh!

nayuta723 commented May 17, 2026 •

edited

Loading

Uh oh!

saku3 commented May 17, 2026

Uh oh!

saku3 left a comment

Uh oh!

nayuta723 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

uran0sH commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Testing

Related Issues

Additional Context

Uh oh!

saku3 commented Apr 18, 2026

Uh oh!

uran0sH commented Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nayuta723 commented Apr 24, 2026

Uh oh!

nayuta723 commented Apr 24, 2026

Uh oh!

saku3 commented Apr 25, 2026

Uh oh!

uran0sH commented Apr 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uran0sH commented May 15, 2026

Uh oh!

nayuta723 commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saku3 commented May 17, 2026

Uh oh!

saku3 left a comment

Choose a reason for hiding this comment

Uh oh!

nayuta723 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

uran0sH commented Apr 17, 2026 •

edited

Loading

uran0sH commented Apr 19, 2026 •

edited

Loading

uran0sH commented Apr 25, 2026 •

edited

Loading

nayuta723 commented May 17, 2026 •

edited

Loading