Fix race condition of timeout thread interrupt to stabilize multi level build tests#4254
Merged
lihaoyi merged 13 commits intocom-lihaoyi:mainfrom Jan 6, 2025
Merged
Conversation
Member
Author
|
I'm hoping this also fixes the flakiness in |
lihaoyi
added a commit
that referenced
this pull request
Jan 7, 2025
This might have been the cause of a lot of flakiness that seems to have gone away with #4254, as the server exiting caused the `runBackground` calls to exit causing the http servers to exit and fail to pick up requests. Might have been caused by com-lihaoyi/os-lib#324 which made `destroyOnExit` the default for spawned subprocesses. This PR explicitly disables `destroyOnExit` for the subprocesses where `background = true` Covered by a new `integration.invalidation` test that runs under both `server` and `fork`, that previously failed when run under `fork`
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The basic problem was that
thread.interrupt()running beforeinterrupt = falsemeant there was a chance the thread'sif (interrupt) {conditional would betrue, resulting in the timeout thread timing out the open socket immediately, and the Mill server process shutting down.The solution is to move
interrupt = falseto beforethread.interrupt()This should hopefully fix the flakiness in multi-level-build tests, where the process shutting down would cause the next command to re-spawn a new process, resulting in all classloaders to be re-spawned, violating our assertions
Improved the test error checking so next time a similar thing happens, we get more precise reporting "an unwanted process restart occurred", rather than just "classloader invalidation was unexpected".
Tested manually with
while ./mill 'integration.invalidation[multi-level-editing].server.test'; do :; done. This seems to reproduce the problem on my laptop in a few tens of minutes without this fix, after this fix I haven't managed to make it appear