Skip to content

Fix SelfContainedMutex deadlock and add loom support#2558

Merged
sporksmith merged 4 commits intoshadow:mainfrom
sporksmith:loom2
Nov 28, 2022
Merged

Fix SelfContainedMutex deadlock and add loom support#2558
sporksmith merged 4 commits intoshadow:mainfrom
sporksmith:loom2

Conversation

@sporksmith
Copy link
Copy Markdown
Contributor

@sporksmith sporksmith commented Nov 19, 2022

Rewrite SelfContainedMutex to fix deadlock and test under loom

Fixes #2557

This collapses the lock state and the number of sleepers into a single atomic, which we also use as the futex word. This makes the concurrency correctness much easier to reason about, at the cost of having to do some bit fiddling, and under contention sometimes having to do some extra atomic operations to synchronize the sleeper-count.

This also adds support for testing SelfContainedMutex using loom. This requires compiling the code specially, and isn't done by default. It can be run under loom with:

LOOM_MAX_PREEMPTIONS=3 \    
RUSTFLAGS="--cfg loom" \    
cargo test \    
--manifest-path=src/Cargo.toml \    
-p shadow-shim-helper-rs \    
--test scmutex-tests \    
--target-dir=loomtarget \    
-- --nocapture 

@github-actions github-actions bot added Component: Build Build/install tools and dependencies Component: Libraries Support functions like LD_PRELOAD and logging labels Nov 19, 2022
@sporksmith sporksmith changed the title Loom2 Fix SelfContainedMutex deadlock and add loom support Nov 19, 2022
@sporksmith sporksmith marked this pull request as ready for review November 19, 2022 22:37
@codecov
Copy link
Copy Markdown

codecov bot commented Nov 19, 2022

Codecov Report

Base: 67.43% // Head: 67.79% // Increases project coverage by +0.35% 🎉

Coverage data is based on head (08fa421) compared to base (58a1849).
Patch coverage: 87.73% of modified lines in pull request are covered.

❗ Current head 08fa421 differs from pull request most recent head 255ce7d. Consider uploading reports for the commit 255ce7d to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2558      +/-   ##
==========================================
+ Coverage   67.43%   67.79%   +0.35%     
==========================================
  Files         193      194       +1     
  Lines       28651    28784     +133     
  Branches     5631     5638       +7     
==========================================
+ Hits        19320    19513     +193     
+ Misses       4875     4787      -88     
- Partials     4456     4484      +28     
Flag Coverage Δ
tests 67.79% <87.73%> (+0.35%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/lib/shadow-shim-helper-rs/src/shim_shmem.rs 78.24% <ø> (ø)
...c/lib/shadow-shim-helper-rs/tests/scmutex-tests.rs 87.50% <87.50%> (ø)
src/lib/shadow-shim-helper-rs/src/scmutex.rs 89.41% <88.00%> (+2.06%) ⬆️
src/main/host/syscall/handler/time.rs 53.65% <0.00%> (-4.88%) ⬇️
src/main/utility/shm_cleanup.rs 59.55% <0.00%> (-2.25%) ⬇️
src/main/host/descriptor/descriptor_table.rs 80.48% <0.00%> (-1.22%) ⬇️
src/main/host/memory_manager/memory_copier.rs 72.41% <0.00%> (-0.92%) ⬇️
...c/main/utility/synchronization/count_down_latch.rs 85.10% <0.00%> (-0.54%) ⬇️
src/main/host/memory_manager/memory_mapper.rs 71.63% <0.00%> (-0.21%) ⬇️
src/main/core/worker.rs 76.38% <0.00%> (ø)
... and 15 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@sporksmith
Copy link
Copy Markdown
Contributor Author

benchmark run shows no change, as expected https://github.com/shadow/benchmark-results/tree/master/tor/2022-11-21-T17-07-28

Copy link
Copy Markdown
Contributor

@stevenengler stevenengler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reviewed up to but not including the "Rewrite SelfContainedMutex" commit so far. I'm reviewing that now, but figured I'd post the comments so far.

@stevenengler stevenengler self-requested a review November 23, 2022 18:28
Loom requires/recommends structuring tests this way, presumably so that
when running loom tests you can easily run only these tests with the
cargo `--test` flag.
At this point, we get a cascading error under loom, telling us that
something is wrong but is a bit tricky to debug.

Tests still pass outside of loom.
Use an atomic fence, with AcqRel ordering. This does *not*
fix the tests under loom, indicating there is still a problem.

See shadow#2557
Fixes shadow#2557

This collapses the lock state and the number of sleepers into a single
atomic, which we also use as the futex word. This makes the concurrency
correctness much easier to reason about, at the cost of having to do
some bit fiddling, and under contention sometimes having to do some
extra atomic operations to synchronize the sleeper-count.
@sporksmith sporksmith merged commit f7d209a into shadow:main Nov 28, 2022
@sporksmith sporksmith deleted the loom2 branch November 28, 2022 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: Build Build/install tools and dependencies Component: Libraries Support functions like LD_PRELOAD and logging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deadlock in SelfContainedMutex

2 participants