Skip to content

Extra Tests CI: Improve Rust caching#2575

Merged
sporksmith merged 1 commit intoshadow:mainfrom
sporksmith:rust-cache
Dec 8, 2022
Merged

Extra Tests CI: Improve Rust caching#2575
sporksmith merged 1 commit intoshadow:mainfrom
sporksmith:rust-cache

Conversation

@sporksmith
Copy link
Copy Markdown
Contributor

@sporksmith sporksmith commented Nov 29, 2022 •
edited
Loading

  • Cache a bit more of ~/.cargo. Unclear whether this ended up being needed in the end, but matches other examples, and didn't seem to increase compressed cache size significantly.
  • Key the ~/.cargo cache on the hash of the lock file instead of the current YYYY-MM. This invalidates the cache when our dependencies change, instead of having an increasingly stale cache over the course of the month.
  • Use sccache to cache compiled dependencies. This ends up only needing ~200 MB of cache, and speeds up the build quite a bit. (~10m-15m -> ~5m)

@github-actions github-actions bot added the Component: Testing Unit and integration tests and frameworks label Nov 29, 2022
@codecov
Copy link
Copy Markdown

codecov bot commented Nov 29, 2022 •
edited
Loading

Codecov Report

Base: 66.64% // Head: 67.67% // Increases project coverage by +1.03% 🎉

Coverage data is based on head (c46a1da) compared to base (a2a4539).
Patch has no changes to coverable lines.

❗ Current head c46a1da differs from pull request most recent head 7d6e71d. Consider uploading reports for the commit 7d6e71d to get more accurate results

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #2575      +/-   ##
==========================================
+ Coverage   66.64%   67.67%   +1.03%     
==========================================
  Files         197      197              
  Lines       29171    29171              
  Branches     5743     5743              
==========================================
+ Hits        19440    19741     +301     
+ Misses       5190     4854     -336     
- Partials     4541     4576      +35
Flag Coverage Δ
tests 67.67% <ø> (+1.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/main/host/memory_manager/memory_copier.rs 73.33% <0.00%> (-2.23%) ⬇️
src/lib/tsc/tsc_test.c 12.67% <0.00%> (ø)
src/main/utility/byte_queue.rs 56.01% <0.00%> (ø)
src/main/host/descriptor/socket/unix.rs 68.22% <0.00%> (+0.22%) ⬆️
src/main/host/host.rs 80.93% <0.00%> (+0.48%) ⬆️
src/main/core/main.rs 40.50% <0.00%> (+0.63%) ⬆️
src/lib/shadow-shim-helper-rs/src/signals.rs 70.70% <0.00%> (+0.63%) ⬆️
src/main/host/syscall/handler/socket.rs 64.85% <0.00%> (+0.68%) ⬆️
src/main/host/descriptor/mod.rs 67.17% <0.00%> (+0.86%) ⬆️
src/main/core/sim_config.rs 55.21% <0.00%> (+0.86%) ⬆️
... and 20 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@sporksmith sporksmith force-pushed the rust-cache branch 2 times, most recently from 61c4c68 to 58bf0a6 Compare November 29, 2022 22:07
@sporksmith sporksmith changed the title Rust cache Extra Tests CI: Improve Rust caching Nov 29, 2022
@sporksmith sporksmith marked this pull request as ready for review November 29, 2022 22:15
@sporksmith
Copy link
Copy Markdown
Contributor Author

sporksmith commented Nov 29, 2022 •
edited
Loading

The sccache cache is small enough that we could think about using it for other jobs as well. It's about 200 MB, and GH allows 10 GB per repository. Maybe we could even go back to "clean" builds + cache instead of (ab)using dockerhub for the full platform matrix.

I've had enough of shaving this yak for the moment though, and it wouldn't hurt to wait and see how our experience is with it in this workflow.

stevenengler
stevenengler approved these changes Dec 1, 2022
View reviewed changes
@sporksmith
Extra tests CI: improve Rust caching
7d6e71d 
* Cache more of ~/.cargo

Unclear whether this is needed, but matches other examples found online,
and doesn't seem to increase (compressed) cache size significantly.

* Add month to cache restore-keys

Using the month in the cache `restore-keys` in addition to `key`
ensures that we *reset* the cache every month, instead of continuing to
grow the previous month's cache.

* Include hash of the lockfile in cache keys but not restore-keys

This ensures we update the cache when our dependencies change.

* In extra_tests: Use sccache to cache built rust dependencies

Largely cargo-culted from
https://www.infinyon.com/blog/2021/04/github-actions-best-practices/#optimizing-rusts-build-speed-with-sccache

Only ends up using ~200 MB of cache, and speeds up the build quite a
bit.
@sporksmith sporksmith enabled auto-merge (squash) December 8, 2022 00:16
@sporksmith sporksmith merged commit 2a6b99c into shadow:main Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@stevenengler stevenengler stevenengler approved these changes

Assignees

@sporksmith sporksmith

Labels

Component: Testing Unit and integration tests and frameworks

Projects

None yet

Milestone

Code health and maintenance

Development

Successfully merging this pull request may close these issues.

3 participants

@sporksmith @stevenengler @robgjansen