Enable CodSpeed's memory benchmarks for simulation benchmarks#24298
Enable CodSpeed's memory benchmarks for simulation benchmarks#24298MichaReiser merged 4 commits intomainfrom
Conversation
|
343dad3 to
893bcb6
Compare
Typing conformance resultsNo changes detected ✅Current numbersThe percentage of diagnostics emitted that were expected errors held steady at 86.61%. The percentage of expected errors that received a diagnostic held steady at 81.56%. The number of fully passing files held steady at 70/132. |
Memory usage reportMemory usage unchanged ✅ |
|
| Lint rule | Added | Removed | Changed |
|---|---|---|---|
invalid-await |
0 | 40 | 0 |
invalid-return-type |
0 | 1 | 0 |
| Total | 0 | 41 | 0 |
Changes in flaky projects detected. Raw diff output excludes flaky projects; see the HTML report for details.
This comment was marked as resolved.
This comment was marked as resolved.
There was a problem hiding this comment.
This is cool, but I wonder how useful it will realistically be. Most of the memory profiles codspeed has just show steady memory growth over time while the benchmark is being executed, and for those that do show bumps, it's hard to associate those bumps with any specific code event since the x axis just shows the amount of seconds elapsed. I guess maybe a counterargument is that we're much more likely to spot bugs like #24301 with this enabled?
|
I agree, CodSpeed currently doesn't provide a great way of debugging a regression other than guessing by the time the allocation happens. However, being able to monitor overall memory usage seems useful to me and isn't much different than what we do with our Salsa memory usage job, except that CodSpeed captures all allocations. It's also more fine-grained, e.g., we can monitor the parser's memory usage, and it also extends coverage (more ecosystem projects). I'd like to give this a try. We can always disable it if it's noisy. |
* main: (35 commits) Store definition indexes as u32 (#24307) Avoid re-using symbol in RUF024 fix (#24316) [ty] Add materialization to `Divergent` type (#24255) [ty] Make `Divergent` a top-level type variant (#24252) [ty] Fix nested global and nonlocal lookups through forwarding scopes (#24279) Fetch the cargo-dist binary directly instead of using the installer (#24258) [ty] Fix panic on `list[Annotated[()]]` (#24303) Don't measure the AST deallocation time in parser benchmarks (#24301) Enable CodSpeed's memory benchmarks for simulation benchmarks (#24298) Upgrade imara-diff to 0.2.0 (#24299) [ty] Represent `InitVar` as a special form internally, not a class (#24248) `RUF067`: Allow dunder-named assignments in non-strict mode [`pyupgrade`] UP018 should detect more unnecessarily wrapped literals (UP018) (#24093) [ty] Remove unused `system.glob` method (#24300) [ty] Reject functional TypedDict with mismatched name (#24295) Update Rust crate arc-swap to v1.9.0 (#24292) [ty] Remove unused `@Todo(Functional TypedDicts)` (#24297) Update CodSpeedHQ/action action to v4.12.1 (#24290) Update taiki-e/install-action action to v2.69.6 (#24293) Update Rust crate toml to v1.0.7 (#24289) ...
Summary
Enable CodSpeed's memory profiling for our simulation benchmarks.
We can consider enabling them for our walltime benchmarks if we find them useful. I decided not to do so for now, because they take longer to run (higher CI cost).
Enabling memory profiling required upgrading the codspeed crates.
Examples