Add SPMI benchmarks run collections for tiered and tiered pgo#84483
Add SPMI benchmarks run collections for tiered and tiered pgo#84483AndyAyersMS merged 2 commits intodotnet:mainfrom
Conversation
Add two new run configurations for SPMI benchmarks: tiered and tiered pgo. So benchmark runs now have 3 separate collections. The new ones are named "run_tiered" and "run_pgo", eg ``` benchmarks.run.windows.x64.checked.mch benchmarks.run_tiered.windows.x64.checked.mch benchmarks.run_pgo.windows.x64.checked.mch ```
|
Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch, @kunalspathak Issue DetailsAdd two new run configurations for SPMI benchmarks: tiered and tiered pgo. So benchmark runs now have 3 separate collections. The new ones are named "run_tiered" and "run_pgo", eg
|
|
Is there a list of benchmarks we run to get a collection? afair the full run of them takes 8 hours or so |
|
@BruceForstall PTAL Ran this on the internal pipeline with temp guid |
The benchmarks get split into 30 sub pieces and farmed out to helix, so I think we may run them all? |
|
Also we are still collecting benchmarks runs with R2R disabled -- wonder if we should revisit that? |
|
I was wondering why with tiering enabled we see a reasonably high fraction of methods bypassing tiering. Not we don't have an explicit flag for this so you have to deduce it by the absence of other flags. One such bin is @EgorBo reminded me that dynamic methods (like the classic regex) are still not eligible for tiered compilation. Sure enough regex seems to be a major contributor to this bin. |
BruceForstall
left a comment
There was a problem hiding this comment.
Just a few requests.
Actually, I'm happy to see not many changes were required.
|
superpmi_benchmarks.py includes: So, R2R is disabled. Do the other BDN arguments make sense for these new collections? |
Hmm, good point. Let me see if we can afford to run the benchmarks "normally" or if it takes too long. I may leave things as is for nontiered as the codegen is not timing sensitive. We are also using a checked runtime which tiers up aggressively and uses the wrong corelib (#60947). So these collections are not as representative of what actually happens as I'd like. |
Yes, we run them all with |
Meaning that they don't tier up properly? |
Certainly seems possible, though with the aggressive tiering up that a checked runtime does, perhaps they do? Let me first try and get this revamped so we are measuring a release runtime/SPC with a checked jit, and then we can see if we can afford to run the benchmarks more realistically. |
|
Another option is to just measure all release bits, but that probably makes the collection too brittle (?). I suppose we could also enable the extra queries in release mode. |
Is it worth getting this checked in now, basically as-is, since it already is showing additional code variety, and trying to implement Release as a follow-up?
Enabling extra queries is Release is an interesting idea. It would be useful to do that, get collections, then ensure they can be replayed (with JitDisasm/JitDump). |
Not a bad idea, as having something now is probably better than having nothing... |
|
@BruceForstall addressed your feedback, so take another look when you can. |
Add two new run configurations for SPMI benchmarks: tiered and tiered pgo. So benchmark runs now have 3 separate collections.
The new ones are named "run_tiered" and "run_pgo", eg
Fixes #68179.