Add mono performance runs#34825
Conversation
|
Tagging @ViktorHofer as an area owner |
|
Are these tests running in PerfLab in the end? Last time I checked there was no public output. I'd be happy to finally retire https://github.com/filipnavara/mono-performance/. |
|
Yes, this PR is not finished. I just needed to test some yml stuff, but when this gets merged we will be running the full set of performance tests from the performance repo on mono. |
|
Note that it may be worth it to also add runs in the Mono+LLVM configuration which differs greatly in the performance characteristics. It supports more hardware intrinsics on x64 and ARM64 platforms. Particularly, the combination of ARM64 with LLVM is interesting to represent the iOS workload. I am not sure what is the best way to do it. For my performance runs I was using the LLVM JIT mode which is not accurate estimation of real-life performance but a good estimate for relative performance numbers on micro-benchmarks. This can be enabled via the Anyway, I am happy that this is being done! |
|
I have been working with @SamMonoRT and he walked us through the four configurations that y'all find interesting right now, with instructions on how to run each of them. The plan is that we will be running all of them at least once a day. |
| echo " --kind <value> Related to csproj. The kind of benchmarks that should be run. Defaults to micro" | ||
| echo " --runcategories <value> Related to csproj. Categories of benchmarks to run. Defaults to \"coreclr corefx\"" | ||
| echo " --internal If the benchmarks are running as an official job." | ||
| echo " --monodotnet Pass the path to the mono dotnet for mono performance testing." |
There was a problem hiding this comment.
I don't think we should do this. It'd be better to just use self-contained build
There was a problem hiding this comment.
Can you explain this a little more? Why is using the patched mono dotnet not a good path forward? This is similar to what we do for the coreclr based runtime tests.
Also, the way that we currently run performance tests relies on having a version of dotnet to build and run the Benchmark DotNet tests, and as far as I know we cannot do a self-contained publish of the performance tests. (@billwert for confirmation).
There was a problem hiding this comment.
as far as I know we cannot do a self-contained publish of the performance tests.
I have been doing that for months on my infrastructure so at least in theory it's possible. On the other hand it's not easy and useful for most cases because BDN by default triggers the compilation for the benchmarks and runs each of them out of process.
For non-AOT benchmarks I ended up using/patching testhost that is produced as output of the dotnet/runtime repository. It it guaranteed to be self-consistent unlike the combination of SDK version determined by global.json and the runtime built from the repository.
For AOT I rely on self-publish and in-process mode of BDN. Not sure if I would recommend that.
There was a problem hiding this comment.
Why is using the patched mono dotnet not a good path
This is not set up we'll ever officially support and can break anything as it was a temporary hack. It also won't support the official AOT mode and possibly other configs. So if you are testing this hack then you are testing config which no customer will use
There was a problem hiding this comment.
For some context, we run the performance testing for the coreclr runtime in two ways. The first is on the pacakges from the installer repo. This is to make sure that we are running the performance tests in a way that the customer will actually use the product. We also run the tests here in the runtime repo by building the product and then making a corerun directory that contains all of our built binaries. This, while not the way our customers use or obtain the product, allows us to get much more granular performance data collected as we don't have to wait for code to flow all the way from here to the installer repo.
This work was designed to get mono to parity with what we are doing here in the runtime repo. So building the product from head and then making up a version of dotnet that contains the built binaries and running the tests. If using the dotnet-mono patch feature is not the right way to do this, please direct me to the canonical way to do this. I was just operating off of the steps given to me by @SamMonoRT.
- When you say you have been doing this for months, do you mean the tests that live in the dotnet/performance repo? If so I would love to get instructions on how you are doing that.
- I would also be interested in how you are doing the AOT testing as well. This is something that we would like to be able to support y'all on, so getting an idea of how y'all are building and running these tests, even if we have to take a different path would be useful.
There was a problem hiding this comment.
So it seems like a good place to land here is to move the performance runs over to the testhost construction, as that seems to be what y'all have been doing in the past. We can then go ahead and get this checked in and running for Windows and Linux and for LLVM and non-LLVM backend. Then we can file an issue that tracks doing the work to move over to the in tree runtime packs as soon as we have good support for that on desktop builds. Does that sound like a good way forward to everyone here?
There was a problem hiding this comment.
Sounds good to me - @steveisok @marek-safar thoughts ?
There was a problem hiding this comment.
AOT won't work this way for sure and any issues with this setup won't be fixed. I'm fine to land this but we should start working on a supported way to run the tests as this can break anything.
There was a problem hiding this comment.
So I did some digging and this is the step that seems to setup the testhost for running functional tests on mono, link. I would assume that since this is how the product is tested in CI that this should be a supported scenario, yes?
If this is the case I am just going to use the same command to setup the testhost and run the performance tests in this way for now.
If this is also unsupported does that mean that the functional CI tests are running in an unsupported manner or am I just missing something here?
There was a problem hiding this comment.
I think that's as close to supported desktop scenario as it is currently available (until desktop runtime packs are introduced).
SamMonoRT
left a comment
There was a problem hiding this comment.
Interesting you chose llvm over the interpreter, but should be same concept. Thank you !
|
I am right now just setting up the config flags we pass for reporting. And was going to add the actual build and run of those configs in a later PR, but you make a good point. I need to make sure that we are already setting all of those configs on this run. |
|
We have failures in the Perf_Image tests on Windows. This is the error information. I have done some digging here and it seems to be a buffer overrun message. I have tried running the test manually, and under a debugger to get more information, but when I do it just hangs on // BeforeRun. Any ideas @AndreyAkinshin @tannergooding @adamsitnik |
If you want to run the auto-generated project manually you need to:
In the BDN docs you can find some more user-friendly alternatives |
|
I have taken out Windows for now, as we have too many tests that don't work on Mono. Everything looks green or is on it's way. Could I get someone to approve this checkin? |
|
Fairly certain that the COreCLR Pri0 Runtime Tests failures are unrelated to this change. |
|
LGTM. |
|
Awesome, thanks Drew ! |
No description provided.