Spmi replay pipeline by kunalspathak · Pull Request #56871 · dotnet/runtime

kunalspathak · 2021-08-04T21:18:58Z

Introduce superpmi-replay public pipeline that will be triggered after JIT changes. Here is how it works:

It will perform windowx-x64 build to cross-compile following jit binaries:
- clrjit_win_x64_x64.dll
- clrjit_win_arm64_x64.dll
- clrjit_unix_x64_x64.dll
- clrjit_unix_arm64_x64.dll
It will perform windowx-x86 build to cross-compile following jit binaries:
- clrjit_win_x86_x86.dll
- clrjit_unix_arm_x86.dll
It will spawn 6 machines corresponding to the 6 configurations about to run superpmi replay for below environment variables:
- "JitStressRegs=0" (default - no flags)
- "JitStressRegs=1"
- "JitStressRegs=2"
- "JitStressRegs=3"
- "JitStressRegs=4"
- "JitStressRegs=8"
- "JitStressRegs=0x10"
- "JitStressRegs=0x80"
- "JitStressRegs=0x1000"
In each of the 6 runs, it will download all the collections corresponding to the OS/architecture and do replay for each of the flags and upload a consolidated superpmi__arch.log as an artifact. Currently the pipeline fails because of a bug in helix tracked by https://github.com/dotnet/core-eng/issues/13983.

Changes:

Add superpmi-replay public pipeline
Added option for --no_progress in "superpmi download" so we do not see the progress logs during mch downloading.
Added a template of download-specific-artifacts.yml.

Since I have introduced superpmi-replay** files, I will submit a follow-up PR to rename existing superpmi* files that does the collection to superpmi-collect**.

Also, as a follow-up work, we would trigger this pipeline after superpmi-collect pipeline completes so the collection are fresh and we don't get "missing key errors".

Fixes: #52392

kunalspathak · 2021-08-05T16:27:22Z

@dotnet/jit-contrib

kunalspathak · 2021-08-11T15:40:17Z

Can someone review this? Want to make sure that this goes in before we freeze the main.

eng/pipelines/common/download-specific-artifact-step.yml

echesakov · 2021-08-11T16:36:49Z

eng/pipelines/coreclr/templates/build-jit-job.yml

Suggested change

- ${{ if eq(parameters.uploadAsArtifacts, false) }}:

- ${{ if not(parameters.uploadAsArtifacts) }}:

eng/pipelines/coreclr/templates/build-jit-job.yml

eng/pipelines/coreclr/templates/run-superpmi-job.yml

eng/pipelines/coreclr/templates/run-superpmi-replay-job.yml

eng/pipelines/common/download-specific-artifact-step.yml

ghost · 2021-08-11T20:58:01Z

Hello @kunalspathak!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

briansull

Looks Good to Me

BruceForstall · 2021-08-25T23:26:15Z

@kunalspathak Love this work!

Are you planning to move the pipeline from internal to public, so it can be auto-triggered or manually triggered on JIT PR's? Or is there some reason why it must remain on "internal" and run post-merge?

kunalspathak · 2021-08-25T23:43:26Z

We could definitely make it public but there is one scenario in which it doesn't work - If the PR changes the JITEE guid, then we don't get new collection for new guid because we haven't collected it. What should ideally happen in such cases is that collection pipeline should trigger the replay pipeline. But other than that, this can be moved to public (let me know if you plan on doing it).

BruceForstall · 2021-08-26T20:53:06Z

The "run" legs of the pipeline take 1.5 or 2 hours. Is that because they do all the work on one Helix machine each? Would it be hard to split the work up to run more in parallel? We'd need to do this if we want to run in a PR.
Did implementing this find any asserts in the JIT? Did you test that if an assert is found, the job is marked as failing?

kunalspathak · 2021-08-26T22:35:21Z

The "run" legs of the pipeline take 1.5 or 2 hours. Is that because they do all the work on one Helix machine each? Would it be hard to split the work up to run more in parallel? We'd need to do this if we want to run in a PR.

Currently, the partitions are hardcoded in superpmi-replay.proj. Further splitting based on collection would definitely reduce the run time, but the list in below file will grow (no. of collection X 6 entries), but is definitely do-able.

runtime/src/coreclr/scripts/superpmi-replay.proj

Lines 45 to 55 in 7994e59

    
           <ItemGroup Condition="'$(Architecture)' == 'x64'"> 
        
             <SPMI_Partition Include="win-x64" Platform="windows" Architecture="x64" /> 
        
             <SPMI_Partition Include="win-arm64" Platform="windows" Architecture="arm64" /> 
        
             <SPMI_Partition Include="unix-x64" Platform="Linux" Architecture="x64" /> 
        
             <SPMI_Partition Include="unix-arm64" Platform="Linux" Architecture="arm64" /> 
        
           </ItemGroup> 
        
           <ItemGroup Condition="'$(Architecture)' == 'x86'"> 
        
             <SPMI_Partition Include="win-x86" Platform="windows" Architecture="x86" /> 
        
             <SPMI_Partition Include="unix-arm" Platform="Linux" Architecture="arm" /> 
        
           </ItemGroup>

Did implementing this find any asserts in the JIT? Did you test that if an assert is found, the job is marked as failing?

I didn't see any assert yet. The pipeline failed once and reported when we didn't had .mch files to run on.

ghost added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Aug 4, 2021

kunalspathak changed the title ~~Spmi replay~~ Spmi replay pipeline Aug 5, 2021

kunalspathak force-pushed the spmi-replay branch from 0443b0a to a09bac7 Compare August 5, 2021 16:25

kunalspathak marked this pull request as ready for review August 5, 2021 16:27

echesakov reviewed Aug 11, 2021

View reviewed changes

eng/pipelines/common/download-specific-artifact-step.yml Outdated Show resolved Hide resolved

echesakov reviewed Aug 11, 2021

View reviewed changes

kunalspathak added 22 commits August 11, 2021 11:26

Add superpmi-replay pipeline file

9c1284e

Add superpmi-replay-job.yml that downloads collection

60b44b4

Moved superpmi-replay-job to right location

1919da8

make sure to have spmi collection after jit build

f85e072

annotate displayName with windows/OSX

224adcd

Make osGroup=windows

33072e3

Add helixType (if that matters)

207ce41

Add some more variables

84e3dfe

skip restore internal tools

2b232d0

Just use build-jit-job directly

971d198

Hardcode windows.x64.checked

bc571f9

Add -download_all -download_raw options to superpmi.py

f7f77ae

Seperate 2 jobs

a7bdc5d

Fix Upload artifacts

fabab8a

Fix uploadAsArtifacts

9f8a0ba

Fix to uploadAsArtifacts

08650f6

comment upload to azure storage temporarily

d14111f

Set variables inside superpmi-replay-job

a3916fe

Entire pipeline

fcd9855

minor fix

dd12070

Add some logging

b136c9d

Fix the path

333927e

kunalspathak added 16 commits August 11, 2021 11:27

fix the appending logic

dc5ca48

Remove download-all and download-raw and instead add -no-progres

b9ba241

do not show download progress, try fixing download file

4d97688

TO REVERT: Just try benchmark on windows x64/x86

53eb396

Use uploadAsArtifacts in build-jit-job

0c12c61

Create dummy.txt in workitem directory

4a48c37

create workitem_directory

1c14146

Build clrjit

8fbcace

fix build-jit-job

5a3168d

add dependsOn

f796b6a

print params

deeb625

TO REVERT: Just run jitbuild win-x64

faa8294

minor

7c73aa6

move under jobParameters

b06725f

Cleanup

a729794

review feedback

c8dc88a

kunalspathak force-pushed the spmi-replay branch from a09bac7 to c8dc88a Compare August 11, 2021 18:28

kunalspathak added the auto-merge label Aug 11, 2021

briansull approved these changes Aug 11, 2021

View reviewed changes

ghost merged commit 0812322 into dotnet:main Aug 11, 2021

kunalspathak mentioned this pull request Aug 13, 2021

Fix no_progress parameter in superpmi #57377

Merged

This was referenced Sep 22, 2021

Move SuperPMI replay pipeline to "public" #59443

Closed

Create checked/release SuperPMI asm diffs AzDO pipeline #49949

Closed

superpmi: create asm diffs AzDO pipeline #59445

Closed

ghost locked as resolved and limited conversation to collaborators Sep 25, 2021

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spmi replay pipeline#56871

Spmi replay pipeline#56871
66 commits merged intodotnet:mainfrom
kunalspathak:spmi-replay

kunalspathak commented Aug 4, 2021 •

edited

Loading

Uh oh!

kunalspathak commented Aug 5, 2021

Uh oh!

kunalspathak commented Aug 11, 2021

Uh oh!

Uh oh!

echesakov Aug 11, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ghost commented Aug 11, 2021

Uh oh!

briansull left a comment

Uh oh!

BruceForstall commented Aug 25, 2021

Uh oh!

kunalspathak commented Aug 25, 2021

Uh oh!

BruceForstall commented Aug 26, 2021

Uh oh!

kunalspathak commented Aug 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	- ${{ if eq(parameters.uploadAsArtifacts, false) }}:
	- ${{ if not(parameters.uploadAsArtifacts) }}:

Conversation

kunalspathak commented Aug 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kunalspathak commented Aug 5, 2021

Uh oh!

kunalspathak commented Aug 11, 2021

Uh oh!

Uh oh!

echesakov Aug 11, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ghost commented Aug 11, 2021

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

Uh oh!

briansull left a comment

Choose a reason for hiding this comment

Uh oh!

BruceForstall commented Aug 25, 2021

Uh oh!

kunalspathak commented Aug 25, 2021

Uh oh!

BruceForstall commented Aug 26, 2021

Uh oh!

kunalspathak commented Aug 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kunalspathak commented Aug 4, 2021 •

edited

Loading

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.