Currently, summarization is done for each AzDO job that spawns all the Helix work items. Thus, we end up with a section for Windows x64 and one for Windows x86 -- and each section contains all the runs that happened in that section (e.g., in the "Windows x86" section, there are both asmdiffs and tpdiff for both Windows x86 and Linux arm).

It was implemented this way for convenience, but the output summarization could be better, and more integrated.

There would be a "Summarization" AzDO job that depends on all the jobs that compute the diffs, and it should take all the results and compute the final summary file that becomes the "Extensions" page (via the magic ##vso[task.uploadsummary] tag). The superpmi_diffs_summarize.py script would run in this job.

Alternatively, there could be a single AzDO job that kicks off all the Helix jobs. It would have to depend on all the builds, so starting the Helix work would be unnecessarily waiting on builds that not all Helix work requires. However, it might be much simpler to implement better summarization with a single AzDO job, and be more robust than synchronizing multiple "work" jobs.

category:eng-sys
theme:super-pmi
skill-level:intermediate
cost:medium
impact:small