Right now if you utilize the repo level prebuilt baseline infrastructure from arcade, you will have to add exclusions for the source-build intermediate packages.

Example from source-build-reference-packages - https://github.com/dotnet/source-build-reference-packages/blob/main/eng/SourceBuildPrebuiltBaseline.xml#L7

<UsagePattern IdentityGlob="Microsoft.SourceBuild.Intermediate.*/*" />

Given these are common prebuilts they should be included as part of the infrastructure rather than requiring each repo to add these exclusions.