Skip to content

maui-pr-uitests: File upload failures from artifact name collision across 98 test jobs #34477

@lewing

Description

@lewing

Problem

The maui-pr-uitests pipeline (definition 313) has 98+ test jobs that all publish artifacts using PublishBuildArtifacts@1 with no artifact name, defaulting to drop. When multiple jobs complete around the same time and upload the same file path to the same drop container, AzDO blob storage encounters write conflicts, causing intermittent "File upload failed even after retry" errors.

Recent failures

Build Failed Jobs Error
1334980 Controls CollectionView, Controls (vlatest), Controls (vlatest) CollectionView Blob is incomplete (missing block)
1334245 Controls (vlatest) File upload failed even after retry

Root cause

In eng/pipelines/common/ui-tests-steps.yml, the publish step has no inputs:

- task: PublishBuildArtifacts@1
  condition: always()
  displayName: publish artifacts

This defaults to artifact name drop and publishes from $(Build.ArtifactStagingDirectory). All 98 test jobs upload to the same container.

The specific collision occurs when two jobs upload the exact same file. For example, both "Controls (vlatest)" and "Controls (vlatest) CollectionView" upload drop/logs/appium_ios_Controls.TestCases.iOS.Tests-Release-ios-CollectionView.log — same blob ID 29adda685a1ff1119a49000d3a9183a5. The concurrent uploads produce:

Blob is incomplete (missing block). Blob: 29adda685a1ff1119a49000d3a9183a5, Expected Offset: 0, Actual Offset: 8388608

This is a well-known AzDO artifact upload pattern (see dotnet/dnceng#1916).

Suggested fix

Add a unique artifact name per job using $(System.StageName) and $(System.JobName), similar to how snapshot diffs are already handled in ui-tests-collect-snapshot-diffs.yml:

- task: PublishBuildArtifacts@1
  condition: always()
  displayName: publish artifacts
  inputs:
    artifactName: uitest-logs-$(System.StageName)-$(System.JobName)-$(System.JobAttempt)

This ensures each job uploads to its own artifact container, eliminating the collision.

Note: If downstream tooling expects a single drop artifact, an alternative approach is to switch to PublishPipelineArtifact@1 which handles concurrent uploads better, or to add the job name to the file path prefix instead of the artifact name.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-infrastructureCI, Maestro / Coherency, upstream dependencies/versions

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions