Skip to content

Internal pipeline: Validate Signing fails with "No space left on device" on Linux #5480

@steveisok

Description

@steveisok

Summary

The "Validate Signing" task in internal unified-build pipeline runs out of disk space on Linux and Mac validation legs, causing builds to report partiallySucceeded. This has been recurring on main and release/10.0.1xx branches for at least 2 days.

Error

/mnt/vss/_work/1/s/eng/signing-validation.proj(20,7): error : Signing validation failed: No space left on device

Affected Builds

Build Branch Date
2928004 release/10.0.1xx Mar 17
2926555 main Mar 14
2926429 release/10.0.1xx Mar 14
2926017 release/10.0.1xx Mar 13
2925742 release/10.0.1xx Mar 13

Analysis

The signing-validation task iterates over all per-vertical artifact manifests and unpacks them for sign-check. With the growing number of verticals (30+ manifests observed in build 2926555 logs), the aggregate disk consumption exceeds the available space on the agent.

The failure occurs in the "Validate Signing - Linux" job, suggesting the issue is systematic rather than one-off.

Impact

  • Every internal rolling build on release/10.0.1xx reports partiallySucceeded
  • Internal main builds are also affected
  • The partiallySucceeded status masks real failures and makes pipeline health assessment unreliable

Possible Mitigations

  • Use larger disk agents for signing validation jobs
  • Stream/validate manifests incrementally instead of unpacking all at once
  • Clean up between vertical validations

/cc @dotnet/dnceng

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions