Skip to content

Track AUTO mode telemetry and classifier parity #4475

@qqqys

Description

@qqqys

Summary

Follow up on the recent AUTO mode safety alignment work. Qwen now has the core
behavior for reason separation, PermissionDenied hooks, and cumulative denial
caps, but it still lacks richer telemetry and classifier metadata needed for
debugging and fleet-level monitoring.

Current State

  • AUTO mode distinguishes unavailable, fallback-to-ask, and permission-denied
    reasons.
  • PermissionDenied fires only when the AUTO classifier blocks a tool call.
  • Fallback-to-ask remains separate from PermissionDenied.
  • Denial tracking supports both consecutive and total denial caps.

Remaining Gap

Qwen should expose richer AUTO mode observability:

  • denial-limit-exceeded metadata with cap kind: consecutive or total
  • fallback-to-ask reason metadata
  • classifier decision metadata, including classifier stage and model
  • request/message identifiers where the provider stack exposes them
  • span attributes for auto-mode fallback, cap kind, and denial counters

Qwen currently has the safety behavior but not enough structured signal to
debug why AUTO mode allowed, blocked, or fell back to manual approval.

Proposed Work

  • Add OTel/span attributes for AUTO mode decisions and fallback:
    • qwen-code.auto_mode.fallback_reason
    • qwen-code.auto_mode.cap_kind
    • qwen-code.auto_mode.consecutive_denials
    • qwen-code.auto_mode.total_denials
  • Surface a denial_cap_triggered decision or equivalent on the
    blocked-on-user span when denial tracking forces fallback.
  • Add classifier metadata to the decision path where available:
    • classifier stage
    • classifier model
    • request/message identifiers
  • Preserve the semantic boundary:
    • classifier block -> PermissionDenied
    • fallback-to-ask -> PermissionRequest / manual approval path
    • denial cap -> fallback-to-ask, not PermissionDenied

Acceptance Criteria

  • Total-cap and consecutive-cap fallback can be distinguished in telemetry.
  • Ask-rule fallback can be distinguished from denial-cap fallback.
  • PermissionDenied remains limited to classifier-denied calls.
  • Tests cover denial-cap telemetry metadata and fallback reason metadata.
  • Developer-facing docs or constants describe the new AUTO mode telemetry
    fields.

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions