Skip to content

[doc-healer] DDUw improvement: cross-reference open documentation issues and scan for stale doc-code drift #18554

@github-actions

Description

@github-actions

Background

The Daily Documentation Healer ran on 2026-02-26 and analyzed 17 recently-closed documentation issues (closed ≥ 2026-02-19). Of these, 5 were closed as completed:

Issue Title Root cause type
#18090 MultiRepoOps example: steps: nested under engine: (compilation error) Stale doc example not matching current schema
#17941 Update docs for removal of tools.playwright.allowed_domains Long-standing doc-code drift (feature removed in v0.9.0)
#16863 Remove emoji from CLI reference heading Doc quality issue unrelated to recent code changes
#16827 CLI consistency issues (wrong flag letter -f vs -F, wrong registry URL) Code–help-text discrepancy
#17800 Add github-ci-trogger-token: option to trigger CI New feature not documented

Zero DDUw PRs were created in the same 7-day window. All fixes were applied manually or via other agents, not by DDUw.

All documentation gaps have since been resolved in the codebase. This issue proposes concrete improvements to prevent recurrence.


Root Cause Analysis

DDUw's current logic (Step 1 in daily-doc-updater.md) is:

Scan for merged PRs from the last 24 hours → analyze features added/removed/modified → update docs.

This design has three structural blind spots:

Blind Spot 1: No cross-referencing of open documentation issues

DDUw never reads the GitHub issue tracker. Issues like #18090 and #17941 were open for days before being manually closed — DDUw would have caught them automatically if it searched for open documentation issues alongside its PR scan.

Proposed fix: Add a new step after "Scan Recent Activity" that queries:

repo:github/gh-aw is:issue is:open label:documentation

For each open issue found, check whether the described gap exists in the current docs. If it does, fix it as part of the PR. Reference and close the issue in the PR body.

Blind Spot 2: 24-hour lookback misses long-standing doc-code drift

tools.playwright.allowed_domains was removed in v0.9.0 but references persisted in docs. DDUw would only have caught this the day of that removal PR. Any drift that survives beyond 24 hours is permanently invisible to DDUw.

Proposed fix: Add a periodic "stale reference scan" step (can run weekly or on workflow_dispatch) that:

  1. Extracts all property names documented in tools: sections across docs
  2. Cross-checks them against the current JSON schema (pkg/schema/*.json or compiled schema output)
  3. Flags any documented properties that no longer exist in the schema

Blind Spot 3: No doc example validation

The MultiRepoOps example in #18090 had steps: incorrectly nested under engine:, which the compiler rejects. DDUw doesn't validate that YAML code blocks in docs are structurally valid.

Proposed fix: Add a step that runs gh aw compile --dry-run (or equivalent) on YAML snippets extracted from docs that contain --- frontmatter fences. Flag any that fail to compile.


Specific Wording Changes for daily-doc-updater.md

Add to Step 1 (after existing PR/commit scan):

### 1b. Check Open Documentation Issues

Search for open issues labeled `documentation` that may represent unaddressed gaps:

```
repo:$\{\{ github.repository }} is:issue is:open label:documentation
```

For each open issue:
1. Read the issue body to understand the described gap.
2. Check the referenced documentation file to verify the gap still exists.
3. If confirmed, include a fix in this run's PR and reference the issue with `Closes #NNN`.
4. If the gap is already fixed, note it (do not reopen or comment on the issue).

Add to Step 2 (Analyze Changes) — new bullet:

- **Removed Features in Docs**: Search docs for references to properties, flags, or options that no longer exist in the current schema. Check `pkg/schema/` or run `gh aw compile` on representative workflows to confirm current valid properties.

Add to Guidelines section:

- **Issue-Driven**: Proactively check open `documentation` issues — do not wait for them to be reported manually.
- **Validate Examples**: YAML frontmatter examples in docs must be structurally valid. When in doubt, test with `gh aw compile`.

Expected Impact

These changes would allow DDUw to detect and fix the following classes of issues autonomously:

  • Stale property references (like allowed_domains) that survive API removal
  • Doc examples that fail compilation
  • Open documentation issues that have not been addressed within 24 hours of being filed

No architectural changes to DDUw are needed — these are additive scan steps.

Generated by Daily Documentation Healer

  • expires on Mar 1, 2026, 7:26 PM UTC

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions