Skip to content

[Chore] test_site_counts.sh — close synonym-coverage + multiline gaps flagged on PR #340 #342

@atlas-apex

Description

@atlas-apex

Driver

PR #340 (closes #325 + #326 + #331 + #332) shipped the count-drift smoke test at .claude/hooks/tests/test_site_counts.sh along with AgDR-0046 documenting the chosen drift-prevention mechanism. AgDR-0046's Consequences section explicitly flagged a v1 risk: pattern coverage is naturally narrow, and any new copy that uses synonyms or wraps a count across a newline is invisible to the smoke test.

Rex's review of #340 (verdict APPROVED, non-blocking nit) found the exact instances of that risk in site/llms-full.txt:

The smoke test missed all three because:

  1. Pattern phrasing mismatchshell +scripts? / mechanical +gates? don't match shell hooks / mechanical enforcement scripts
  2. Multiline gap — line-by-line grep can't bridge 52\n slash commands (count and noun on separate lines)

This is the exact failure mode AgDR-0046 § Consequences anticipated. Worth closing in v1.1 of the drift detector.

Scope

Single PR, two-part fix:

1. Refresh the 3 stale strings in site/llms-full.txt

  • Line 92: 5253 (skills count)
  • Line 131: 2429 (hooks count)
  • Line 132-133: 52\n slash commands53\n slash commands (skills count, multiline)

2. Extend test_site_counts.sh to catch the missed patterns

Add 2 sub-fixes:

a) Broaden the regex patterns to match the synonym phrasings:

  • Hooks: add shell hooks? + enforcement scripts? + mechanical (enforcement )?(scripts?|hooks?) to the hook-count pattern
  • Skills: add (slash )?commands? (already-partial) + ensure [0-9]+\s+slash\s+commands? matches with \s+ (any whitespace including newline)

b) Add a multiline-flatten pre-pass

Before running the line-by-line patterns, also run a "flattened" pass over each file:

# Flatten consecutive whitespace (including newlines) into single spaces
flattened=$(tr -s '[:space:]' ' ' < "$file")
# Then apply the same patterns against the flattened text

This catches 52\n slash commands because after flatten it becomes 52 slash commands which matches [0-9]+ slash commands?.

Report both the line-numbered hits (for actionable fixes) AND the flattened-text hits (for completeness). When flatten-only hits surface, name them as "multiline-spanned drift" so the operator can find them by inspecting the file region rather than expecting line:column precision.

3. Smoke-test the smoke test

Extend .claude/hooks/tests/test_site_counts.sh itself with a self-test: when run with an env flag like APEXYARD_TEST_SMOKE=1, plant deliberate drift in temp fixture files at known synonyms + multiline patterns, run the detector against the fixture, and assert it catches each. Fail loud if any are missed. Keeps the detector honest as patterns evolve.

Acceptance Criteria

  • All 3 stale-count strings in site/llms-full.txt refreshed (53 skills, 29 hooks)
  • test_site_counts.sh patterns extended with the synonym phrasings from Rex's nit
  • Multiline-flatten pre-pass added to the detector (catches 52\n slash commands)
  • Self-test fixture path exists at .claude/hooks/tests/fixtures/test_site_counts/ (or similar) with deliberate drift planted
  • Self-test run via APEXYARD_TEST_SMOKE=1 bash test_site_counts.sh passes
  • No regression on the existing PASS state of site/*.html (drift detector still PASS green there)
  • AgDR-0046 amended with a short "Update — v1.1" entry in Consequences noting the synonym-coverage + multiline gap fix landed

Out of scope

  • Adopters extending the detector for their own custom count claims (separate feature ticket if requested)
  • Auto-fix mode (the detector reports drift; an operator decides whether to refresh manually)
  • Cross-file count-aggregation (current scope is per-file)

Glossary

Term Definition
Drift detector The test_site_counts.sh smoke test that asserts framework counts in site copy match find output
Multiline-flatten pre-pass Run regex patterns against a whitespace-collapsed version of the file (joins counts and nouns split across newlines)
Synonym-coverage gap The pattern set missing valid English-language variants of the count phrasing (shell hooks vs shell scripts)
AgDR-0046 § Consequences Source-of-truth section that flagged exactly this v1 risk

Refs #340 / Rex review nit (non-blocking) / AgDR-0046 § Consequences

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low — ergonomics, nice-to-haveenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions