Skip to content

Add check_changelog_updated.py: pre-push + CI CHANGELOG enforcement (Closes #478)#479

Merged
aallan merged 4 commits into
mainfrom
check-changelog-updated
Apr 16, 2026
Merged

Add check_changelog_updated.py: pre-push + CI CHANGELOG enforcement (Closes #478)#479
aallan merged 4 commits into
mainfrom
check-changelog-updated

Conversation

@aallan

@aallan aallan commented Apr 16, 2026

Copy link
Copy Markdown
Owner

Closes #478.

Summary

Adds scripts/check_changelog_updated.py, which diffs the current branch against origin/main and fails if any substantive file (vera/, spec/, SKILL.md) was changed without a matching new entry in CHANGELOG.md. Wired into the pre-commit chain at the pre-push stage and into the CI lint job.

Catches the kind of missed release-prep that happened on #474 (calls.py decomposition merged without a CHANGELOG entry).

Files

File Kind
scripts/check_changelog_updated.py New — ~180 lines
tests/test_check_changelog_updated.py New — 54 unit + end-to-end tests
.pre-commit-config.yaml New hook at pre-push stage
.github/workflows/ci.yml New step in lint job + fetch-depth: 0
CONTRIBUTING.md New "Pre-push hook: CHANGELOG enforcement" subsection
TESTING.md Hook count 23 → 24, test count 3,253 → 3,307, new row
ROADMAP.md Test count + totals line
CHANGELOG.md New [Unreleased] Added entry

Classification

Substantive (requires a CHANGELOG entry): vera/, spec/, SKILL.md.

Exempt: tests/, scripts/, .github/, docs/, examples/, editors/, assets/; all root-level docs (README.md, HISTORY.md, ROADMAP.md, KNOWN_ISSUES.md, FAQ.md, CONTRIBUTING.md, TESTING.md, AGENTS.md, CLAUDE.md, DE_BRUIJN.md, EXAMPLES.md, CHANGELOG.md, LICENSE); pyproject.toml, uv.lock, .pre-commit-config.yaml, .coderabbit.yaml, .gitignore.

Unknown paths default to substantive (conservative — a hypothetical new stdlib/ or runtime/ folder won't accidentally bypass the check).

Escape hatches

  • Git-native: a Skip-changelog: <reason> trailer in any commit message on the branch (works locally and in CI).
  • GitHub-native: add the skip-changelog label to the PR (CI-only; the if: condition on the step skips it when the label is present).

Why pre-push, not pre-commit

Feature branches typically have 5–20 commits. Running the check per-commit would block every intermediate commit with "no CHANGELOG entry" — pure friction. Pre-push gives the whole branch's worth of commits before checking. Enable locally with pre-commit install --hook-type pre-push (CONTRIBUTING.md updated).

Self-test

Running the script on this PR exits 0 — all files touched are exempt. The CHANGELOG entry added here is a bonus (following project convention for CI tooling, mirroring the v0.0.107 pattern), not a requirement. The substantive-without-CHANGELOG path is covered by the end-to-end pytest suite against temp git repos.

Verification

  • check_changelog_updated.py self-test (exit 0)
  • mypy clean
  • 3,296 tests pass (54 new), 11 skipped
  • doc counts consistent (3,307 tests, 24 hooks)
  • version sync + site assets clean

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added automated CHANGELOG enforcement running in CI and as an optional local pre-push check.
  • Chores

    • Enforced CHANGELOG updates for substantive repository changes; opt-out via a commit trailer or PR label.
  • Tests

    • Added comprehensive unit and end-to-end tests covering changelog-check logic and real git scenarios.
  • Documentation

    • Updated CONTRIBUTING, CHANGELOG, TESTING, ROADMAP and HISTORY to document the checks, setup and escape hatches.

…478)

Closes #478.

Adds scripts/check_changelog_updated.py, which diffs the current
branch against origin/main and fails if any substantive file
(vera/, spec/, SKILL.md) was changed without a matching new
entry in CHANGELOG.md.

## Files

- scripts/check_changelog_updated.py: the check itself
- tests/test_check_changelog_updated.py: 54 unit + end-to-end tests
  (classification, diff parsing, trailer detection, temp-repo
  integration covering substantive/exempt/label/trailer paths)

## Integration

- .pre-commit-config.yaml: new 'check-changelog-updated' hook at
  'pre-push' stage (not per-commit, to avoid feature-branch noise)
- .github/workflows/ci.yml: new step in the 'lint' job with
  'fetch-depth: 0' so origin/main is available for the diff; the
  step is skipped when the PR has a 'skip-changelog' label

## Escape hatches

- Commit trailer 'Skip-changelog: <reason>' (Git-native, local + CI)
- PR label 'skip-changelog' (CI-only)
- CHANGELOG_CHECK_BASE=<ref> env var to override the base ref

## Documentation

- CONTRIBUTING.md: new 'Pre-push hook: CHANGELOG enforcement'
  subsection documenting the rule, install command, and escape
  hatches; install instructions updated to 'pre-commit install
  --hook-type pre-push'
- TESTING.md: hook count 23 -> 24, test count 3,253 -> 3,307,
  new row for test_check_changelog_updated.py
- ROADMAP.md: test count 3,307 + v0.0.113 totals line
- CHANGELOG.md: new '[Unreleased] Added' entry documenting the
  rule

Motivation: #474 merged without a CHANGELOG entry, HISTORY.md
update, or KNOWN_ISSUES.md cleanup because none of those files
were in the commit — the existing hooks only run on files they
touch. This check reverses the logic: run always, check that the
PR diff is self-consistent.

Co-Authored-By: Claude <noreply@anthropic.invalid>
@coderabbitai

coderabbitai Bot commented Apr 16, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: d716328c-ab9e-42e3-bd2b-d7c9706e655b

📥 Commits

Reviewing files that changed from the base of the PR and between 2c6dadf and c406d33.

📒 Files selected for processing (5)
  • CONTRIBUTING.md
  • ROADMAP.md
  • TESTING.md
  • scripts/check_changelog_updated.py
  • tests/test_check_changelog_updated.py

📝 Walkthrough

Walkthrough

The PR adds enforcement that substantive changes (e.g., vera/, spec/, SKILL.md) require a CHANGELOG.md entry by introducing scripts/check_changelog_updated.py, running it in CI and as a local pre-push hook, with Skip-changelog: commit trailers and a skip-changelog PR label as escape hatches.

Changes

Cohort / File(s) Summary
CI Workflow
​.github/workflows/ci.yml
actions/checkout now fetches full history (fetch-depth: 0). Adds a lint step to run python scripts/check_changelog_updated.py, skipped for PRs labelled skip-changelog.
Pre-commit Hook
.pre-commit-config.yaml
Adds a local pre-push hook check-changelog-updated that runs python scripts/check_changelog_updated.py with pass_filenames: false and always_run: true.
Core Script & Tests
scripts/check_changelog_updated.py, tests/test_check_changelog_updated.py
New script diffs against a base ref (configurable via CHANGELOG_CHECK_BASE, otherwise origin/main/main), classifies changed paths as substantive/exempt, checks for added CHANGELOG entries (new version heading or bullets under [Unreleased]), and supports commit-trailer or env/PR-label bypasses. Tests cover unit parsing and end-to-end git scenarios.
Docs & Metadata
CHANGELOG.md, CONTRIBUTING.md, TESTING.md, ROADMAP.md, HISTORY.md
Document the new enforcement, pre-push hook install instructions, updated hook/test counts and metrics, and HISTORY/CHANGELOG entries describing the change.

Sequence Diagram(s)

sequenceDiagram
    participant Dev as Developer
    participant Local as Pre-push hook (local)
    participant CI as CI lint job
    participant Git as Git remote / origin/main
    participant Script as check_changelog_updated.py
    participant CH as CHANGELOG.md

    Dev->>Local: git push
    Local->>Git: fetch base (origin/main)
    Local->>Script: run check (diff HEAD..base)
    Script->>Git: git diff & git log
    Script->>CH: inspect CHANGELOG diff
    alt changelog present or skip
        Script-->>Local: exit 0 (allow push)
    else
        Script-->>Local: exit 1 (block push)
    end

    CI->>Git: workflow run (push/PR)
    CI->>Script: run check (same flow)
    Script-->>CI: pass/fail outcome
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested labels

tests, ci, docs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately identifies the main change: adding a new script for pre-push and CI CHANGELOG enforcement, with the issue reference.
Linked Issues check ✅ Passed All major objectives from #478 are met: the script diffs against origin/main, classifies files as substantive/exempt, requires CHANGELOG updates, provides Skip-changelog trailer and skip-changelog label escape hatches, and integrates via pre-push hook and CI lint job.
Out of Scope Changes check ✅ Passed All changes directly support the #478 objectives. Auxiliary files (CONTRIBUTING.md, TESTING.md, ROADMAP.md, HISTORY.md) properly document the new feature without introducing unrelated changes.
Docstring Coverage ✅ Passed Docstring coverage is 88.10% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch check-changelog-updated

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov

codecov Bot commented Apr 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.37%. Comparing base (56f14ad) to head (c406d33).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #479   +/-   ##
=======================================
  Coverage   90.37%   90.37%           
=======================================
  Files          58       58           
  Lines       19588    19588           
  Branches      225      225           
=======================================
  Hits        17703    17703           
  Misses       1881     1881           
  Partials        4        4           
Flag Coverage Δ
javascript 50.67% <ø> (ø)
python 95.29% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Matches the existing pattern: HISTORY rows track work alongside
the CHANGELOG's [Unreleased] section, with '—' as a placeholder
version that gets replaced during release-prep for the version
that ships this work.

Co-Authored-By: Claude <noreply@anthropic.invalid>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@CONTRIBUTING.md`:
- Line 88: The sentence incorrectly implies all 24 hooks run on every commit;
update the wording after "pre-commit install" to clearly separate which hooks
run on commit vs which run on push by stating something like: X commit-time
hooks run on each commit and Y pre-push hooks run only when pushing (explicitly
call out the changelog/pre-push hook), and adjust the subsequent guideline about
"**/*.md" to remain as-is for Markdown file review scope; locate and edit the
line containing "pre-commit install" / "pre-commit install --hook-type pre-push"
and the mention of "changelog hook" to make this distinction explicit.

In `@scripts/check_changelog_updated.py`:
- Around line 139-151: The current is_substantive uses any(path == p or
path.startswith(p) for p in EXEMPT_PREFIXES) which overmatches file-like
exemptions (e.g. "README.md" also matches "README.md.bak"); change the check so
that EXEMPT_PREFIXES entries that represent directories (e.g. end with '/' or
os.sep, or explicitly contain a trailing slash) are matched with
path.startswith(prefix) but entries that represent files are matched with exact
equality (path == file_name). Update is_substantive to normalize path and then
iterate EXEMPT_PREFIXES, using startswith only for directory-style entries and
exact match for file-style entries to avoid accidental overmatching.
- Around line 153-172: _changelog_has_new_entry currently treats any added "- "
line as a valid new entry; tighten it so added bullets only count if they appear
inside the Unreleased section (or if the diff itself adds a new version
heading). Modify _changelog_has_new_entry to iterate diff lines while tracking
the current section header: whenever a line (context " " or added "+") has a
heading starting with "## [", parse the bracketed name (e.g., "Unreleased" or a
version) and set current_section accordingly; then only treat an added "+ - "
bullet as a new entry if current_section == "Unreleased". Additionally, keep the
existing behavior that an added "## [" line itself (i.e., an added version
heading) immediately returns True. Update the function _changelog_has_new_entry
to implement this section-tracking logic.

In `@TESTING.md`:
- Line 377: Rephrase the sentence that currently states "After running
`pre-commit install`, every commit is checked by 24 hooks:" to avoid implying
all 24 hooks run on every commit—clarify that 24 hooks are configured overall
but some (e.g., `check-changelog-updated`) run at pre-push rather than
per-commit; also update the pre-commit section to include the coding guideline
line exactly as given: "`**/*.md`: Review Markdown files for factual accuracy
against the codebase, broken links, and outdated information." Ensure you update
the line in TESTING.md where the original sentence and hook count appear and
keep the hook count (24) but separate per-commit vs pre-push behavior in the
wording.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 3991da71-3b7e-4d89-8df0-a657d98bd929

📥 Commits

Reviewing files that changed from the base of the PR and between 56f14ad and 45ba777.

📒 Files selected for processing (8)
  • .github/workflows/ci.yml
  • .pre-commit-config.yaml
  • CHANGELOG.md
  • CONTRIBUTING.md
  • ROADMAP.md
  • TESTING.md
  • scripts/check_changelog_updated.py
  • tests/test_check_changelog_updated.py

Comment thread CONTRIBUTING.md Outdated
Comment thread scripts/check_changelog_updated.py
Comment thread scripts/check_changelog_updated.py
Comment thread TESTING.md Outdated
Three legitimate correctness findings from CodeRabbit on #479:

1. is_substantive overmatched file-style exemptions.  With
   EXEMPT_PREFIXES containing entries like 'README.md' (no trailing
   slash), 'path.startswith("README.md")' returned True for
   'README.md.bak' — silently exempting files that aren't actually
   README.md.  Split matching by shape: directory-style entries
   (trailing '/') use startswith; file-style entries use exact
   equality.  Added _is_exempt helper.

2. _changelog_has_new_entry counted any added '+- ' line as a new
   entry, including bullets inserted inside released-version entries
   (e.g. a prose fix on the v0.0.111 description would pass the
   check).  Rewrote to walk the diff while tracking the current
   section header (from both context lines and added lines); added
   bullets now only count when current_section == 'Unreleased'.
   Added version headings (+## [X.Y.Z]) still immediately count.
   Fixed an off-by-one in section-name parsing along the way
   (content[3:end] included the opening '[').

3. CONTRIBUTING.md and TESTING.md wording implied all 24 hooks run
   on every commit.  Rewrote to clarify: 23 at commit time, 1
   (check-changelog-updated) at push time.  Kept the 'checked by
   24 hooks' phrase so check_doc_counts.py still matches.

Tests: +9 regression cases (file-style overmatch for .bak/.orig/etc,
bullet outside [Unreleased], bullet inside [Unreleased], section
tracking across context lines, directory-style nested paths).  63
tests in test_check_changelog_updated.py (was 54), 3,316 project
total (was 3,307).

Co-Authored-By: Claude <noreply@anthropic.invalid>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@CONTRIBUTING.md`:
- Around line 110-111: Update the sentence describing the pre-push hook so it
reflects the actual classifier behaviour: instead of saying only changes under
`vera/`, `spec/`, and `SKILL.md` trigger the check, state that the hook flags
any change to a non-exempt top-level path as substantive and requires adding an
entry to `CHANGELOG.md`; keep the note that the same check runs in CI and
mention the current exemption mechanism (e.g., maintained patterns like
`**/*.md` or other exempt paths) so contributors aren’t surprised by failures on
new or unknown top-level files.

In `@scripts/check_changelog_updated.py`:
- Around line 177-180: The current heading-check logic in
scripts/check_changelog_updated.py treats any added "## [...]" as valid
including "## [Unreleased]"; change the check so that an added heading is only
accepted if it is not exactly "Unreleased" OR if it is "Unreleased" then there
is at least one added bullet line inside that Unreleased section. Locate the
code that inspects added_lines and the regex that detects new headings (the
variable/logic that sets has_new_version_heading or matches "## \\[(.*?)\\]")
and update it to reject a sole "+## [Unreleased]" by verifying either
match.group(1) != "Unreleased" or, when it is "Unreleased", scanning the
subsequent added_lines for a "+- " (or other bullet marker) within the
Unreleased block before returning success.

In `@tests/test_check_changelog_updated.py`:
- Around line 177-192: Add a regression test alongside
test_detects_new_version_heading that constructs a diff where the only change is
an added "## [Unreleased]" heading with no bullets (e.g. a line starting with
"+## [Unreleased]" and no subsequent "+" bullet lines), use
monkeypatch.setattr(_mod, "_run", lambda cmd: diff) like the existing tests, and
assert that _mod._changelog_has_new_entry("origin/main") returns False to ensure
a heading-only addition does not count as a new entry.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: bcdcd595-c074-4608-9ef5-1d498180bac9

📥 Commits

Reviewing files that changed from the base of the PR and between 73097d3 and 2c6dadf.

📒 Files selected for processing (5)
  • CONTRIBUTING.md
  • ROADMAP.md
  • TESTING.md
  • scripts/check_changelog_updated.py
  • tests/test_check_changelog_updated.py

Comment thread CONTRIBUTING.md Outdated
Comment thread scripts/check_changelog_updated.py
Comment thread tests/test_check_changelog_updated.py
Three findings, all valid:

1. scripts/check_changelog_updated.py was short-circuiting on *any*
   added '## [...]' heading, including a bare '+## [Unreleased]'
   that represents structural scaffolding rather than actual
   content.  Changed the heading branch so it only short-circuits
   on non-'Unreleased' version headings; a bare '+## [Unreleased]'
   now falls through to the bullet-under-section check, which
   will return True only if there's a subsequent '+- ' bullet.

2. Added two regression tests matching the new semantics:
   - test_bare_unreleased_heading_alone_does_not_count
     (asserts '+## [Unreleased]' alone returns False)
   - test_added_unreleased_heading_with_bullet_counts
     (asserts '+## [Unreleased]' + '+- bullet' returns True)

3. CONTRIBUTING.md prose said only 'vera/, spec/, SKILL.md'
   trigger the check, but the classifier is actually exempt-list-
   based (any non-exempt top-level path is substantive).  Rewrote
   the 'Pre-push hook: CHANGELOG enforcement' subsection to list
   the exempt categories explicitly and note the conservative
   default for new top-level folders, so contributors aren't
   surprised by failures on files like 'stdlib/' or 'runtime/'.

Test count: 3,316 -> 3,318 (2 new regression tests).

Co-Authored-By: Claude <noreply@anthropic.invalid>
@aallan aallan merged commit ab1ec10 into main Apr 16, 2026
19 checks passed
@aallan aallan deleted the check-changelog-updated branch April 16, 2026 21:54
@coderabbitai coderabbitai Bot mentioned this pull request Apr 16, 2026
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add check_changelog_updated.py: fail PR if substantive changes lack CHANGELOG entry

1 participant