MkDocs preview: https://d08e8095.dd-docs-preview.pages.dev

Fern preview: https://nvidia-preview-pr-656.docs.buildwithfern.com/nemo/datadesigner

Fern previews include the docs-website version archive with PR changes synced into latest. Notebook tutorials are rendered without execution outputs in previews.

Greptile Summary

This PR fixes the Fern versioning regression from PR #608 by extracting v0.6.0-specific plugin and dev-note pages from the shared v0.5.8 tree into their own frozen copies, restoring older pages to their experimental-warning wording, and wiring up CI so the release nav snapshot is prepared and validated automatically on GitHub release events.

• Adds a frozen fern/versions/v0.6.0.yml nav with v0.6.0-specific plugin pages, and updates latest.yml to point at those versioned copies; the v0.5.8 plugin pages are restored to experimental-warning text with a cross-version link.
• Extends fern-release-version.py with check_latest_display_name, check_as_of_versions, and check_latest_matches_release checks, and a new update_latest_nav helper that rewires latest.yml when prepare copies latest/pages/ to a release slug.
• Updates the build-fern-docs.yml workflow to run prepare --force + check --require-latest-matches-release on release events and opens a follow-up PR back to main via a new sync-release-version-to-main job; build-notebooks.yml is extended to accept a checkout_ref input so it builds against the same release ref.

Important Files Changed

_{Reviews (4): Last reviewed commit: "ci: automate Fern release sync" | Re-trigger Greptile}

PR #656 Review — docs: fix Fern versioned publishing

Summary

This PR pivots Fern publishing to a Dynamo-style branch model. main keeps only the latest authoring tree (fern/versions/latest{.yml,/pages/}), while a CI-managed docs-website branch owns historical version snapshots (v0.5.8, v0.5.9, v0.6.0, older). It replaces the previous "shared v0.5.8/pages/ tree reused by every release nav" model that broke after #608. Net change is +815/-1518, but most deletions are historical versioned MDX trees being moved off main.

The substantive logic lives in:

• fern/scripts/fern-published-branch.py (new, 288 lines): sync-source and patch-devnotes subcommands.
• fern/scripts/fern-release-version.py: extended validation (latest display name, "As of vX.Y.Z" stale-content guard, latest-matches-release).
• .github/workflows/build-fern-docs.yml: split into resolve-release → prepare-published-release (new) → build-notebooks → publish, with publish reading from docs-website.
• .github/workflows/publish-fern-devnotes.yml: now patches Dev Notes onto docs-website and pushes before publishing.
• .github/workflows/docs-preview.yml: previews are built from a docs-website snapshot with the PR source synced over the latest tree, so previews show the version picker reviewers will see in production.
• Two new __init__.py files under data_designer.config.utils and data_designer.config.analysis.utils so py2fern traverses them for the API reference.

Findings

Correctness / risk

• build-fern-docs.yml job graph (lines 240–402): publish runs only when resolve-release and build-notebooks succeed AND prepare-published-release is success or skipped. The if: always() && ... guard is necessary because prepare-published-release is conditional on prepare_release_docs == '1'. Correctly handles the manual-republish path (no new snapshot, just re-publish from docs-website).
• gh release list swap (line 279): replaces gh release view with gh release list --exclude-drafts --exclude-pre-releases --limit 1. Stricter and more explicit. Note this changes default behavior on prerelease tags; if a future "publish prerelease" path is wanted, it will need a separate flag.
• Script working dirs in prepare-published-release: the workflow checks out workflow from default_branch, source from source_ref, and website from docs-website. Invoking workflow/fern/scripts/... ensures the current publish logic runs even when republishing an old release_tag. Subtle and important — worth a brief comment in the workflow but not a blocker.
• fern-published-branch.py:sync_source (lines 1078–1097): clears published_root (except .git), copies source_root over, then merges back preserved version dirs from the prior docs-website snapshot, and finally restores the versions: block in docs.yml. Two concerns:
  1. Only the versions: block of docs.yml is preserved from the published branch; everything else (redirects, experimental, libraries, etc.) is replaced from main. That is the intent (single source for non-version config), but if a redirect targeting an archived slug is removed on main while the slug remains preserved on docs-website, the redirect will be silently lost. validate_redirect_targets only catches missing slugs vs redirects, not missing redirects vs slugs. Worth a follow-up symmetry check.
  2. clear_published_tree removes everything except .git. If docs-website ever holds a hand-curated artifact that isn't in source_root and isn't under fern/versions, it will be wiped each release. AGENTS.md/README forbid hand edits, so this is fine, but the foot-gun should be documented in the script's module docstring, not just in contributor docs.
• extract_devnotes_block / replace_devnotes_block (lines 1100–1147): rely on hard-coded indentation (^ - section:\s+Dev Notes\s*$) and end-of-block detected by the next line that starts with - and is non-empty. Will break if anyone reformats latest.yml so a - section: ... sibling doesn't appear at exactly that indent. Low likelihood, and the script fails loudly with Dev Notes section not found rather than silently — acceptable.
• AS_OF_VERSION_RE (line 1204): r"As of Data Designer\s+\[?v?(...)" — the optional [ is not paired with a closing ]. Pattern matches As of Data Designer v0.5.10 and As of Data Designer [v0.5.10. Markdown link forms [v0.5.10](...) still match because ] is outside the captured group. Correct, but documenting the supported phrasings in the script would help future authors.
• version_key (line 1213): int(not suffix) makes 1.0.0 rank above 1.0.0-rc1, which is correct for SemVer prerelease ordering. ✅
• update_latest_nav (line 1326): rewrites ./latest/pages/ → ./{slug}/pages/ inside latest.yml via plain str.replace. If a path component containing latest/pages/ as a prefix ever exists (e.g., ./latest/pages_archive/...), it would be mangled. Today no such path exists; a guard checking for the leading nav-path segment would be safer.
• build-notebooks.yml checkout (line 433): when called from build-fern-docs.yml with prepare_release_docs != '1', the parent passes release_tag (not source_ref) as checkout_ref. That means the no-snapshot rerun path builds notebooks from the release tag, which may differ from workflow_ref. Appears intentional (republish should match the tag's notebook content) but worth confirming.
• docs-preview.yml (lines 472–493): mv published-fern-preview "$FERN_PREVIEW_ROOT" then sync-source over it. sync-source calls clear_published_tree, which preserves only .git. After the move, the .git from the docs-website checkout lives at $FERN_PREVIEW_ROOT/.git, so clear_published_tree keeps it. Behavior is correct but depends on mv preserving .git — which it does. The PR text could note this dependency.
• Concurrency: both build-fern-docs.yml and publish-fern-devnotes.yml use group: fern-docs-website, cancel-in-progress: false. Good — serializes pushes to docs-website so the non-force pushes can't race.

Minor / style

• SKIP_NAMES in fern-published-branch.py (line 929) excludes dist, site, caches, .venv, .git. Consider adding node_modules and code-reference (gitignored py2fern output), though the latter is regenerated downstream and shouldn't be present in a clean checkout.
• The new __init__.py files (config/utils/, config/analysis/utils/) contain only SPDX headers. The AGENTS.md namespace constraint is "no top-level __init__.py"; these are nested inside a single owning package, so they do not break the PEP 420 merge across config/engine/interface. ✅
• VERSIONING.md and fern/README.md give a one-time backfill instruction ("rerun Build Fern docs with release_tag=v0.6.0 and source_ref=main"). Once executed, that guidance becomes stale — flag for cleanup post-merge.
• DCO sign-off missing (PR checklist already calls this out). NV org policy requires signoff before merge.

Tests

• No unit tests for fern-published-branch.py or the new validation paths in fern-release-version.py. The PR validates via local simulations (per the test plan). For docs/CI tooling this is the established convention — flagging only because the new script has nontrivial branching (merge_preserved_versions, extract_devnotes_block, regex-based YAML mutation) where a silent regression would corrupt the published archive. A handful of pytest cases over fixture trees would harden this. Worth a follow-up issue.

Security

• Workflow concurrency + non-force git push keeps the audit log on docs-website linear and clean.
• permissions: contents: write on prepare-published-release and the devnotes deploy job is the minimum needed to push. ✅
• FERN_TOKEN exposure unchanged (already used by previous workflow).

Verdict

Approve with minor follow-ups. The architecture switch is sound, removes an entire class of release-time PRs, and is clearly documented in VERSIONING.md, fern/README.md, fern/AGENTS.md, and the SKILL doc. The prepare → check → commit → publish ordering and concurrency guards are correct. Main risks: (1) silent-overwrite behavior of sync_source / clear_published_tree on hand edits to docs-website, (2) regex-based YAML mutation in fern-published-branch.py, (3) absence of unit tests for the new scripts. None block this PR — all warrant follow-up issues.

Suggested follow-ups (non-blocking):

1. Add unit tests over fixture trees for fern-published-branch.py (sync-source, patch-devnotes, validate_redirect_targets) and the new check_release options.
2. Symmetry check in validate_redirect_targets so a slug present on docs-website but missing a redirect on main is also flagged.
3. Once v0.6.0 is republished via the manual workflow dispatch, remove the one-time backfill instructions from VERSIONING.md and fern/README.md.
4. Sign off the commits (DCO) before merging.

The v0.6.0 release today hit a Fern publish failure, and while debugging that I realized the Fern setup had drifted pretty far from the MkDocs model we were used to.

The main thing this PR does is make Fern behave much more like MkDocs did:

• main has the docs we actively edit.
• Published docs are built into a published-docs branch, like gh-pages before.
• Each release freezes the current docs into vX.Y.Z.
• latest points at the latest released docs, with Dev Notes patchable on top.
• Older frozen versions stay frozen instead of being accidentally changed by normal docs edits.

Along the way this also fixes a few things that surfaced during the release:

• PR previews now use the same versioned archive shape as publish.
• Release publishing snapshots latest into the release version before publishing.
• Dev Notes publishing updates latest without touching frozen versions.
• The generated API nav is flatter and has fewer duplicate click-through labels.
• Fern CLI is bumped from 4.106.0 to 5.24.0.

Validation is looking good:

• All PR checks are green, including hosted docs preview.
• I ran a real Build Fern docs smoke publish from this PR branch with release_tag=v0.6.0, output is here.
• That passed end to end: notebooks, published-docs branch update, Fern check, and fern generate --docs.
• Fern published 329 pages with 0 errors. The only warning I can reproduce now is the existing light-mode contrast warning.
• Publish points at https://docs.nvidia.com/nemo/datadesigner, which is expected as we are moving there soon.

@johnnygreco @nabinchha when you get a chance, could you take a look and sanity-check whether this feels like the right Fern equivalent of our old MkDocs release flow?

Thanks for putting this together, @andreatgretel — moving Fern publishing onto an orphan docs-website branch is a clean way out of the PR #608 regression, and the manual production smoke from this branch is reassuring.

Summary

The PR migrates Fern release publishing to a Dynamo-style orphan docs-website branch: main keeps only the latest authoring docs (fern/versions/latest{.yml,/pages/...}), while docs-website owns published version snapshots, generated frozen page copies, and publish-provenance metadata. New Python scripts (fern-published-branch.py, fern-release-version.py, normalize-py2fern-indexes.py) drive the sync, validation, and Dev Notes patching from CI, and the workflows (build-fern-docs.yml, publish-fern-devnotes.yml, docs-preview.yml) are reshaped to operate against docs-website checkouts. The implementation matches the stated intent: I traced sync → prepare → check → publish for both release and Dev Notes paths, plus the preview path, and the data flow lines up with the described model.

Findings

Warnings — Worth addressing

.agents/skills/datadesigner-docs/SKILL.md:414 — Skill docs describe the old Code Reference nav

• What: The "Python API Reference" section still says the nav wires generated output via "Code Reference > Python API" folder entry (folder: ../code-reference/data-designer) and that the nav "includes prose pages under 'Topic Overviews'". The new fern/versions/latest.yml no longer has either: prose + generated folders are now nested under Config, Interface, and Engine Extension API sections, and make generate-fern-api-reference writes to fern/code-reference/{data-designer,interface,engine/...} (six outputs, not one).
• Why: This skill is the playbook agents read when extending Fern docs. Leaving the structural description out of date in the same PR that does the restructuring is likely to produce wrong nav edits the next time someone adds an API reference page or library entry.
• Suggestion: Update the "Python API Reference (libraries:)" section to describe the new layout (multiple py2fern outputs under fern/code-reference/, folders wired inside the per-package sections, no "Topic Overviews"). The example "add another package as a library entry" snippet around lines 416–432 also reads like the pre-PR model and is worth refreshing. fern/README.md is already updated correctly — could mostly mirror its phrasing.

packages/data-designer-engine/src/data_designer/engine/processing/processors/schema_transform.py:30 — HTML entities in a Python docstring

• What: The docstring for _escape_value_for_json was changed from ({{ col.sub.field }}) to , such as {{ col.sub.field }}, to keep py2fern from emitting a literal {{ ... }} that MDX parses as a JSX expression.
• Why: It fixes the rendered API reference, but anyone reading the source now sees HTML entities in a Python docstring, which is a surprising regression in source readability. The same py2fern-vs-MDX problem will likely come up for other docstrings that mention Jinja syntax, so worth a more reusable answer.
• Suggestion: Two options that keep the source clean:
  • Wrap the example in a backticked code span: use `{{ col.sub.field }}` on deserialized JSON columns — Fern/MDX should leave the contents of an inline code span alone. (Worth a quick make check-fern-docs to confirm.)
  • Or move the curly-brace example into a fenced code block in the docstring, which py2fern typically passes through verbatim.
• If neither works because of py2fern's MDX rendering rules, a comment above the docstring noting why the entities are there would at least save the next reader a git blame.

fern/scripts/fern-published-branch.py, fern/scripts/fern-release-version.py — No automated test coverage

• What: Together these add ~600 lines of Python with non-trivial logic (version-block parsing, nav rewriting, page materialization, Dev Notes patching, redirect-target validation, publish-metadata writing). The PR description lists thorough manual simulations but no tests/ additions.
• Why: AGENTS.md treats "no untested code paths" as a structural invariant, and the failure modes here aren't cheap — a regression in materialize_version_nav_pages, merge_preserved_versions, or validate_redirect_targets would corrupt docs-website and the only signal would be a CI failure (best case) or a published archive that quietly drops a frozen version (worst case). The manual simulations from the PR description are exactly the shape of cases that would slot nicely into pytest.
• Suggestion: A small fern/scripts/tests/test_fern_published_branch.py (and a peer for fern-release-version.py) using tmp_path would give us regression coverage cheaply. The simulations the PR already ran translate well — sync-source preservation, missing-archive guard, Dev Notes patch, frozen-version materialization, and check_as_of_versions. If wiring them into make test is heavyweight, a dedicated make test-fern-scripts target would still be a meaningful safety net.

Suggestions — Take it or leave it

fern/scripts/fern-release-version.py — Local prepare is single-use per checkout

• What: update_latest_nav rewrites every ./latest/pages/ reference in latest.yml to ./{slug}/pages/. After running make prepare-fern-release VERSION=0.6.1, latest.yml no longer mentions ./latest/pages/, so a follow-up make prepare-fern-release VERSION=0.6.2 FORCE=1 finds nothing to replace in write_release_nav, and the new v0.6.2.yml ends up referencing ./0.6.1/pages/... files instead of getting its own copy.
• Why: Not a CI concern (each release run starts from a fresh main checkout), but it's a foot-gun for anyone iterating locally on the release scripts or recovering from a botched prep.
• Suggestion: Either teach write_release_nav to also detect already-rewritten ./vX.Y.Z/pages/ references in latest.yml and re-point them at ./{slug}/pages/, or document the "re-run from a clean checkout" expectation near the prepare-fern-release make target.

fern/scripts/fern-published-branch.py:215 — Frozen-nav comment is briefly inaccurate

• What: write_release_nav writes the comment # Frozen {slug} release nav. Reuses shared pages until content needs to diverge., but immediately copies the pages into {slug}/pages/. The CI pipeline then runs materialize_version_nav_pages, which rewrites the same comment to # Frozen {slug} release nav. Pages are materialized under ./{slug}/pages/..
• Why: It's self-correcting in CI, but locally (after make prepare-fern-release) the comment is misleading on disk.
• Suggestion: Just write the materialized comment from the start in write_release_nav and drop the materialize_version_nav_pages comment-rewrite branch. Keeps a single source of truth for the explanation.

fern/scripts/fern-published-branch.py:215 — __pycache__ could live in SKIP_NAMES

• What: ignore_source checks name in SKIP_NAMES or name == "__pycache__".
• Why: Symmetry — SKIP_NAMES already holds .git, .venv, .ruff_cache, etc. Adding __pycache__ to the set lets the function be a one-liner.
• Suggestion: SKIP_NAMES = {..., "__pycache__"} and return {n for n in names if n in SKIP_NAMES}.

fern/scripts/*.py — Function names that don't lead with a verb

• What: STYLEGUIDE.md asks function names to start with an action verb (get_value_from, not value_from). New helpers like versions_block, versions_block_text, version_slugs, version_entries, referenced_mdx_paths, code_reference_page_root, version_key, version_slug read like nouns.
• Why: Mostly a style consistency thing — most other helpers in these files (extract_*, validate_*, normalize_*, materialize_*, etc.) already follow the rule, so the noun-shaped ones stand out.
• Suggestion: Optional rename pass — get_versions_block, get_version_slugs, find_referenced_mdx_paths, get_code_reference_page_root, compute_version_key, etc. Pure scripts may legitimately be exempt from the project rule, so feel free to leave as-is if you'd prefer.

fern/scripts/ — Duplicated find_top_level_block and YAML helpers

• What: find_top_level_block (and the pattern of slicing top-level YAML blocks) lives identically in both fern-published-branch.py and fern-release-version.py.
• Why: They'll drift the next time someone fixes a parsing edge case in one and forgets the other.
• Suggestion: Once a third caller appears (per STYLEGUIDE's KISS-over-DRY guidance), extract into a fern/scripts/_yaml_block.py helper — until then, fine to leave.

packages/data-designer-config/src/data_designer/config/{utils,analysis/utils}/__init__.py — Missing from __future__ import annotations

• What: AGENTS.md lists from __future__ import annotations as a structural invariant for every Python source file. The two new __init__.py files added here only carry the SPDX header. The neighboring source __init__.py (e.g. data_designer/config/testing/__init__.py) does include it.
• Why: They're empty today, but anything added later (lazy re-exports, package-level type hints) would silently get eager evaluation without it.
• Suggestion: Add the future import to match the source-package pattern, or leave a brief """Marker for py2fern package discovery.""" docstring + from __future__ import annotations to make the intent obvious.

What Looks Good

• The split between authoring on main and publishing on docs-website is the right call, and the orphan-branch model with fern/publish-metadata.json provenance is a much cleaner story than the previous release-time PR cleanup. The doc updates in fern/AGENTS.md, fern/README.md, CONTRIBUTING.md, and VERSIONING.md make the new contract easy to follow.
• The workflow security posture is thoughtful — release-triggered runs check out workflow scripts from default_branch while workflow_dispatch honors the dispatcher's ref. The concurrency: group: fern-docs-website shared between build-fern-docs.yml and publish-fern-devnotes.yml cleanly serializes Dev Notes patches behind release publishes.
• validate_redirect_targets is a great early-failure guard. Coupled with the redirect-version regex and the friendly error message, this turns a class of "deploy succeeds, users hit dead links" bugs into a CI failure.
• The defensive fallback in sync_code_reference_pages (target flat → target nested → source nested) is the kind of detail that pays off when older release navs have slightly different layouts. The comments around the historical archive in fern/docs.yml redirects are also unusually well written.
• All ruff and format checks pass on the new Python (make check-all clean for the changed files).

Verdict

Ship it (with nits) — nothing blocking. The skill-doc drift in .agents/skills/datadesigner-docs/SKILL.md is the most worth-doing item before merge so the next agent doesn't re-introduce a "Topic Overviews" section. The schema-transform docstring and the missing tests are good targets for follow-up work; both fine to stage after the merge given the production smoke already passed.

This review was generated by an AI assistant.

Thanks @nabinchha, this was helpful. I pushed adb4661a addressing the nits that felt worth folding into this PR:

• refreshed .agents/skills/datadesigner-docs/SKILL.md so it matches the new Code Reference layout and no longer points agents at Topic Overviews
• replaced the HTML entities in the schema transform docstring with an inline code span for {{ col.sub.field }}
• fixed the local repeated-prepare footgun so preparing 0.6.1 and then 0.6.2 rewrites/copies pages to the new slug instead of leaving refs to the prior version
• wrote the materialized release-nav comment from the start
• moved __pycache__ into SKIP_NAMES
• added from __future__ import annotations to the two new package marker files

I left the pytest coverage / helper rename / YAML helper extraction suggestions as follow-ups so this PR does not grow more than it needs to.

Validation passed locally:

• repeated local prepare test
• fern-release-version.py check --require-latest-matches-release
• make check-fern-docs
• .venv/bin/ruff check --fix .
• .venv/bin/ruff format .

CI is good now.