feat: plumb prerelease metadata through the TOML pipeline by jakedgy · Pull Request #135 · jdx/mise-versions

jakedgy · 2026-04-26T03:20:58Z

Summary

Companion to jdx/mise#9329, which adds an opt-in `prerelease` tool option for the `github` and `aqua` backends. That PR plumbs the upstream pre-release flag into mise's `ls-remote --json` output and reads it back from the versions-host TOML; this PR is the producer side that makes the flag actually appear in the rendered TOML.

Changes

`scripts/generate-toml.js` — reads `prerelease` from the NDJSON stdin and emits `prerelease = true` in the inline table when set. False/missing values are omitted to keep the TOML compact.
`scripts/sync-versions-to-d1.js` — forwards the field to the sync API.
`web/src/pages/api/admin/versions/sync.ts` — includes `prerelease` in the upsert SQL.
`src/analytics/migrations.ts` — idempotent migration: `versions.prerelease INTEGER NOT NULL DEFAULT 0`. Same pattern as the existing `from_mise` / `sort_order` migrations a few lines up.
`scripts/generate-toml.test.js` — round-trip tests for emit, omit, preserve-from-existing, and API-override semantics.

Design note: definitive vs. unknown

The merge logic distinguishes "API explicitly said false" (definitive — overrides any prior state) from "this fetch fell back to the plain-text path" (no info — preserve whatever the existing TOML said) by treating absent as `null` rather than `false`. This matches the existing fallback behavior for `created_at` / `release_url` and means a prior run's flag survives a transient JSON-fetch failure. Tests cover both paths.

Compatibility

Direction	Behavior
Old mise clients reading new TOML	`toml-rs` ignores unknown fields by default → see exactly today's `created_at` + `release_url`
New mise clients reading old TOML	`#[serde(default)]` on the field → absence parses as `false` (stable)
This pipeline reading old `mise ls-remote --json` output	`obj.prerelease` is `undefined` → treated as unknown, existing TOML wins → no regression while the Docker image is on a pre-mise#9329 build

Test plan

`bun run test` — 37/37 pass (24 JS + 13 shell)
On the first run after deploy, watch `updated_tools.txt` and confirm a github/aqua tool with known prereleases (e.g. tools that already get prereleases via aqua's `--security` flag, or post-mise#9329 deploy) renders `prerelease = true` in its TOML
D1 migration runs idempotently (it's wrapped in a column-existence check)

Sequencing

This PR is safe to merge in either order with jdx/mise#9329. The feature only fully activates once both have shipped and the `jdxcode/mise` Docker image rebuilds with the new mise version. Until then, the producer side reads `undefined` for `prerelease` from `ls-remote --json` and the rendered TOML simply doesn't carry the field — same as today.

This pull request was prepared by an AI coding assistant.

Companion to jdx/mise#9329, which adds an opt-in `prerelease` tool option for the github + aqua backends. That PR plumbs the upstream pre-release flag into mise's `ls-remote --json` output and reads it back from the versions-host TOML; this PR is the producer side that makes the flag actually appear in the rendered TOML. - generate-toml.js reads `prerelease` from the NDJSON stdin and emits `prerelease = true` in the inline table when set. False/missing values are omitted to keep the TOML compact. - An "unknown" sentinel (null) distinguishes "API explicitly said false" (definitive — overrides any prior state) from "this fetch fell back to the plain-text path" (preserve whatever the existing TOML said). This matches the existing fallback behavior for `created_at` / `release_url`. - sync-versions-to-d1.js forwards the field to the API. - /api/admin/versions/sync includes `prerelease` in the upsert. - Adds an idempotent D1 migration: `versions.prerelease INTEGER NOT NULL DEFAULT 0`. Same pattern as the existing `from_mise` / `sort_order` migrations a few lines up. - Round-trip tests cover input → TOML, omission, preservation, and override semantics. All 24 generate-toml tests pass. Forward + backward compatibility: - Old mise clients reading new TOML: toml-rs ignores unknown fields by default, so `prerelease = true` is silently dropped — they see exactly the same `created_at` + `release_url` as today. - New mise clients reading old TOML: the field defaults to false on deserialization (`#[serde(default)]`), so absence parses as stable. - This pipeline reading old `mise ls-remote --json` output (from the Docker image before mise#9329 ships): `obj.prerelease` is `undefined`, treated as "unknown", existing TOML wins. No regressions.

greptile-apps · 2026-04-26T03:24:19Z

Greptile Summary

This PR plumbs the upstream prerelease flag from mise ls-remote --json through the full TOML-generation and D1-sync pipeline. The null-vs-boolean distinction for "unknown" vs "definitive" is well-designed and correctly handled at every stage, and tests cover all four meaningful branches (emit, omit, preserve-from-existing, API override).

Confidence Score: 5/5

Safe to merge — all changed paths are correct, the migration is idempotent, and the COALESCE trade-off was consciously resolved in the prior review thread.

No P0 or P1 findings. The null/boolean merge logic in generate-toml.js is sound, the migration follows the established pattern, and the test suite covers all four semantic cases.

No files require special attention.

Important Files Changed

Filename	Overview
scripts/generate-toml.js	Adds prerelease field parsing from NDJSON (null = unknown, boolean = definitive) and correct fallback-to-existing TOML merge logic; output correctly omits the field when false/unknown.
scripts/generate-toml.test.js	Adds four tests covering emit, omit, preserve-from-existing-TOML, and API-override semantics; all four meaningful branches are covered.
scripts/sync-versions-to-d1.js	Reads prerelease from TOML (only ever true or absent) and forwards as explicit boolean to the sync API; the === true guard is correct for this shape of data.
src/analytics/migrations.ts	Idempotent column-existence guard adds prerelease INTEGER NOT NULL DEFAULT 0 to the versions table, matching the existing from_mise / sort_order migration pattern.
web/src/pages/api/admin/versions/sync.ts	Adds prerelease to INSERT column list and binds it as 0/1 integer; direct assignment in the UPDATE SET is intentional (TOML is authoritative) as discussed and resolved in prior review thread.

Sequence Diagram

sequenceDiagram
    participant LS as mise ls-remote --json
    participant GT as generate-toml.js
    participant ET as existing .toml
    participant TOML as output .toml
    participant Sync as sync-versions-to-d1.js
    participant API as sync.ts (POST /api/admin/versions/sync)
    participant D1 as D1 versions table

    LS->>GT: NDJSON lines {version, prerelease: true|false|undefined}
    ET->>GT: existing prerelease state (true|false)
    Note over GT: prerelease logic:<br/>1. API boolean → use it (definitive)<br/>2. null (no JSON) → keep existing TOML flag
    GT->>TOML: prerelease = true (omitted when false/unknown)

    Sync->>TOML: read parsed TOML
    Note over Sync: data?.prerelease === true<br/>→ always sends explicit boolean
    Sync->>API: POST {versions: [{prerelease: true|false}]}

    Note over API: runAnalyticsMigrations()<br/>adds column if absent
    API->>D1: INSERT … prerelease = ?<br/>ON CONFLICT DO UPDATE SET prerelease = excluded.prerelease

_{Reviews (2): Last reviewed commit: "docs: correct generate-toml.js header da..." | Re-trigger Greptile}

greptile-apps · 2026-04-26T03:24:23Z

          ON CONFLICT(tool_id, version) DO UPDATE SET
            created_at = COALESCE(excluded.created_at, versions.created_at),
            release_url = COALESCE(excluded.release_url, versions.release_url),
+            prerelease = excluded.prerelease,


Asymmetric COALESCE pattern for prerelease

created_at and release_url both use COALESCE(excluded.value, versions.value) to preserve the existing D1 value when the incoming data has no meaningful new value. prerelease uses direct assignment and will always overwrite. This is safe as long as every sync is driven by a TOML produced by generate-toml.js (which correctly applies the fallback logic), but D1 becomes the only downstream store that loses the null-vs-false distinction — a re-sync from a TOML regenerated without an existing file would silently reset any previously recorded prerelease = 1 rows to 0. Using COALESCE here would add the same resilience the other two fields already have.

Suggested change

prerelease = excluded.prerelease,

prerelease = COALESCE(excluded.prerelease, versions.prerelease),

Considered this but keeping the direct assignment. The release_url consistency argument is appealing, but the two fields aren't symmetric:

release_url: D1 is the only place the URL persists across runs. The producer can't always fetch it (rate limits, plain-text fallback), and TOML may omit it. COALESCE in D1 is the resilience layer.

prerelease: the producer (scripts/generate-toml.js:158-159) already implements the same null-vs-definitive fallback against the existing TOML. So TOML is already authoritative for the latest known state. D1 mirrors TOML; it's not a second source of truth.

Adding COALESCE here would introduce a divergence on the demotion path: if upstream un-checks a GH release's prerelease flag, the producer correctly drops prerelease = true from TOML, but D1 with COALESCE retains the stale 1 indefinitely. Two layers, two answers.

The "TOML clobbered by regen without existing path" scenario is real but operationally recoverable — re-run the pipeline once and TOML+D1 realign. The COALESCE-introduced divergence is permanent until the row is deleted.

Mechanically: versions.prerelease is INTEGER NOT NULL DEFAULT 0. Making COALESCE(NULL, ...) work would require dropping NOT NULL (introducing a tristate that consumers would have to handle as WHERE COALESCE(prerelease, 0) = 0), or wrapping the bind in COALESCE(?, 0) which defeats the COALESCE on update. Neither pays off given the producer already does the work.

Copilot

Pull request overview

This PR adds support for carrying an upstream “prerelease” signal end-to-end through the versions TOML generation pipeline and into the analytics D1 database, enabling downstream consumers (e.g. mise) to distinguish prerelease versions.

Changes:

Emit prerelease = true in generated TOML entries when the NDJSON input explicitly marks a version as prerelease, and preserve prior TOML state when the input lacks prerelease info.
Forward the prerelease flag through the versions sync script and API, persisting it in D1 during upserts.
Add an idempotent D1/SQLite migration to introduce versions.prerelease as an integer boolean column, plus tests covering the new TOML behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`scripts/generate-toml.js`	Parses prerelease metadata from NDJSON and conditionally renders `prerelease = true` in inline TOML tables; merges with existing TOML.
`scripts/generate-toml.test.js`	Adds tests for emitting/omitting/preserving/overriding prerelease behavior.
`scripts/sync-versions-to-d1.js`	Includes prerelease when posting parsed TOML versions to the sync API.
`web/src/pages/api/admin/versions/sync.ts`	Extends upsert to write prerelease into the `versions` table.
`src/analytics/migrations.ts`	Adds `prerelease INTEGER NOT NULL DEFAULT 0` to the `versions` table if missing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-26T03:24:38Z

@@ -152,6 +154,7 @@ export const POST: APIRoute = async ({ request, locals }) => {
            v.version,
            v.created_at || null,
            v.release_url || null,
+            v.prerelease === true ? 1 : 0,
            v.sort_order ?? null,


prerelease is optional in VersionData, but the handler currently treats undefined the same as false (binds 0) and then unconditionally overwrites the DB value (prerelease = excluded.prerelease). This makes the endpoint non-backwards-compatible with older sync clients that don't send the field (they would clear any previously recorded prerelease=1). Consider binding NULL when the field is absent and using a COALESCE-style update so only explicit booleans (true/false) overwrite the existing value, while inserts still default to 0.

Tied to the COALESCE decision in the adjacent thread.

On the backwards-compat angle specifically: there's exactly one client of this endpoint (scripts/sync-versions-to-d1.js in this repo), and it's updated atomically with the server. So "old client doesn't send the field" isn't a real scenario worth defending against. If a sync ever did clobber a 1 to 0 on a transient mismatch, the next pipeline run re-derives from upstream and writes the correct value — the producer is authoritative.

Keeping the column as INTEGER NOT NULL DEFAULT 0 (clean two-state, no tristate for future query consumers to handle).

gemini-code-assist

Code Review

This pull request introduces a prerelease flag across the versioning pipeline, including TOML generation, database schema migrations, and synchronization scripts. The changes ensure that versions marked as pre-releases are correctly identified and stored. The review feedback highlights a potential data integrity issue in the database sync logic: the prerelease field should use COALESCE during updates and handle null values for missing input to avoid accidentally resetting existing flags to zero.

gemini-code-assist · 2026-04-26T03:27:16Z

          ON CONFLICT(tool_id, version) DO UPDATE SET
            created_at = COALESCE(excluded.created_at, versions.created_at),
            release_url = COALESCE(excluded.release_url, versions.release_url),
+            prerelease = excluded.prerelease,


The prerelease field is updated directly without a COALESCE check, unlike created_at, release_url, and sort_order. This means that if a sync request is made where the prerelease field is missing (e.g., from an older client or a partial update), any existing prerelease flags in the database will be reset to 0. To maintain consistency and prevent accidental data loss, this should use COALESCE to preserve the existing value when the incoming value is null.

Suggested change

prerelease = excluded.prerelease,

prerelease = COALESCE(excluded.prerelease, versions.prerelease),

Same reasoning as the reply on Greptile's parallel comment: keeping direct assignment because the producer (scripts/generate-toml.js) is already the source of truth for prerelease — it implements the same null-vs-definitive fallback there. Adding COALESCE in D1 would create permanent TOML/D1 divergence on the demotion path (when upstream un-checks a GH release's prerelease flag) without buying real resilience the producer doesn't already provide.

gemini-code-assist · 2026-04-26T03:27:16Z

            v.version,
            v.created_at || null,
            v.release_url || null,
+            v.prerelease === true ? 1 : 0,


To support the COALESCE logic in the SQL update, the bound value for prerelease should be null when the field is missing from the input, rather than defaulting to 0. This allows the database to distinguish between an explicit 'false' and an 'unknown' state.

Suggested change

v.prerelease === true ? 1 : 0,

typeof v.prerelease === "boolean" ? (v.prerelease ? 1 : 0) : null,

Tied to the COALESCE decision in the adjacent thread — keeping the bind as boolean → 0/1 so the column stays NOT NULL and queryable as a clean two-state. Sending null only makes sense paired with COALESCE on update, which we're not doing here.

Header docstring showed `created_at = "2024-..."` (quoted), but smol-toml emits TOML offset datetimes unquoted. Update the example to match actual output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…140) ## Summary Sibling to #135. That PR plumbed `prerelease = true` through the TOML pipeline, but on its own the field is dead code: `mise ls-remote --json` filters pre-release tags out before emitting JSON unless the caller has opted in. With ~900 tools, per-tool config isn't viable. [jdx/mise#9415](jdx/mise#9415) adds a global `MISE_PRERELEASES` env var (and a matching `--prerelease` CLI flag) that flips the filter for every tool at once. ## Changes - **`.github/workflows/update.yml`** — set `MISE_PRERELEASES: "1"` at the job env level. Covers the host-side `mise ls-remote --json` call in `generate_toml_file` (the JSON path that actually feeds `generate-toml.js`). - **`scripts/update.sh`** — forward `-e MISE_PRERELEASES` into the Docker container. Covers the plain-text `docker run jdxcode/mise -y ls-remote` call in `fetch()`. ## Verification (after deploy) After the next run, expect to see `prerelease = true` flags on tools whose upstream releases are flagged as pre-releases — e.g. tools with `-rc1` / `-beta` / `-dev.N` GitHub releases. Until then: zero hits, despite #135 being merged. Confirmed against the most recent run ([24968319913](https://github.com/jdx/mise-versions/actions/runs/24968319913)) which touched 646 tools through the JSON path with zero `prerelease = true` flags emitted. ## Sequencing Optimistic — depends on: 1. A mise release containing jdx/mise#9415 in the host mise installed by `aube install` / `mise.run`. (Today's host mise = v2026.4.23, which has #9329 but not #9415.) 2. A rebuild of the `jdxcode/mise` Docker image with the same release for the plain-text fallback path. Until both ship, `MISE_PRERELEASES` is silently ignored — same TOML output as today, no regression. Safe to merge whenever; it activates on its own as soon as the upstream pieces land.  --- > [!NOTE] > **Low Risk** > Low risk: only toggles an environment flag and forwards it into the Dockerized `mise ls-remote` call, changing which versions are included but not altering control flow or security-sensitive logic. > > **Overview** > Enables global inclusion of prerelease tags during the automated versions update run by setting `MISE_PRERELEASES=1` in the `update` GitHub Actions workflow. > > Also forwards `MISE_PRERELEASES` into the `docker run jdxcode/mise ... ls-remote` invocation in `scripts/update.sh`, allowing downstream TOML generation to populate the new `prerelease = true` metadata when upstream marks releases as prereleases. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit b8f30b5. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>  Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…9415) ## Summary Companion to #9329, which added per-tool `prerelease = true` opt-in for the `github:` and `aqua:` backends. That covers the case where a user wants pre-releases for one specific tool. This PR adds the **global** escape hatch — equivalent to setting `prerelease = true` on every tool — for consumers that mirror the full release catalog or want pre-release tags to surface in `ls-remote` without per-tool config. The motivating case: the [versions host pipeline](https://github.com/jdx/mise-versions) calls `mise ls-remote --json` for every tracked tool to populate `docs/*.toml`. Per-tool config there isn't viable (≈900 tools), but without a global opt-in, mise filters every prerelease before emitting JSON, and the versions host can't carry the new `prerelease = true` field that jdx/mise-versions#135 just plumbed through. ## Changes - **New setting `prereleases`** (`MISE_PRERELEASES`) in `settings.toml`. Layered into `include_prereleases()` ahead of the per-tool option, so flipping it acts like `prerelease = true` on every tool. Has no effect on backends that don't carry an upstream prerelease flag (e.g. `github_tag` version source — git tags don't carry the prerelease bit). - **New `--prerelease` flag on `mise ls-remote`**. Calls a small `Settings::override_with` helper that merges into the existing `CLI_SETTINGS` partial — `Settings::reset` replaces wholesale, which would clobber overrides installed earlier in startup like `--offline` / `--quiet`. The merge approach lets a subcommand layer one extra override on top. - **Test** for the global-setting path alongside the existing per-tool test in `backend::tests`. ## Test plan - [x] `cargo test --bin mise backend::` — 193 pass - [x] `cargo test --bin mise config::settings` — 22 pass (incl. `test_settings_toml_is_sorted`) - [x] Smoke: `mise ls-remote --help` shows the new flag, `mise ls-remote --json --prerelease github:cli/cli` runs without error - [ ] CI ## Compatibility - Default behavior unchanged: `prereleases = false`, prereleases stay filtered. All existing tool configs continue to work. - The per-tool `prerelease = true` option is unaffected — both paths feed into the same `include_prereleases()` helper, which now ORs them together.  --- > [!NOTE] > **Medium Risk** > Medium risk because it changes version filtering behavior and settings layering/caching for remote version listings (and `latest`/fuzzy resolution via `include_prereleases`), which could affect tool resolution outputs across backends when enabled. > > **Overview** > Adds a new global `prereleases` setting (`MISE_PRERELEASES`) that, when enabled, makes `include_prereleases()` opt into upstream pre-release versions regardless of per-tool `prerelease` options. > > Extends `mise ls-remote` with `--prerelease`, implemented by merging a CLI-layer settings override (`Settings::override_with`) so the flag can be applied without clobbering other startup overrides. > > Updates generated CLI docs/usage spec accordingly and adds a regression test ensuring the global setting enables prerelease inclusion by default. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit c6f5ffb. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>  --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 26, 2026 03:21

Copilot started reviewing on behalf of jakedgy April 26, 2026 03:21 View session

greptile-apps Bot reviewed Apr 26, 2026

View reviewed changes

Copilot AI reviewed Apr 26, 2026

View reviewed changes

gemini-code-assist Bot reviewed Apr 26, 2026

View reviewed changes

docs: correct generate-toml.js header datetime format

e66a363

Header docstring showed `created_at = "2024-..."` (quoted), but smol-toml emits TOML offset datetimes unquoted. Update the example to match actual output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

jdx merged commit 646556a into jdx:main Apr 26, 2026
5 checks passed

This was referenced Apr 26, 2026

feat(ls-remote): add prereleases setting and --prerelease flag jdx/mise#9415

Merged

ci: enable MISE_PRERELEASES so the new prerelease field populates #140

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: plumb prerelease metadata through the TOML pipeline#135

feat: plumb prerelease metadata through the TOML pipeline#135
jdx merged 2 commits intojdx:mainfrom
jakedgy:feat/prerelease-metadata

jakedgy commented Apr 26, 2026

Uh oh!

greptile-apps Bot commented Apr 26, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Apr 26, 2026

Uh oh!

jakedgy Apr 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Apr 26, 2026

Uh oh!

jakedgy Apr 26, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Apr 26, 2026

Uh oh!

jakedgy Apr 26, 2026

Uh oh!

gemini-code-assist Bot Apr 26, 2026

Uh oh!

jakedgy Apr 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	prerelease = excluded.prerelease,
	prerelease = COALESCE(excluded.prerelease, versions.prerelease),

	v.prerelease === true ? 1 : 0,
	typeof v.prerelease === "boolean" ? (v.prerelease ? 1 : 0) : null,

Conversation

jakedgy commented Apr 26, 2026

Summary

Changes

Design note: definitive vs. unknown

Compatibility

Test plan

Sequencing

Uh oh!

greptile-apps Bot commented Apr 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

greptile-apps Bot Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

jakedgy Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

jakedgy Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

jakedgy Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

jakedgy Apr 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

greptile-apps Bot commented Apr 26, 2026 •

edited

Loading