feat: opt-in to pre-release versions for github and aqua backends#9329
feat: opt-in to pre-release versions for github and aqua backends#9329
Conversation
By default the `github:` backend filters out releases flagged `prerelease: true` from `ls-remote` output and `latest` resolution. This is a sensible default, but rules out projects whose active releases are all pre-releases — e.g. an internal tool that ships continuous dev builds and only occasionally cuts a stable release. Add a per-tool `prerelease = true` opt. When set: - `_list_remote_versions` uses a new `list_releases_including_prereleases` helper that skips the API flag filter (drafts are still dropped). - `latest_stable_version` skips the `/releases/latest` shortcut (which is stable-only by design) and resolves `latest` against the full list. - `fuzzy_match_filter` is overridden to skip the VERSION_REGEX filter, so tags like `1.0.0-rc1` or `0.1.2-dev.86` survive matching. The fuzzy match logic is refactored into a shared `fuzzy_match_versions` helper so the trait default and the github override share one code path. Discussed in jdx#9323.
The aqua backend resolves versions via GitHub releases for most
packages (and git tags for `version_source = github_tag`). Like the
github backend, it filters out releases flagged `prerelease: true` by
default, excluding them from `ls-remote` and `latest` resolution.
Mirror the github backend's opt: when `prerelease = true` is set,
`get_tags{,_with_created_at}` pulls from the prerelease-including
variant, and the `fuzzy_match_filter` override skips the
VERSION_REGEX filter so tags like `1.0.0-rc1` survive matching. Has
no effect for packages using `github_tag` as the version source
(git tags don't carry a prerelease flag).
Discussed in jdx#9323.
There was a problem hiding this comment.
Code Review
This pull request introduces a prerelease tool option for the Aqua and GitHub backends, allowing users to include pre-release versions in ls-remote output and version resolution. The implementation refactors fuzzy matching logic into a shared helper function to support optional pre-release filtering. Review feedback identifies that the include_prereleases helper should be centralized to avoid duplication and updated to correctly handle boolean TOML values. Additionally, the GitHub backend's fuzzy matching should be updated to respect global tool configurations, and the Aqua backend should be refactored to utilize the new shared matching utility.
Greptile SummaryAdds a Confidence Score: 5/5Safe to merge; one P2 documentation inaccuracy for the github_tag source edge case, but no logic bugs in the core code paths. All previously raised P1 issues (stale cache on upgrade, inline-opts-only in get_version_tags, include_prereleases duplication) are addressed in the current HEAD. The remaining finding is a P2 doc discrepancy that doesn't affect correctness. docs/dev-tools/backends/aqua.md — the github_tag sentence on the prerelease opt needs updating to reflect that the regex filter is also disabled. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["list_remote_versions(config)"] --> B["get_tool_opts → want_prereleases"]
B --> C{"versions host\nhas data?"}
C -- "Yes (prerelease=false on all)" --> D["filter_cached_prereleases\n(passes all through)"]
C -- "No → _list_remote_versions" --> E{"Backend type"}
E -- "github: / aqua: releases" --> F["list_releases_including_prereleases\n(superset, drafts only removed)"]
F --> G["VersionInfo.prerelease stamped\nper entry"]
G --> H["Stored in remote_versions_v2 cache\n(unfiltered superset)"]
H --> D
E -- "gitlab / forgejo" --> I["VersionInfo.prerelease = false\n(no flag available)"]
I --> D
D --> J{"want_prereleases?"}
J -- "true" --> K["Full list returned"]
J -- "false" --> L["Drop entries where\nVersionInfo.prerelease = true"]
L --> M["list_versions_matching"]
K --> M
M --> N{"filter_prereleases flag\n= !want_prereleases"}
N -- "true (default)" --> O["VERSION_REGEX drops\npre-release-shaped strings"]
N -- "false (opted in)" --> P["Regex filter skipped\nAll matching versions kept"]
O --> Q["Final version list"]
P --> Q
Reviews (8): Last reviewed commit: "[autofix.ci] apply automated fixes" | Re-trigger Greptile |
- Move `include_prereleases` helper into `src/backend/mod.rs` and share it
across the github/aqua backends (previously duplicated). Match on both
`toml::Value::Boolean` and `toml::Value::String` for defense-in-depth.
- Add `filter_prereleases: bool` param to the `fuzzy_match_filter` trait
method. Callers in `list_versions_matching{,_with_opts}` compute the
flag from config-aware opts (`config.get_tool_opts`), so values set in
`mise.toml` are honored during fuzzy matching — not just inline opts.
- Drop the github backend's `fuzzy_match_filter` override (the default
impl + config-aware caller handle it). Update aqua/java/ubi overrides
to take the new param.
- Return `Ok(None)` from github `latest_stable_version` when
`prerelease = true` so the trait's `latest_version` falls through to
`latest_version_for_query` (replaces the manual recursive call).
- `aqua::_list_remote_versions` now looks up opts via
`config.get_tool_opts`, matching the github backend pattern.
- Reword the `/releases/latest` doc note — it returns whichever release
the owner pinned as "Latest", not strictly stable-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-9329 releases of mise wrote the filtered (no-prerelease) release
list to `{key}-releases.msgpack.z`. After upgrading, users opting in
with `prerelease = true` would get cache hits on that stale file and
see no prereleases until the cache expired (up to
`fetch_remote_versions_cache`, default 1h), with no error.
Rename the on-disk cache file to `{key}-all-releases.msgpack.z` so the
post-upgrade cache starts fresh and the old filtered data is ignored.
|
I think there is still a cache correctness issue here around
That means the result depends on which mode populated the cache first:
I think the better fix is probably to make This comment was generated by an AI coding assistant. This is written by Codex, but I pointed out the bug to it so should be true. |
…lease The shared `remote_versions.msgpack.z` cache for github + aqua was keyed without a `prerelease` opt component, so flipping the option served a stale list until cache TTL. Worst case: a project-local `mise.local.toml` toggling `prerelease = true` had no visible effect until refetch. Cache the pre-release superset and filter on read. `VersionInfo` gains a `prerelease: bool` (sourced from GitHub's release flag), and the cache read path drops `prerelease = true` entries when the current opts don't opt in. Cache content is option-independent, so flips take effect instantly with no refetch — matching the UX of a developer toggling in and out of a project that pins a pre-release version. Versions host is bypassed for github + aqua: it serves a stable-only payload, which would re-introduce the option-dependent staleness if reused. Cache filenames bumped (`remote_versions_v2.msgpack.z`, `version_tags_v2.msgpack.z`) to invalidate pre-schema caches. Aqua's `get_version_tags` (used during install/lock resolution) now always fetches the superset, so a project-local override pinning a pre-release resolves correctly regardless of ambient config. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
@risu729 thanks for catching this — you were right that the original change only solved half the problem. Pushed eba6206 to fix it. A few notes on the approach and the principle behind it. The fixI went with your superset suggestion. Versions host: bypass always for github + aquaI extended the bypass to always skip the host for these backends, not only when The UX principle that drove the choiceThe case I had in mind: a developer who needs a pre-release version of a tool while working on something specific. The natural way to express that is a project-local override — Under that workflow the right behavior is for flipping in/out to be invisible — no spinner, no API call, just the right list each time. Anything that requires a refetch on flip violates the principle of least surprise: "why did mise hit GitHub? I didn't change anything, I just The two main alternatives didn't satisfy that:
The cost of always-bypass for default users is small. The on-disk releases cache ( Other things in this commit
VerificationManual integration test against
Plus two new unit tests in This comment was generated by an AI coding assistant. |
| // locally so a user flipping `prerelease = true` (e.g. via a project | ||
| // override) sees the right list immediately. The versions host serves | ||
| // a stable-only list, so reusing it would either re-introduce the | ||
| // option-dependent cache bug or force per-opt cache slots. |
There was a problem hiding this comment.
We shouldn't do this, the purpose of the versions host is to avoid GitHub rate limits, which applies for Aqua and GitHub backends mainly. We should include the prerelease metadata in the versions host ideally, but check if it doesn't break old mise versions. Then include prerelease metadata in mise ls-remote --json output. I think you also need to create a PR to jdx/mise-versions to update the DB schema.
There was a problem hiding this comment.
ah I think you're right, I didn't think of this
There was a problem hiding this comment.
@risu729 @jdx — agreed. Pushed aac50e0 taking the host-routes-everything approach you outlined.
Summary
- Add
prerelease: bool(withserde(default = false)) to the versions-hostVersionEntryschema, propagated ontoVersionInfofromversions_host.rs. - Drop the
BackendType::Github | BackendType::Aqua => falsebypass so both backends route through the host like every other rate-limited backend. - Add
prereleasetomise ls-remote --all --json(it was already serialized in single-tool mode viaVersionInfo; now also present in--all --jsonviaVersionOutputAll). - Backend-side
_list_remote_versions(still used as fallback when the host is rate-limited / down / opted-out /prefer_offline) continues to populateprereleasefromgithub::list_releases_including_prereleases*, so the offline/non-host path is consistent.
Backwards compatibility
Old mise versions reading the host data: toml-rs ignores unknown fields by default (we don't set deny_unknown_fields on VersionEntry), so adding prerelease = true/false to entries in mise-versions is forward-compatible — old clients see exactly the same created_at + release_url they see today.
New mise reading old host data: serde(default) on prerelease falls back to false, so any host TOML that hasn't been re-rendered yet looks the same to the new client as today (no prereleases visible). Once the companion mise-versions PR ships and re-renders, prereleases start surfacing automatically — no client change required.
Net: this PR is safe to merge in either order with the mise-versions schema update.
Companion PR
The host TOML producer side is in jdx/mise-versions. I'll open a PR there to:
- Include pre-release entries (currently filtered out at fetch time).
- Stamp each entry with the GitHub
prereleaseflag (and the equivalent for aqua sources). - Verify the rendered TOML stays parseable by the released mise versions before tagging.
The feature is functionally inert for github+aqua users until that ships and renders, but no regressions land in the meantime: with prerelease = false (the default) the user sees what they see today; with prerelease = true they see the stable-only list until the host catches up, which is the same as today minus the local-bypass shortcut.
This comment was generated by an AI coding assistant.
… metadata The bypass introduced for the prerelease feature defeated the host's main purpose — protecting github + aqua (the most rate-limit-sensitive backends) from upstream API quotas. Add an optional `prerelease` field to the host TOML schema and propagate it onto VersionInfo, so the host can serve the full superset for those backends as it does for everyone else. - Add `prerelease: bool` (with `serde(default = false)`) to VersionEntry in versions_host.rs; host TOML without the field stays parseable, so this is forward-compatible with the current mise-versions schema. - Old mise clients ignore unknown TOML fields by default, so populating `prerelease` on the producer side is backward-compatible. - Drop the BackendType::Github | BackendType::Aqua => false bypass in list_remote_versions_with_info; both now flow through the host like every other rate-limited backend. The backend-side _list_remote_versions remains as the offline / opted-out / rate-limited fallback and already populates `prerelease` from the upstream release flag. - Surface `prerelease` in `mise ls-remote --all --json` (single-tool JSON already serialized it via VersionInfo). Companion change required in jdx/mise-versions to actually emit the field; until that ships, github+aqua users see the same stable-only lists they see today regardless of `prerelease` opt-in.
### 🚀 Features - **(backend)** add global libc preference by @jdx in [#9404](#9404) - opt-in to pre-release versions for github and aqua backends by @jakedgy in [#9329](#9329) ### 🐛 Bug Fixes - **(backend)** allow unresolved latest opt-in by @jdx in [#9401](#9401) - **(install)** stop rewriting healthy runtime symlinks by @jdx in [#9410](#9410) - **(node)** route musl tarball URLs to unofficial-builds by @jdx in [#9409](#9409) - **(prune)** skip remote version resolution for tracked configs by @jdx in [#9406](#9406) - **(schema)** allow array values in tool additionalProperties by @JP-Ellis in [#9400](#9400) ### 📦️ Dependency Updates - bump communique to 1.1.2 by @jdx in [#9402](#9402) ### 📦 Registry - use aqua for rumdl by @scop in [#9397](#9397) ### Chore - **(ci)** improve pr-closer workflow by @jdx in [#9403](#9403) - **(release)** stop appending sponsor blurb when communique succeeds by @jdx in [#9395](#9395) ### New Contributors - @JP-Ellis made their first contribution in [#9400](#9400)
## Summary Companion to [jdx/mise#9329](jdx/mise#9329), which adds an opt-in \`prerelease\` tool option for the \`github\` and \`aqua\` backends. That PR plumbs the upstream pre-release flag into mise's \`ls-remote --json\` output and reads it back from the versions-host TOML; this PR is the producer side that makes the flag actually appear in the rendered TOML. ## Changes - **\`scripts/generate-toml.js\`** — reads \`prerelease\` from the NDJSON stdin and emits \`prerelease = true\` in the inline table when set. False/missing values are omitted to keep the TOML compact. - **\`scripts/sync-versions-to-d1.js\`** — forwards the field to the sync API. - **\`web/src/pages/api/admin/versions/sync.ts\`** — includes \`prerelease\` in the upsert SQL. - **\`src/analytics/migrations.ts\`** — idempotent migration: \`versions.prerelease INTEGER NOT NULL DEFAULT 0\`. Same pattern as the existing \`from_mise\` / \`sort_order\` migrations a few lines up. - **\`scripts/generate-toml.test.js\`** — round-trip tests for emit, omit, preserve-from-existing, and API-override semantics. ## Design note: definitive vs. unknown The merge logic distinguishes \"API explicitly said false\" (definitive — overrides any prior state) from \"this fetch fell back to the plain-text path\" (no info — preserve whatever the existing TOML said) by treating absent as \`null\` rather than \`false\`. This matches the existing fallback behavior for \`created_at\` / \`release_url\` and means a prior run's flag survives a transient JSON-fetch failure. Tests cover both paths. ## Compatibility | Direction | Behavior | |---|---| | Old mise clients reading new TOML | \`toml-rs\` ignores unknown fields by default → see exactly today's \`created_at\` + \`release_url\` | | New mise clients reading old TOML | \`#[serde(default)]\` on the field → absence parses as \`false\` (stable) | | This pipeline reading old \`mise ls-remote --json\` output | \`obj.prerelease\` is \`undefined\` → treated as unknown, existing TOML wins → no regression while the Docker image is on a pre-mise#9329 build | ## Test plan - [x] \`bun run test\` — 37/37 pass (24 JS + 13 shell) - [ ] On the first run after deploy, watch \`updated_tools.txt\` and confirm a github/aqua tool with known prereleases (e.g. tools that already get prereleases via aqua's \`--security\` flag, or post-mise#9329 deploy) renders \`prerelease = true\` in its TOML - [ ] D1 migration runs idempotently (it's wrapped in a column-existence check) ## Sequencing This PR is safe to merge in either order with [jdx/mise#9329](jdx/mise#9329). The feature only fully activates once both have shipped *and* the \`jdxcode/mise\` Docker image rebuilds with the new mise version. Until then, the producer side reads \`undefined\` for \`prerelease\` from \`ls-remote --json\` and the rendered TOML simply doesn't carry the field — same as today. *This pull request was prepared by an AI coding assistant.* --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…9415) ## Summary Companion to #9329, which added per-tool `prerelease = true` opt-in for the `github:` and `aqua:` backends. That covers the case where a user wants pre-releases for one specific tool. This PR adds the **global** escape hatch — equivalent to setting `prerelease = true` on every tool — for consumers that mirror the full release catalog or want pre-release tags to surface in `ls-remote` without per-tool config. The motivating case: the [versions host pipeline](https://github.com/jdx/mise-versions) calls `mise ls-remote --json` for every tracked tool to populate `docs/*.toml`. Per-tool config there isn't viable (≈900 tools), but without a global opt-in, mise filters every prerelease before emitting JSON, and the versions host can't carry the new `prerelease = true` field that jdx/mise-versions#135 just plumbed through. ## Changes - **New setting `prereleases`** (`MISE_PRERELEASES`) in `settings.toml`. Layered into `include_prereleases()` ahead of the per-tool option, so flipping it acts like `prerelease = true` on every tool. Has no effect on backends that don't carry an upstream prerelease flag (e.g. `github_tag` version source — git tags don't carry the prerelease bit). - **New `--prerelease` flag on `mise ls-remote`**. Calls a small `Settings::override_with` helper that merges into the existing `CLI_SETTINGS` partial — `Settings::reset` replaces wholesale, which would clobber overrides installed earlier in startup like `--offline` / `--quiet`. The merge approach lets a subcommand layer one extra override on top. - **Test** for the global-setting path alongside the existing per-tool test in `backend::tests`. ## Test plan - [x] `cargo test --bin mise backend::` — 193 pass - [x] `cargo test --bin mise config::settings` — 22 pass (incl. `test_settings_toml_is_sorted`) - [x] Smoke: `mise ls-remote --help` shows the new flag, `mise ls-remote --json --prerelease github:cli/cli` runs without error - [ ] CI ## Compatibility - Default behavior unchanged: `prereleases = false`, prereleases stay filtered. All existing tool configs continue to work. - The per-tool `prerelease = true` option is unaffected — both paths feed into the same `include_prereleases()` helper, which now ORs them together. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Medium risk because it changes version filtering behavior and settings layering/caching for remote version listings (and `latest`/fuzzy resolution via `include_prereleases`), which could affect tool resolution outputs across backends when enabled. > > **Overview** > Adds a new global `prereleases` setting (`MISE_PRERELEASES`) that, when enabled, makes `include_prereleases()` opt into upstream pre-release versions regardless of per-tool `prerelease` options. > > Extends `mise ls-remote` with `--prerelease`, implemented by merging a CLI-layer settings override (`Settings::override_with`) so the flag can be applied without clobbering other startup overrides. > > Updates generated CLI docs/usage spec accordingly and adds a regression test ensuring the global setting enables prerelease inclusion by default. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit c6f5ffb. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Implements the opt-in proposed in discussion #9323 (approved by @jdx). Also wires support into the
aqua:backend as a small follow-up, as requested.What changed
Adds a per-tool
prerelease = trueoption to thegithub:andaqua:backends. When set:prerelease: trueon GitHub appear inmise ls-remotelatestresolves against the full list including pre-releases1.2) match pre-release tags under that prefixDefault behavior is unchanged (
prerelease = false). Draft releases are always excluded.Where the filters lived
Two independent filters needed an escape hatch:
src/github.rs::list_releases_(!r.prerelease). Split so the cache holds unfiltered-except-drafts;list_releasespreserves old behavior via read-time filter, newlist_releases_including_prereleases{,_from_url}exposes the unfiltered view. Cache is shared — no extra API cost.VERSION_REGEXfilter infuzzy_match_filter(trait default +aquaoverride). Extracted into a sharedfuzzy_match_versions(versions, query, filter_prereleases)helper so both call paths can opt out.Also, the
github:backend'slatest_stable_versionuses the/releases/latestendpoint (stable-only by design) — whenprerelease = trueit now falls through tolatest_version_for_query("latest", None)to resolve against the full list. Uses the_for_queryvariant deliberately to avoid recursing back throughlatest_stable_version.Tests
fuzzy_match_versions: unit tests pinning both the default (filters prereleases) and opted-in (includes them) behavior acrosslatestand partial queriesUnifiedGitBackend::fuzzy_match_filter: unit tests verifying the override readsself.ba.opts()and toggles correctlyinclude_prereleases: bool-parsing sanity checktest_inline_install_before_wins_over_config_entry— reproduces on cleanmain, unrelated)What isn't here
No e2e test yet —
jdx/mise-test-fixturesonly has one stable release, so exercising the opt end-to-end needs a fixture with at least one release flaggedprerelease: true. Happy to add that in a follow-up if you can flag an existing tag (or I can mock this differently if you'd prefer).Not touching GitLab/Forgejo — the opt is documented as a no-op for those backends. Can extend later if there's demand.