Make DataDesignerPlugins an install-ready plugin catalog#36
Merged
Conversation
…ract Define tap catalog schema v2 contract
Add repo-level tap metadata config
Add schema v2 catalog source metadata
Add scaffold plugin type support
Add tap source install target derivation
Document default tap catalog URL
Document plugin tap governance and trust
Add tap catalog contract fixtures
…tion Strengthen release validation
Document plugin tap workflows
10 tasks
johnnygreco
added a commit
to NVIDIA-NeMo/DataDesigner
that referenced
this pull request
May 8, 2026
Validate tap catalogs against the schema v2 contract used by NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields, docs URLs, package paths, compatibility metadata, and unique runtime plugin names. Derive Git install targets as package-qualified PEP 508 direct references so git tap entries install the package described by the catalog source metadata. Refs #617
johnnygreco
approved these changes
May 8, 2026
johnnygreco
added a commit
to NVIDIA-NeMo/DataDesigner
that referenced
this pull request
May 9, 2026
Validate tap catalogs against the schema v2 contract used by NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields, docs URLs, package paths, compatibility metadata, and unique runtime plugin names. Derive Git install targets as package-qualified PEP 508 direct references so git tap entries install the package described by the catalog source metadata. Refs #617
johnnygreco
added a commit
to NVIDIA-NeMo/DataDesigner
that referenced
this pull request
May 11, 2026
Validate tap catalogs against the schema v2 contract used by NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields, docs URLs, package paths, compatibility metadata, and unique runtime plugin names. Derive Git install targets as package-qualified PEP 508 direct references so git tap entries install the package described by the catalog source metadata. Refs #617
6 tasks
johnnygreco
added a commit
to NVIDIA-NeMo/DataDesigner
that referenced
this pull request
May 11, 2026
Validate tap catalogs against the schema v2 contract used by NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields, docs URLs, package paths, compatibility metadata, and unique runtime plugin names. Derive Git install targets as package-qualified PEP 508 direct references so git tap entries install the package described by the catalog source metadata. Refs #617
johnnygreco
added a commit
to NVIDIA-NeMo/DataDesigner
that referenced
this pull request
May 12, 2026
Validate tap catalogs against the schema v2 contract used by NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields, docs URLs, package paths, compatibility metadata, and unique runtime plugin names. Derive Git install targets as package-qualified PEP 508 direct references so git tap entries install the package described by the catalog source metadata. Refs #617
johnnygreco
added a commit
to NVIDIA-NeMo/DataDesigner
that referenced
this pull request
May 13, 2026
* feat(cli): add plugin catalog services Add typed catalog and tap models, persistent tap storage, cached catalog loading, compatibility evaluation, install plan generation, and runtime plugin discovery helpers. Refs #617 * feat(cli): add plugins command group Wire list, search, info, install, installed, and tap management commands through the existing command-controller CLI pattern. Refs #617 * test(cli): cover plugin catalog workflows Add regression coverage for tap caching, catalog compatibility, installer command generation, local path resolution, and Typer command delegation. Refs #617 * fix(cli): align plugin taps with schema v2 Validate tap catalogs against the schema v2 contract used by NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields, docs URLs, package paths, compatibility metadata, and unique runtime plugin names. Derive Git install targets as package-qualified PEP 508 direct references so git tap entries install the package described by the catalog source metadata. Refs #617 * fix(cli): address plugin review feedback - Invalidate import caches before post-install entry point verification - Make tap aliases case-insensitive and cache catalogs by alias plus URL - Prefer compatible catalog entries before falling back to forced installs - Clarify unused --tap behavior and list installed entry points without imports - Add direct controller coverage and update CLI plugin documentation Refs #617 * fix(cli): gate incompatible plugin installs Fetch install targets before compatibility filtering so the controller owns the final --force decision and the incompatible install guard stays reachable. Refs #617 * style(cli): format plugin catalog files Apply ruff formatting to the plugin command and tap repository tests so CI format checks pass on the PR merge commit. Refs #617 * fix(cli): reject duplicate plugin entry names Key catalog duplicate detection by entry_point.name so distinct catalog entries cannot register the same runtime plugin name. Refs #617 * fix(cli): preserve GitHub tree tap paths * fix(cli): verify plugin entry point names * align plugin CLI with catalog schema - adopt catalog terminology for plugin source aliases - parse package-first plugin catalog metadata from the plugin repo - install package requirements with optional catalog indexes * tidy plugin catalog workflow docs * align plugin catalog CLI with package contract * add plugin package uninstall workflow * test plugin package command targets * document plugin package aliases * address plugin catalog review feedback * prefer runtime plugin lookup matches * rename plugins command to plugin * show plugin package descriptions * rename plugin catalogs command * add protected plugin package installs * document plugin package install modes * avoid building project during plugin installs * harden plugin package installs * tighten plugin catalog contracts * fix no-args help exit code * make plugin docs links robust * document plugin CLI catalog workflows * clarify plugin entry point verification * simplify plugin CLI docs * narrow plugin search fields * hide plugin catalog cache ttl * remove plugin catalog trust flag * improve plugin CLI recovery UX * polish plugin catalog table display * stabilize plugin catalog table test * tighten plugin catalog edge cases * harden plugin catalog verification - Escape catalog-provided Rich markup before rendering CLI output - Reject runtime plugin names that collide after enum-key normalization - Load installed runtime entry points in a subprocess before reporting success * simplify plugin entry point verification Load matching entry points directly after install instead of spawning a separate Python process. This keeps the check package-scoped while still catching broken entry-point targets and non-Plugin objects. * require newer uv for plugin plans Use uv >= 0.10.0 as the single supported uv requirement for plugin package commands. Auto mode now falls back to a pip plan with an upgrade warning when uv is unavailable or too old, while explicit uv selection remains strict. * verify pip fallback availability * polish plugin CLI status markers * clarify plugin compatibility labels * simplify plugin info install details * address plugin CLI review nits * support versioned plugin package installs * share plugin install metadata rendering * show installed plugin packages * harden versioned plugin installs - Preserve catalog requirement constraints for versioned installs - Remove stale install-plan metadata fields - Expand parser, uv, controller, and local-catalog dry-run coverage * harden plugin help tests * show plugin package versions Add package version metadata support for plugin catalogs and resolve current versions from exact requirements or simple indexes when catalog entries omit them. Update plugin list/info/install metadata to show the plugin package version and Data Designer compatibility requirement while removing the separate Data Designer version line. * format plugin catalog tests * harden plugin package metadata checks * harden plugin CLI test coverage * add plugin discovery docs (#642) Signed-off-by: Johnny Greco <jogreco@nvidia.com> --------- Signed-off-by: Johnny Greco <jogreco@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes this repository consumable as a Data Designer plugin catalog with a package-first Catalog Schema and a static Python package index.
The final design uses standard Python package installation primitives:
https://nvidia-nemo.github.io/DataDesignerPlugins/catalog/plugins.json;install.requirementand optionalinstall.index_urlmetadata instead of custom source objects;uvresolve them;ddp-package-assetsGitHub Release;dumb-pypibuilds a static Simple API index underhttps://nvidia-nemo.github.io/DataDesignerPlugins/simple/;packages.json, and the Simple API index together.The implementation also adds package-index repo tooling, release metadata merging, package row validation, local scratch QA, schema fixtures, docs, and release checks for the install metadata contract. The repo-level configuration is now
[tool.ddp.catalog], and the internal helper module isddp.catalog_config.Closes #15.
Why
The catalog should describe installable plugin packages without assuming packages live in this repository, requiring wheels to be checked into git, or reimplementing resolver behavior that package managers already provide. Using PEP 508 requirements plus the PyPA Simple API keeps NDD-style clients close to normal
pip/uvbehavior while preserving a curated plugin discovery surface.GitHub Release assets provide the binary/object storage layer, GitHub Pages provides the static installer surface, and
dumb-pypiavoids building a custom package index server.Usage
Generate or validate the catalog and package index locally:
A catalog package looks like:
{ "name": "data-designer-template", "install": { "requirement": "data-designer-template", "index_url": "https://nvidia-nemo.github.io/DataDesignerPlugins/simple/" } }An NDD-style installer using
uvcan install an index-backed package with:External catalogs can still use direct PEP 508 references, for example:
How
ddp.catalogvalidates and rendersinstallmetadata, derives installer-ready targets, and keeps package/runtime plugin grouping intact.ddp.catalog_configowns the catalog URL, package index URL, release asset URL, and asset release tag.ddp package-indexadds subcommands to build the Pages package-index outputs, check package-list rows and generated files, merge built dist files into the mutable JSON-linespackages.json, and run an end-to-end scratch QA install.The publish workflow builds the releasing plugin, runs
twine check, uploads wheel/sdist assets without clobbering changed bytes, updates thepackages.jsonrelease asset, rebuilds docs plus the static package index, and deploys the complete site to GitHub Pages. The docs workflow also rebuilds the package index so docs-only deploys do not deletesimple/.Validation
Latest local validation:
make format uv run pytest devtools/ddp/tests/ -q make docs make lint make check make qa-package-index make test make validateResults:
make format: passed.uv run pytest devtools/ddp/tests/ -q: 144 passed.make docs: strict docs build passed and synced package-index files intosite/.make lint: passed.make check: plugin docs, catalog, package index, CODEOWNERS, and license headers all current.make qa-package-index: passed in/tmp/ddp-package-index-qa.make test: passed 144 devtool tests, 63 retrieval plugin isolated tests, and 11 template plugin isolated tests.make validate: validatedtext-transform,document-chunker, andembedding-dedupentry points.The scratch QA built
data-designer-templatewheel and sdist into/tmp/ddp-package-index-qa/packages/, generated a localdumb-pypiSimple API under/tmp/ddp-package-index-qa/site/simple/, read the bare package requirement from the checked-in catalog, substituted the local scratch index for the configured remote index, installed throughuvinto/tmp/ddp-package-index-qa/venv, and verified the installeddata_designer.pluginsentry point withassert_valid_plugin.