Skip to content

Make DataDesignerPlugins an install-ready plugin catalog#36

Merged
eric-tramel merged 26 commits into
mainfrom
codex/epic-15-plugin-tap
May 8, 2026
Merged

Make DataDesignerPlugins an install-ready plugin catalog#36
eric-tramel merged 26 commits into
mainfrom
codex/epic-15-plugin-tap

Conversation

@eric-tramel

@eric-tramel eric-tramel commented May 7, 2026

Copy link
Copy Markdown
Contributor

What

Makes this repository consumable as a Data Designer plugin catalog with a package-first Catalog Schema and a static Python package index.

The final design uses standard Python package installation primitives:

  • the canonical NVIDIA catalog is served from GitHub Pages at https://nvidia-nemo.github.io/DataDesignerPlugins/catalog/plugins.json;
  • catalog packages carry install.requirement and optional install.index_url metadata instead of custom source objects;
  • catalog metadata does not duplicate package versions; versions stay in package artifacts and the Simple API index, and installers such as uv resolve them;
  • built wheels and source distributions are uploaded to the ddp-package-assets GitHub Release;
  • dumb-pypi builds a static Simple API index under https://nvidia-nemo.github.io/DataDesignerPlugins/simple/;
  • release and docs workflows deploy docs, catalog JSON, packages.json, and the Simple API index together.

The implementation also adds package-index repo tooling, release metadata merging, package row validation, local scratch QA, schema fixtures, docs, and release checks for the install metadata contract. The repo-level configuration is now [tool.ddp.catalog], and the internal helper module is ddp.catalog_config.

Closes #15.

Why

The catalog should describe installable plugin packages without assuming packages live in this repository, requiring wheels to be checked into git, or reimplementing resolver behavior that package managers already provide. Using PEP 508 requirements plus the PyPA Simple API keeps NDD-style clients close to normal pip/uv behavior while preserving a curated plugin discovery surface.

GitHub Release assets provide the binary/object storage layer, GitHub Pages provides the static installer surface, and dumb-pypi avoids building a custom package index server.

Usage

Generate or validate the catalog and package index locally:

make catalog
make package-index
make check-package-index
make qa-package-index

A catalog package looks like:

{
  "name": "data-designer-template",
  "install": {
    "requirement": "data-designer-template",
    "index_url": "https://nvidia-nemo.github.io/DataDesignerPlugins/simple/"
  }
}

An NDD-style installer using uv can install an index-backed package with:

uv pip install \
  --default-index https://pypi.org/simple/ \
  --index https://nvidia-nemo.github.io/DataDesignerPlugins/simple/ \
  data-designer-template

External catalogs can still use direct PEP 508 references, for example:

data-designer-example @ git+https://github.com/acme/plugin.git@v1.0.0
data-designer-example @ https://packages.example.test/data_designer_example-0.1.0-py3-none-any.whl

How

ddp.catalog validates and renders install metadata, derives installer-ready targets, and keeps package/runtime plugin grouping intact. ddp.catalog_config owns the catalog URL, package index URL, release asset URL, and asset release tag.

ddp package-index adds subcommands to build the Pages package-index outputs, check package-list rows and generated files, merge built dist files into the mutable JSON-lines packages.json, and run an end-to-end scratch QA install.

The publish workflow builds the releasing plugin, runs twine check, uploads wheel/sdist assets without clobbering changed bytes, updates the packages.json release asset, rebuilds docs plus the static package index, and deploys the complete site to GitHub Pages. The docs workflow also rebuilds the package index so docs-only deploys do not delete simple/.

Validation

Latest local validation:

make format
uv run pytest devtools/ddp/tests/ -q
make docs
make lint
make check
make qa-package-index
make test
make validate

Results:

  • make format: passed.
  • uv run pytest devtools/ddp/tests/ -q: 144 passed.
  • make docs: strict docs build passed and synced package-index files into site/.
  • make lint: passed.
  • make check: plugin docs, catalog, package index, CODEOWNERS, and license headers all current.
  • make qa-package-index: passed in /tmp/ddp-package-index-qa.
  • make test: passed 144 devtool tests, 63 retrieval plugin isolated tests, and 11 template plugin isolated tests.
  • make validate: validated text-transform, document-chunker, and embedding-dedup entry points.
  • Terminology scan: no tracked legacy catalog-name aliases remain.

The scratch QA built data-designer-template wheel and sdist into /tmp/ddp-package-index-qa/packages/, generated a local dumb-pypi Simple API under /tmp/ddp-package-index-qa/site/simple/, read the bare package requirement from the checked-in catalog, substituted the local scratch index for the configured remote index, installed through uv into /tmp/ddp-package-index-qa/venv, and verified the installed data_designer.plugins entry point with assert_valid_plugin.

eric-tramel added 20 commits May 7, 2026 15:25
…ract

Define tap catalog schema v2 contract
@eric-tramel eric-tramel requested a review from a team as a code owner May 7, 2026 20:12
johnnygreco added a commit to NVIDIA-NeMo/DataDesigner that referenced this pull request May 8, 2026
Validate tap catalogs against the schema v2 contract used by
NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields,
docs URLs, package paths, compatibility metadata, and unique runtime
plugin names.

Derive Git install targets as package-qualified PEP 508 direct
references so git tap entries install the package described by the
catalog source metadata.

Refs #617
@eric-tramel eric-tramel changed the title Make DataDesignerPlugins an install-ready plugin tap Make DataDesignerPlugins an install-ready plugin catalog May 8, 2026
@eric-tramel eric-tramel closed this May 8, 2026
@eric-tramel eric-tramel deleted the codex/epic-15-plugin-tap branch May 8, 2026 17:37
@eric-tramel eric-tramel restored the codex/epic-15-plugin-tap branch May 8, 2026 17:38
@eric-tramel eric-tramel reopened this May 8, 2026
@eric-tramel eric-tramel merged commit e192688 into main May 8, 2026
12 checks passed
johnnygreco added a commit to NVIDIA-NeMo/DataDesigner that referenced this pull request May 9, 2026
Validate tap catalogs against the schema v2 contract used by
NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields,
docs URLs, package paths, compatibility metadata, and unique runtime
plugin names.

Derive Git install targets as package-qualified PEP 508 direct
references so git tap entries install the package described by the
catalog source metadata.

Refs #617
johnnygreco added a commit to NVIDIA-NeMo/DataDesigner that referenced this pull request May 11, 2026
Validate tap catalogs against the schema v2 contract used by
NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields,
docs URLs, package paths, compatibility metadata, and unique runtime
plugin names.

Derive Git install targets as package-qualified PEP 508 direct
references so git tap entries install the package described by the
catalog source metadata.

Refs #617
johnnygreco added a commit to NVIDIA-NeMo/DataDesigner that referenced this pull request May 11, 2026
Validate tap catalogs against the schema v2 contract used by
NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields,
docs URLs, package paths, compatibility metadata, and unique runtime
plugin names.

Derive Git install targets as package-qualified PEP 508 direct
references so git tap entries install the package described by the
catalog source metadata.

Refs #617
johnnygreco added a commit to NVIDIA-NeMo/DataDesigner that referenced this pull request May 12, 2026
Validate tap catalogs against the schema v2 contract used by
NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields,
docs URLs, package paths, compatibility metadata, and unique runtime
plugin names.

Derive Git install targets as package-qualified PEP 508 direct
references so git tap entries install the package described by the
catalog source metadata.

Refs #617
johnnygreco added a commit to NVIDIA-NeMo/DataDesigner that referenced this pull request May 13, 2026
* feat(cli): add plugin catalog services

Add typed catalog and tap models, persistent tap storage, cached
catalog loading, compatibility evaluation, install plan generation,
and runtime plugin discovery helpers.

Refs #617

* feat(cli): add plugins command group

Wire list, search, info, install, installed, and tap management
commands through the existing command-controller CLI pattern.

Refs #617

* test(cli): cover plugin catalog workflows

Add regression coverage for tap caching, catalog compatibility,
installer command generation, local path resolution, and Typer command
delegation.

Refs #617

* fix(cli): align plugin taps with schema v2

Validate tap catalogs against the schema v2 contract used by
NVIDIA-NeMo/DataDesignerPlugins#36, including source union fields,
docs URLs, package paths, compatibility metadata, and unique runtime
plugin names.

Derive Git install targets as package-qualified PEP 508 direct
references so git tap entries install the package described by the
catalog source metadata.

Refs #617

* fix(cli): address plugin review feedback

- Invalidate import caches before post-install entry point verification
- Make tap aliases case-insensitive and cache catalogs by alias plus URL
- Prefer compatible catalog entries before falling back to forced installs
- Clarify unused --tap behavior and list installed entry points without imports
- Add direct controller coverage and update CLI plugin documentation

Refs #617

* fix(cli): gate incompatible plugin installs

Fetch install targets before compatibility filtering so the controller
owns the final --force decision and the incompatible install guard stays
reachable.

Refs #617

* style(cli): format plugin catalog files

Apply ruff formatting to the plugin command and tap repository tests so
CI format checks pass on the PR merge commit.

Refs #617

* fix(cli): reject duplicate plugin entry names

Key catalog duplicate detection by entry_point.name so distinct catalog
entries cannot register the same runtime plugin name.

Refs #617

* fix(cli): preserve GitHub tree tap paths

* fix(cli): verify plugin entry point names

* align plugin CLI with catalog schema

- adopt catalog terminology for plugin source aliases
- parse package-first plugin catalog metadata from the plugin repo
- install package requirements with optional catalog indexes

* tidy plugin catalog workflow docs

* align plugin catalog CLI with package contract

* add plugin package uninstall workflow

* test plugin package command targets

* document plugin package aliases

* address plugin catalog review feedback

* prefer runtime plugin lookup matches

* rename plugins command to plugin

* show plugin package descriptions

* rename plugin catalogs command

* add protected plugin package installs

* document plugin package install modes

* avoid building project during plugin installs

* harden plugin package installs

* tighten plugin catalog contracts

* fix no-args help exit code

* make plugin docs links robust

* document plugin CLI catalog workflows

* clarify plugin entry point verification

* simplify plugin CLI docs

* narrow plugin search fields

* hide plugin catalog cache ttl

* remove plugin catalog trust flag

* improve plugin CLI recovery UX

* polish plugin catalog table display

* stabilize plugin catalog table test

* tighten plugin catalog edge cases

* harden plugin catalog verification

- Escape catalog-provided Rich markup before rendering CLI output
- Reject runtime plugin names that collide after enum-key normalization
- Load installed runtime entry points in a subprocess before reporting success

* simplify plugin entry point verification

Load matching entry points directly after install instead of spawning a
separate Python process. This keeps the check package-scoped while still
catching broken entry-point targets and non-Plugin objects.

* require newer uv for plugin plans

Use uv >= 0.10.0 as the single supported uv requirement for
plugin package commands. Auto mode now falls back to a pip plan with
an upgrade warning when uv is unavailable or too old, while explicit
uv selection remains strict.

* verify pip fallback availability

* polish plugin CLI status markers

* clarify plugin compatibility labels

* simplify plugin info install details

* address plugin CLI review nits

* support versioned plugin package installs

* share plugin install metadata rendering

* show installed plugin packages

* harden versioned plugin installs

- Preserve catalog requirement constraints for versioned installs
- Remove stale install-plan metadata fields
- Expand parser, uv, controller, and local-catalog dry-run coverage

* harden plugin help tests

* show plugin package versions

Add package version metadata support for plugin catalogs and resolve current versions from exact requirements or simple indexes when catalog entries omit them.

Update plugin list/info/install metadata to show the plugin package version and Data Designer compatibility requirement while removing the separate Data Designer version line.

* format plugin catalog tests

* harden plugin package metadata checks

* harden plugin CLI test coverage

* add plugin discovery docs (#642)

Signed-off-by: Johnny Greco <jogreco@nvidia.com>

---------

Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Epic: Make DataDesignerPlugins an install-ready Data Designer plugin tap

2 participants