Skip to content

feat(skills): comfyui skill v3 — official CLI + REST API, no third-party deps#17607

Closed
alt-glitch wants to merge 5 commits into
mainfrom
feat/comfyui-skill-v3
Closed

feat(skills): comfyui skill v3 — official CLI + REST API, no third-party deps#17607
alt-glitch wants to merge 5 commits into
mainfrom
feat/comfyui-skill-v3

Conversation

@alt-glitch

@alt-glitch alt-glitch commented Apr 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Complete rewrite of the ComfyUI optional skill. Replaces the previous approach
(which depended on the unofficial comfyui-skill-cli) with a two-layer
architecture using only official tooling:

  1. comfy-cli (Comfy-Org/comfy-cli) for lifecycle management — install,
    launch, stop, custom node management, model downloads
  2. Direct REST API + helper scripts for workflow execution — parameter
    injection, submission, monitoring, output download, dependency checking

What's Included

File Lines Purpose
SKILL.md 567 Main doc: architecture, full onboarding (5 install paths), decision tree, cloud, pitfalls
references/official-cli.md 268 Full comfy-cli command reference
references/rest-api.md 256 All REST API endpoints (local + Comfy Cloud)
references/workflow-format.md 218 API format spec, common nodes, parameter mapping
scripts/comfyui_setup.sh 77 One-shot setup automation
scripts/extract_schema.py 212 Workflow JSON → controllable parameter schema
scripts/run_workflow.py 352 Inject args → submit → poll → download outputs
scripts/check_deps.py 179 Check missing nodes/models against running server

Why Scripts Are Needed: Gaps in the Official CLI

The official comfy-cli is excellent for installation and server lifecycle but
was not designed for agent-driven workflow execution. Here's what it can and
can't do:

Agent need Official comfy-cli Gap filled by
Install ComfyUI comfy install
Launch/stop server comfy launch/stop
Install custom nodes comfy node install
Download models comfy model download
Submit raw workflow JSON comfy run --workflow
Inject user params into workflow ❌ No support run_workflow.py
Structured JSON output from execution ❌ Human-readable only run_workflow.py
Cloud auth (X-API-Key header) ❌ Not supported run_workflow.py
Poll job status until completion ❌ 30s timeout, no polling run_workflow.py
Download output files (images/video) ❌ Not supported run_workflow.py
Auto-detect controllable parameters ❌ No schema extraction extract_schema.py
Check missing nodes/models per workflow ❌ No per-workflow check check_deps.py
Upload images for img2img/inpainting ❌ Not supported REST API (documented)
Queue management (list/clear/cancel) ❌ Not supported REST API (documented)
Free GPU VRAM ❌ Not supported REST API (documented)

The comfy run command in particular is very limited — it takes a raw JSON file,
submits it verbatim to a local server with a 30-second timeout, shows a progress
bar, and exits. No parameter modification, no cloud support, no output download,
no structured response the agent can parse.

Our scripts (run_workflow.py, extract_schema.py, check_deps.py) fill these
gaps using stdlib Python + requests (~740 lines total), making the full ComfyUI
execution flow agent-friendly without depending on any unofficial third-party CLI.

Onboarding

Comprehensive setup guide covering all installation pathways:

  • Path A: Comfy Cloud (zero setup, API key)
  • Path B: ComfyUI Desktop (Windows/macOS one-click)
  • Path C: Portable build (Windows extract-and-run)
  • Path D: comfy-cli (recommended for agents, all platforms)
  • Path E: Manual install (advanced, all hardware)

Each path links to official docs (docs.comfy.org).

Credits

The approach for workflow parameter extraction, schema mapping, and execution
orchestration in our helper scripts was informed by
@HuangYuChuh's work on
comfyui-skill-cli
particularly the patterns for identifying controllable nodes by class type,
the model loader → folder mapping for dependency checking, and the
WebSocket/polling execution flow. Credit to their project for pioneering the
agent-friendly ComfyUI interface patterns.

Closes #17316
Closes #11143
Closes #13271

…y dependency

Complete rewrite of the ComfyUI skill to use:
- comfy-cli (official, Comfy-Org/comfy-cli) for lifecycle management:
  install, launch, stop, node management, model downloads
- Direct REST API + helper scripts for workflow execution:
  parameter injection, submission, monitoring, output download
- No dependency on comfyui-skill-cli or any unofficial tool

New files:
- SKILL.md: full rewrite with two-layer architecture, decision tree, pitfalls
- references/official-cli.md: complete comfy-cli command reference
- references/rest-api.md: all REST endpoints (local + cloud)
- references/workflow-format.md: API format spec, common nodes, param mapping
- scripts/extract_schema.py: analyze workflow → extract controllable params
- scripts/run_workflow.py: inject args, submit, poll, download outputs
- scripts/check_deps.py: check missing nodes/models against running server
- scripts/comfyui_setup.sh: full setup automation with official CLI

Removed:
- references/cli-reference.md (was for unofficial comfyui-skill-cli)
- references/api-notes.md (replaced by rest-api.md)

Addresses feedback from PR #17316 comment:
- Correct author attribution
- Remove references to unofficial OpenClaw project
- License field reflects hermes-agent repo (MIT)
Comment thread optional-skills/creative/comfyui/scripts/check_deps.py Fixed
Comment thread optional-skills/creative/comfyui/scripts/run_workflow.py Fixed
alt-glitch and others added 3 commits April 30, 2026 00:56
…links, cloud setup

Adds structured onboarding flow to SKILL.md:
- Decision table: which install path for which situation
- Path A: Comfy Cloud (zero setup, API key, pricing)
- Path B: Desktop app (Windows/macOS, one-click)
- Path C: Portable build (Windows, extract-and-run)
- Path D: comfy-cli (recommended for agents, all platforms)
- Path E: Manual install (advanced, all hardware types)
- Post-install: model downloads, custom nodes, verification

All paths link to official docs:
- https://docs.comfy.org/installation
- https://docs.comfy.org/comfy-cli/getting-started
- https://docs.comfy.org/get_started/cloud
- https://docs.comfy.org/installation/desktop
- https://docs.comfy.org/installation/comfyui_portable_windows
- https://docs.comfy.org/installation/manual_install
…ring sanitization'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…ring sanitization'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
@alt-glitch alt-glitch marked this pull request as ready for review April 29, 2026 19:31
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have tool/skills Skills system (list, view, manage) labels Apr 29, 2026
…to onboarding

Instead of asking the user what they have, the agent now:
1. Runs system detection commands (OS, GPU, VRAM, RAM, disk, Python)
2. Checks if ComfyUI is already installed/running
3. Recommends the best path based on findings

Adds:
- Step 1: detection script block (nvidia-smi, system_profiler, etc.)
- Step 2: decision table mapping detected system → recommended path
- Hardware requirements table (VRAM tiers, RAM, disk)
- Specific recommendations per platform:
  macOS → Desktop app, Linux+NVIDIA → comfy-cli, no GPU → Cloud, etc.
@teknium1

Copy link
Copy Markdown
Contributor

Merged via #17610 — all four of your commits (v4 rewrite, onboarding docs, two CodeQL fixes) were cherry-picked onto current main with your authorship preserved via rebase-merge. A hardware-feasibility check (scripts/hardware_check.py) was layered on top to auto-gate the local install so users with <6 GB VRAM, Intel Macs, or <16 GB Apple unified memory get routed to Comfy Cloud instead of silently hitting OOM on first workflow. Thanks for the excellent rewrite!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P3 Low — cosmetic, nice to have tool/skills Skills system (list, view, manage) type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants