fix(kanban): reject toolset names in task skills field (salvage #22933)#23273
Merged
Conversation
Follow-up to the previous commit's toolset-vs-skill validation.
The contributor's fix raises ValueError on the first toolset name found
in the skills list. That works for one mistake, but agents that confuse
skills with toolsets usually pass several at once
(`skills=["web", "browser", "terminal"]`) — and serial-correcting one
per failure round-trip wastes tokens. Collect all toolset-shaped
entries first, then raise once with the full list.
The error message is also slightly clearer:
'web', 'browser', 'terminal' are toolset names, not skill name(s).
Put toolsets in the assignee profile's `toolsets:` config instead of
per-task skills. Skills are named skill bundles (e.g. `kanban-worker`,
`blogwatcher`); toolsets are runtime capabilities (e.g. `web`,
`browser`, `terminal`).
vs. the previous "the assignee profile's toolsets" — explicitly naming
the YAML key (`toolsets:`) and giving concrete examples in both
categories closes the conceptual gap that produced the bug to begin
with.
Adds one regression test (test_create_task_skills_lists_all_toolset_typos)
covering the multi-name aggregation path. The single-typo test from
the original PR still passes (the loose `match="toolset name"` matches
both singular and plural forms).
Contributor
🔎 Lint report:
|
| Rule | Count |
|---|---|
invalid-argument-type |
8 |
unsupported-operator |
3 |
unresolved-attribute |
3 |
not-subscriptable |
1 |
First entries
run_agent.py:2590: [invalid-argument-type] invalid-argument-type: Argument to function `build_anthropic_client` is incorrect: Expected `str`, found `(Unknown & ~AlwaysFalsy) | (str & ~AlwaysFalsy) | (dict[str, str] & ~AlwaysFalsy) | ... omitted 4 union elements`
run_agent.py:2339: [invalid-argument-type] invalid-argument-type: Argument to function `query_ollama_num_ctx` is incorrect: Expected `str`, found `(str & ~AlwaysFalsy) | (dict[str, str] & ~AlwaysFalsy) | (Any & ~AlwaysFalsy) | ... omitted 4 union elements`
tests/run_agent/test_provider_attribution_headers.py:156: [unsupported-operator] unsupported-operator: Operator `not in` is not supported between objects of type `Literal["X-OpenRouter-Cache-TTL"]` and `Unknown | str | dict[str, str] | ... omitted 3 union elements`
tests/run_agent/test_provider_attribution_headers.py:155: [unsupported-operator] unsupported-operator: Operator `not in` is not supported between objects of type `Literal["X-OpenRouter-Cache"]` and `Unknown | str | dict[str, str] | ... omitted 3 union elements`
run_agent.py:6989: [invalid-argument-type] invalid-argument-type: Argument to function `_codex_cloudflare_headers` is incorrect: Expected `str`, found `Unknown | str | dict[str, str] | ... omitted 3 union elements`
run_agent.py:7160: [invalid-argument-type] invalid-argument-type: Argument to function `build_anthropic_client` is incorrect: Expected `str`, found `str | dict[Unknown, Unknown] | Any | ... omitted 3 union elements`
run_agent.py:2641: [invalid-argument-type] invalid-argument-type: Argument to function `get_model_context_length` is incorrect: Expected `str`, found `str | dict[str, str] | Any | ... omitted 3 union elements`
tests/agent/test_codex_cloudflare_headers.py:163: [unresolved-attribute] unresolved-attribute: Attribute `startswith` is not defined on `dict[str, str]` in union `Unknown | str | dict[str, str]`
tests/run_agent/test_provider_attribution_headers.py:154: [not-subscriptable] not-subscriptable: Cannot subscript object of type `int` with no `__getitem__` method
tests/agent/test_codex_cloudflare_headers.py:163: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str & ~AlwaysFalsy`, `int & ~AlwaysFalsy` in union `(Unknown & ~AlwaysFalsy) | (str & ~AlwaysFalsy) | (dict[str, str] & ~AlwaysFalsy) | ... omitted 3 union elements`
run_agent.py:2593: [invalid-argument-type] invalid-argument-type: Argument to function `_is_oauth_token` is incorrect: Expected `str`, found `(Unknown & ~AlwaysFalsy) | (str & ~AlwaysFalsy) | (dict[str, str] & ~AlwaysFalsy) | ... omitted 4 union elements`
tests/agent/test_codex_cloudflare_headers.py:181: [unsupported-operator] unsupported-operator: Operator `in` is not supported between objects of type `Literal["originator"]` and `(Unknown & ~AlwaysFalsy) | (str & ~AlwaysFalsy) | (dict[str, str] & ~AlwaysFalsy) | ... omitted 3 union elements`
tests/run_agent/test_provider_attribution_headers.py:90: [unresolved-attribute] unresolved-attribute: Attribute `startswith` is not defined on `dict[str, str]` in union `Unknown | str | dict[str, str]`
run_agent.py:13284: [invalid-argument-type] invalid-argument-type: Argument to function `_is_oauth_token` is incorrect: Expected `str`, found `str | dict[Unknown, Unknown] | Any | ... omitted 3 union elements`
run_agent.py:13287: [invalid-argument-type] invalid-argument-type: Argument to function `len` is incorrect: Expected `Sized`, found `(str & ~AlwaysFalsy) | (dict[Unknown, Unknown] & ~AlwaysFalsy) | (Any & ~AlwaysFalsy) | ... omitted 3 union elements`
✅ Fixed issues (9):
| Rule | Count |
|---|---|
invalid-argument-type |
8 |
unresolved-attribute |
1 |
First entries
tests/agent/test_codex_cloudflare_headers.py:163: [unresolved-attribute] unresolved-attribute: Attribute `get` is not defined on `str & ~AlwaysFalsy` in union `(Unknown & ~AlwaysFalsy) | (str & ~AlwaysFalsy) | (dict[str, str] & ~AlwaysFalsy) | dict[Unknown, Unknown] | Divergent`
run_agent.py:2641: [invalid-argument-type] invalid-argument-type: Argument to function `get_model_context_length` is incorrect: Expected `str`, found `str | dict[str, str] | Any | ... omitted 4 union elements`
run_agent.py:2593: [invalid-argument-type] invalid-argument-type: Argument to function `_is_oauth_token` is incorrect: Expected `str`, found `(Unknown & ~AlwaysFalsy) | (str & ~AlwaysFalsy) | (dict[str, str] & ~AlwaysFalsy) | ... omitted 5 union elements`
run_agent.py:2590: [invalid-argument-type] invalid-argument-type: Argument to function `build_anthropic_client` is incorrect: Expected `str`, found `(Unknown & ~AlwaysFalsy) | (str & ~AlwaysFalsy) | (dict[str, str] & ~AlwaysFalsy) | ... omitted 5 union elements`
run_agent.py:7160: [invalid-argument-type] invalid-argument-type: Argument to function `build_anthropic_client` is incorrect: Expected `str`, found `str | dict[Unknown, Unknown] | Any | ... omitted 4 union elements`
run_agent.py:2339: [invalid-argument-type] invalid-argument-type: Argument to function `query_ollama_num_ctx` is incorrect: Expected `str`, found `(str & ~AlwaysFalsy) | (dict[str, str] & ~AlwaysFalsy) | (Any & ~AlwaysFalsy) | ... omitted 5 union elements`
run_agent.py:6989: [invalid-argument-type] invalid-argument-type: Argument to function `_codex_cloudflare_headers` is incorrect: Expected `str`, found `Unknown | str | dict[str, str] | dict[Unknown, Unknown] | Divergent`
run_agent.py:13284: [invalid-argument-type] invalid-argument-type: Argument to function `_is_oauth_token` is incorrect: Expected `str`, found `str | dict[Unknown, Unknown] | Any | ... omitted 4 union elements`
run_agent.py:13287: [invalid-argument-type] invalid-argument-type: Argument to function `len` is incorrect: Expected `Sized`, found `(str & ~AlwaysFalsy) | (dict[Unknown, Unknown] & ~AlwaysFalsy) | (Any & ~AlwaysFalsy) | ... omitted 4 union elements`
Unchanged: 4232 pre-existing issues carried over.
Diagnostics are surfaced as warnings — this check never fails the build.
This was referenced May 10, 2026
19 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kanban_create(and every other entry point that lands inkanban_db.create_task) now rejects toolset names likeweb,browser,terminalin theskillsfield with a clean ValueError, instead of silently accepting them and crashing the worker at spawn time withError: Unknown skill(s): web, browser.Why this layer
The validation lives in
kanban_db.create_task, which is the single chokepoint every entry point hits — CLI (hermes kanban create --skills web), dashboard POST (/api/plugins/kanban/tasks), and the model tool surface (kanban_create(...)). Catching it at the DB layer rather than per-tool means future entry points (a future REST API, a future MCP server, etc.) inherit the validation for free.Why aggregate the typos
Agents that confuse skills with toolsets usually pass several at once (
skills=["web", "browser", "terminal"]) — listing only the first mistake forces serial fix-then-retry; listing all of them lets the caller correct in one round-trip. The error message also names the YAML key (toolsets:) and gives concrete examples of both categories so the conceptual gap that produced the bug to begin with closes:Changes
hermes_cli/kanban_db.py: importget_toolset_namesfromtoolsets.py, build aKNOWN_TOOLSET_NAMESfrozenset (case-folded for friendliness), check each skill against it inside the existingcreate_taskskills loop. Aggregate matches into one error rather than raising on the first.tools/kanban_tools.py: catchValueErrorseparately in_handle_createso the tool surface returns a cleantool_errorpayload instead of the generic exception path.tests/hermes_cli/test_kanban_core_functionality.py: original "single toolset name → reject" test plus a new "multi-toolset names → all listed in error" test pinning the aggregation behavior.tests/plugins/test_kanban_dashboard_plugin.py: dashboard POST level —POST /taskswithskills=["web"]returns 400 with "toolset name" in the detail.Validation
tests/hermes_cli/test_kanban_db.py + test_kanban_core_functionality.pytests/plugins/test_kanban_dashboard_plugin.pytests/tools/test_kanban_tools.py["web", "browser"]with ValueError{"error": "kanban_create: 'terminal' is a toolset name..."}kanban-worker)Closes #22921 via salvage. Salvage of #22933; original commit by @LeonSGP43 preserved as the committing author. The aggregation improvement (collect all toolset typos before raising, expand the error message to name
toolsets:explicitly + give concrete examples) is a separate follow-up commit. AUTHOR_MAP entry already existed.Closes #23105 as superseded — that PR validated at the tool layer only (with a hardcoded toolset list); this PR validates at the DB layer (using
get_toolset_names()as the source of truth) so it covers all entry points without a list-rot maintenance burden.