register_token_handler() plugin hook for custom API token backends by simonw · Pull Request #2650 · simonw/datasette

simonw · 2026-02-25T22:06:25Z

Initial prompt was the description and first comment from:

Plugin hook to allow issuing of API tokens #2649

Summary

This PR introduces a pluggable token handler system that allows plugins to provide custom token backends while maintaining backward compatibility with the existing signed token implementation.

Key Changes

New TokenHandler base class (datasette/tokens.py): Abstract base class that plugins can subclass to implement custom token creation and verification logic
SignedTokenHandler implementation: Moved existing signed token logic into a dedicated handler class that implements the TokenHandler interface
Plugin hook register_token_handler: New hookspec in datasette/hookspecs.py allowing plugins to register custom token handlers
Async token methods: Changed create_token() to async and added new verify_token() async method to Datasette class
Token handler discovery: Added _token_handlers() method to collect all registered handlers from plugins
Handler selection: create_token() now accepts optional handler parameter to select a specific handler by name; defaults to first registered handler
Multi-handler verification: verify_token() tries all registered handlers until one recognizes the token
Simplified token authentication: Refactored datasette/default_permissions/tokens.py to delegate to the new handler system instead of duplicating logic
CLI update: Updated datasette create-token command to use the new async create_token() method
HTTP integration: Token verification through HTTP Bearer authentication now works with all registered handlers
Comprehensive test suite: Added 258 lines of tests covering default handler, custom handlers, verification, HTTP integration, and error cases

Implementation Details

The SignedTokenHandler is automatically registered via the default permissions plugin
Token handlers are tried in order during verification, allowing multiple backends to coexist
The system maintains full backward compatibility—existing signed tokens continue to work
Custom handlers can implement simple string-based tokens or complex database-backed systems
All token handler methods are async to support I/O operations (e.g., database lookups)

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

📚 Documentation preview 📚: https://datasette--2650.org.readthedocs.build/en/2650/

Adds a new register_token_handler hook that allows plugins to provide custom token creation and verification backends. This enables plugins like datasette-oauth to issue tokens without depending on specific backend plugins like datasette-auth-tokens. Key changes: - New datasette/tokens.py with TokenHandler base class and SignedTokenHandler (the default signed-token implementation moved here) - New register_token_handler hookspec in hookspecs.py - Datasette.create_token() is now async and delegates to token handlers - New Datasette.verify_token() method tries all handlers in sequence - handler= parameter on create_token() to select a specific backend - TokenHandler exported from datasette package for plugin use - Fixed actor_from_request loop to await all coroutines (avoids warnings) https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

Fixes CI failures: the new hook needs a section in docs/plugin_hooks.rst (checked by test_plugin_hooks_are_documented) and a test_hook_* function in test_plugins.py (checked by test_plugin_hooks_have_tests). https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

Instead of re-exporting hookimpls from default_permissions/__init__.py, register datasette.default_permissions.tokens as its own DEFAULT_PLUGINS entry. Cleaner and avoids confusing import-for-side-effect patterns. https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

Consolidates the three separate restrict_all, restrict_database, and restrict_resource parameters into a single TokenRestrictions dataclass. Cleaner API surface for both Datasette.create_token() and TokenHandler.create_token(). Also clarifies docs re: default handler selection via pluggy ordering. https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

simonw · 2026-02-25T22:24:27Z

datasette/tokens.py

+        restrict_all: Optional[Iterable[str]] = None,
+        restrict_database: Optional[Dict[str, Iterable[str]]] = None,
+        restrict_resource: Optional[Dict[str, Dict[str, Iterable[str]]]] = None,


Let's do better than this - a more type-friendly API.

simonw · 2026-02-25T22:37:46Z

datasette/tokens.py

+@dataclasses.dataclass
+class TokenRestrictions:
+    """
+    Restrictions to apply to a token, limiting which actions it can perform.
+
+    - all: actions allowed against all databases/resources
+    - database: {database_name: [actions]} for database-level restrictions
+    - resource: {database_name: {resource_name: [actions]}} for resource-level
+    """
+
+    all: list[str] = dataclasses.field(default_factory=list)
+    database: dict[str, list[str]] = dataclasses.field(default_factory=dict)
+    resource: dict[str, dict[str, list[str]]] = dataclasses.field(
+        default_factory=dict
+    )


@dataclasses.dataclass class TokenRestrictions: """ Restrictions to apply to a token, limiting which actions it can perform. - all: actions allowed against all databases/resources - database: {database_name: [actions]} for database-level restrictions - resource: {database_name: {resource_name: [actions]}} for resource-level """ all: list[str] = dataclasses.field(default_factory=list) database: dict[str, list[str]] = dataclasses.field(default_factory=dict) resource: dict[str, dict[str, list[str]]] = dataclasses.field( default_factory=dict )

Don't write code until I say, but propose a design for this that uses more of a builder pattern and hence is less dictionary-heavy in terms of how it is interacted with

Claude suggested:

(TokenRestrictions() .allow("view-instance") .allow("view-database") .allow("create-table", database="mydb") .allow("insert-row", database="mydb", resource="mytable"))

I countered:

no lets do three methods for each category of restriction

Adds allow_all(), allow_database(), and allow_resource() methods that return self for chaining. Callers no longer need to manipulate nested dicts directly: restrictions = (TokenRestrictions() .allow_all("view-instance") .allow_database("mydb", "create-table") .allow_resource("mydb", "mytable", "insert-row")) https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

codecov · 2026-02-25T22:52:37Z

Codecov Report

❌ Patch coverage is 93.43066% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.42%. Comparing base (6a2c27b) to head (0345b7a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
datasette/app.py	90.62%	3 Missing ⚠️
datasette/cli.py	57.14%	3 Missing ⚠️
datasette/tokens.py	96.38%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2650      +/-   ##
==========================================
- Coverage   90.44%   90.42%   -0.02%     
==========================================
  Files          52       53       +1     
  Lines        8018     8075      +57     
==========================================
+ Hits         7252     7302      +50     
- Misses        766      773       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…hange Ref: #2649 (comment) https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

- internals.rst: new async create_token() signature with restrictions and handler params, add TokenRestrictions reference docs - plugin_hooks.rst: show full create_token signature in TokenHandler example, note list returns and error cases - authentication.rst: cross-reference TokenRestrictions from the restrictions section https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

…dler Covers allow_database/allow_resource builders, _r payload encoding, and token_expires in verified actors. Coverage 76% -> 90%. https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

simonw · 2026-02-25T23:36:16Z

I tested this breaking change in a branch of datasette-auth-tokens - I had Claude look at the upgrade guide.

Clone all of simonw/datasette to /tmp and checkout the claude/api-token-plugin-hook-jaSXV branch in that repo, then run the test suite for this (using uv run pytest) but with that /tmp/datasette folder set to be used instead of the regular Datasette depenedency.

In the Datasette repo read docs/upgrade_guide.md and tell me what you are going to change as a result

https://claude.ai/code/session_01RbZveeSogxSnjJ5yr6AFtq

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

simonw · 2026-02-25T23:51:29Z

tests/test_plugins.py

+@pytest.mark.asyncio
+async def test_hook_register_token_handler(ds_client):
+    handlers = ds_client.ds._token_handlers()
+    assert len(handlers) >= 1
+    # Default signed handler should be present
+    assert any(h.name == "signed" for h in handlers)
+    # Should be able to create and verify a token
+    token = await ds_client.ds.create_token("test")
+    assert token.startswith("dstok_")
+    actor = await ds_client.ds.verify_token(token)
+    assert actor["id"] == "test"
+


I don't like the test for test_hook_register_token_handler - the other tests in that module act against a real plugin that has been registered, including testing that it is used. I think that test should use a special handler that creates a token with e.g. dstok_hardcoded_token_1 which we can then create (as the default plugin handler and also by specifying its name) and then validate

Adds a HardcodedTokenHandler to the test plugins dir that creates tokens like dstok_hardcoded_token_1. The test now exercises creating tokens via the default handler (which is the plugin's hardcoded one), by explicitly naming the hardcoded handler, and by explicitly naming the signed handler -- then verifies each token round-trips correctly. https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

…andler https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

simonw · 2026-02-26T00:07:32Z

Finally got it to run the full test suite and it spotted:

The problem is much bigger than test_token_handler.py. The hardcoded handler gets globally registered and becomes the default for ALL token creation — including the /-/create-token web view and create-token CLI, which then try to unsign the hardcoded token and crash.

The fix: the /-/create-token view and CLI are specifically for signed tokens, so they should use handler="signed". Let me find and fix all the affected code.

Fix was pretty simple:

The HardcodedTokenHandler in my_plugin.py gets globally registered, so create_token() without a handler name picks it up as the default. Fix the create-token view, CLI, and tests to explicitly request the signed handler where they depend on signed token behavior. https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

Refs #2636, #2641, #2646, #2647, #2650

Refs #38 (comment) Refs simonw/datasette#2650 (comment) Refs simonw/datasette#2649

* Add build/ and dist/ to .gitignore These directories are generated by build tools (uv, pip, setuptools) and should not be tracked. * Use TokenRestrictions API and await async create_token Adapt to Datasette's new token API: build a TokenRestrictions object using builder methods instead of separate restrict_all/restrict_database/ restrict_resource dicts, and await the now-async create_token() call. * datasette>=1.0a25 Refs #38 (comment) Refs simonw/datasette#2650 (comment) Refs simonw/datasette#2649

simonw marked this pull request as draft February 25, 2026 22:06

claude added 3 commits February 25, 2026 22:14

simonw commented Feb 25, 2026

View reviewed changes

simonw linked an issue Feb 25, 2026 that may be closed by this pull request

Plugin hook to allow issuing of API tokens #2649

Closed

simonw mentioned this pull request Feb 25, 2026

Plugin hook to allow issuing of API tokens #2649

Closed

claude added 8 commits February 25, 2026 23:02

docs: add 1.0a25 upgrade guide section for create_token() signature c…

03273d9

…hange Ref: #2649 (comment) https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

docs: note that create_token() is now async in upgrade guide

56d4fe3

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

style: apply black formatting to token handler files

0980c81

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

docs: fix RST heading underline length in internals.rst

aa7ca68

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

tests: add restrictions round-trip and expiration tests for token han…

307ec0b

…dler Covers allow_database/allow_resource builders, _r payload encoding, and token_expires in verified actors. Coverage 76% -> 90%. https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

tests: add test for signed tokens disabled

d5b817d

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

fix: add TokenRestrictions TYPE_CHECKING import to fix ruff F821

cdc3d93

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

simonw changed the title ~~Add pluggable token handler system for custom token backends~~ register_token_handler() plugin hook for custom API token backends Feb 25, 2026

docs: regenerate plugins.rst with cog

56993e2

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

simonw mentioned this pull request Feb 25, 2026

Use TokenRestrictions API for token creation simonw/datasette-auth-tokens#38

Merged

docs: reformat code blocks in plugin_hooks.rst with blacken-docs

cdcb259

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

simonw added plugins breaking-change labels Feb 25, 2026

docs: add await .verify_token() to internals.rst

b0ff5ae

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

simonw commented Feb 25, 2026

View reviewed changes

claude added 2 commits February 25, 2026 23:53

tests: clarify test_token_handler_via_http tests the default signed h…

ed252fa

…andler https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

simonw marked this pull request as ready for review February 26, 2026 00:12

claude added 2 commits February 26, 2026 00:20

fix: use handler="signed" in test_create_table_permissions

0345b7a

https://claude.ai/code/session_013cQFiDQjYRrRBH2biFfKuS

simonw merged commit c96dc5c into main Feb 26, 2026
39 checks passed

simonw deleted the claude/api-token-plugin-hook-jaSXV branch February 26, 2026 00:32

simonw added a commit that referenced this pull request Feb 26, 2026

Release 1.0a25

1246c65

Refs #2636, #2641, #2646, #2647, #2650

simonw added a commit to simonw/datasette-auth-tokens that referenced this pull request Feb 26, 2026

datasette>=1.0a25

c9eba1a

Refs #38 (comment) Refs simonw/datasette#2650 (comment) Refs simonw/datasette#2649

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

register_token_handler() plugin hook for custom API token backends#2650

register_token_handler() plugin hook for custom API token backends#2650
simonw merged 20 commits intomainfrom
claude/api-token-plugin-hook-jaSXV

simonw commented Feb 25, 2026 •

edited

Loading

Uh oh!

simonw Feb 25, 2026

Uh oh!

simonw Feb 25, 2026

Uh oh!

simonw Feb 25, 2026

Uh oh!

codecov bot commented Feb 25, 2026 •

edited

Loading

Uh oh!

simonw commented Feb 25, 2026 •

edited

Loading

Uh oh!

simonw Feb 25, 2026

Uh oh!

simonw commented Feb 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

simonw commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

Implementation Details

Uh oh!

simonw Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

simonw Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

simonw Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

simonw commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonw Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

simonw commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

simonw commented Feb 25, 2026 •

edited

Loading

codecov bot commented Feb 25, 2026 •

edited

Loading

simonw commented Feb 25, 2026 •

edited

Loading

simonw commented Feb 26, 2026 •

edited

Loading