feat: add database-backed verified models for dynamic model managemen… by bittoby · Pull Request #12833 · OpenHands/OpenHands

bittoby · 2026-02-11T12:17:48Z

Summary

Store the list of supported models in a database (when used on SaaS) so it can be updated without cutting a release.

Problem

Adding new models to the OpenHands Cloud model selector requires a code release, which is slow and heavyweight for what should be a simple configuration change.

Solution

In SaaS mode, load verified models from a verified_models database table instead of the hardcoded list. Self-hosted deployments are unaffected and continue using the hardcoded model arrays.

Changes

New files (enterprise):

storage/verified_model.py — SQLAlchemy model with composite unique key (model_name, provider)
storage/verified_model_store.py — static-method store for CRUD operations (follows UserStore/OrgMemberStore conventions)
server/routes/verified_models.py — admin CRUD API at /api/admin/verified-models
migrations/versions/097_create_verified_models_table.py — creates table and seeds current 11 openhands provider models
tests/unit/storage/test_verified_model_store.py — 14 unit tests

Modified files (core):

openhands/utils/llm.py — get_supported_llm_models() accepts optional verified_models list that replaces hardcoded models when provided
openhands/server/routes/public.py — /api/options/models endpoint loads models from DB in SaaS mode, falls back to hardcoded list on error or in self-hosted mode
enterprise/saas_server.py — registers the admin router
enterprise/tests/unit/conftest.py — imports VerifiedModel so shared test DB includes the table

Change Type

Checklist

I have read and reviewed the code and I understand what the code is doing.
I have tested the code to the best of my ability and ensured it works as expected.

Fixes

Resolves #12807

Release Notes

Include this change in the Release Notes.

Added database-backed verified models for SaaS deployments, allowing administrators to dynamically manage supported LLM models without code deployments.

…t in SaaS without code deployments

all-hands-bot

Summary: This PR introduces useful functionality for dynamic model management, but has critical security and data integrity issues that must be addressed before merging.

Key Issues:

🔴 Critical: No admin authorization on admin API endpoints
🔴 Critical: Model names contain unrealistic future dates
🟠 Important: Type safety issues with conditional imports
🟠 Important: Missing input validation and transaction handling

See inline comments for details.

all-hands-bot

Solid implementation overall! The database-backed model management is well-architected with good test coverage. I found several important issues around session management, consistency, and error handling that should be addressed before merging.

…/database-backed-verified-models

…nd unique constraints, input validation, and proper timestamp handling

…mpliance" This reverts commit 636fc83.

bittoby · 2026-02-11T16:37:24Z

@neubig Could you please review this PR?
And regarding CI is failing:
The Migrations test error looks like an auth/permission issue.
I can’t figure out the lint error even after a few tries, but the code works in my testing.
Could you take a quick look and let me know what I should fix? Thanks.

neubig · 2026-02-11T23:41:36Z

@OpenHands: check if the review comments have been reflected, and if so mark them as "resolved" using graphql. If not, note them here.

openhands-ai · 2026-02-11T23:41:45Z

I'm on it! neubig can track my progress at all-hands.dev

openhands-ai · 2026-02-11T23:45:34Z

I reviewed all 19 review comment threads on PR #12833 and verified that all of them have been addressed in the current code. I then marked all threads as "resolved" using the GitHub GraphQL API.

Summary of Review Comments and Their Resolution Status

Critical Issues (All Addressed ✅)

Admin authorization missing - Now uses get_admin_user_id instead of get_user_id
Future dates in model names - Migration now uses actual existing model versions (e.g., claude-3-5-sonnet-20241022, gpt-4o)
Migration unique constraint failure - Changed to compound constraint on (model_name, provider)
onupdate doesn't work with text() - Now uses func.now() for both server_default and onupdate

Important Issues (All Addressed ✅)

Missing input validation - Added _validate_model_input() function with regex validation and length limits
Provider field not being used - Now constructs {provider}/{model_name} format correctly
Type safety issue - Uses simpler try/except import pattern
Transaction handling - Added explicit try/except with rollback in bulk_create_models
Detached SQLAlchemy instances - Added documentation and session.expunge_all()
Bulk create all-or-nothing behavior - Documented in docstring
No upper bound on limit parameter - Added validation (1-1000 range)
Bare except Exception - Now uses logger.exception() for full traceback
Inconsistent behavior between SaaS/self-hosted - Clarifai models extracted to _get_clarifai_models() function, used by both paths

Suggestions (All Addressed ✅)

Duplicated response model building - Extracted to _model_to_response() helper function
Add pagination - Added limit and offset parameters with validation
Redundant index on model_name - Removed (unique constraint creates index automatically)
Stricter regex pattern - Now uses ^[a-zA-Z0-9]([a-zA-Z0-9_.-]*[a-zA-Z0-9])?$
TYPE_CHECKING import unnecessarily complex - Uses simpler try/except pattern
Tests only cover store layer - This was a suggestion for additional tests (not blocking)

All 19 review threads have been marked as resolved via the GitHub GraphQL API. No code changes were needed as all comments had already been addressed in the current branch.

View full conversation

bittoby · 2026-02-12T19:06:24Z

@neubig @hieptl Could you please review the PR? I'd appreciate your feedback.

neubig · 2026-02-14T01:52:26Z

Hi @bittoby , will do, I'll take a look

…/database-backed-verified-models

bittoby · 2026-02-18T01:20:28Z

@neubig @hieptl Sorry for tagging again. I'd appreciate your feedback.

neubig · 2026-02-19T15:57:10Z

I think it'd probably be better for someone more familiar with the code than me to take a look at this one!

bittoby · 2026-02-19T16:01:31Z

@enyst @hieptl I'd appreciate your feedback about this PR.
Thank you

…/database-backed-verified-models

…models can be updated without cutting a release

bittoby · 2026-02-21T11:50:07Z

@tofarr @malhotra5 Sorry for tagging several people. If possible, could you please review this PR? I’d really appreciate your feedback.
thank you

bittoby · 2026-02-23T18:13:51Z

@hieptl please review the PR. thanks

bittoby · 2026-02-25T21:08:00Z

@enyst @hieptl Sorry for tagging again. would appreciate to let me know feedback
thank you

tofarr

Great work! One nit mentioned - it's more worry about maintaining established patterns than this PR.

…/database-backed-verified-models

bittoby · 2026-02-26T12:22:33Z

@tofarr Thanks for your feedback. I updated. could you please review again when you get a time?
thank you

tofarr

🍰 - I'll create a follow up PR for database level filtering

Add check for whether models are available in the SaaS verified_models database (app.all-hands.dev). This is separate from frontend code support since SaaS uses a database-backed system since PR OpenHands/OpenHands#12833. Changes: - Add check_saas_verified_model() function that queries the live API - Add frontend_saas_available boolean field to track output - Update all_models.json with the new field - Add tests for the new functionality This helps identify models like claude-opus-4-6 that have frontend code support but are not yet in the SaaS dropdown because they haven't been added to the verified_models database. Co-authored-by: openhands <openhands@all-hands.dev>

…ity (#39) * feat: add frontend_saas_available field to track SaaS model availability Add check for whether models are available in the SaaS verified_models database (app.all-hands.dev). This is separate from frontend code support since SaaS uses a database-backed system since PR OpenHands/OpenHands#12833. Changes: - Add check_saas_verified_model() function that queries the live API - Add frontend_saas_available boolean field to track output - Update all_models.json with the new field - Add tests for the new functionality This helps identify models like claude-opus-4-6 that have frontend code support but are not yet in the SaaS dropdown because they haven't been added to the verified_models database. Co-authored-by: openhands <openhands@all-hands.dev> * refactor: frontend_support_timestamp requires both code AND SaaS availability Change frontend_support_timestamp to only be set when a model is in BOTH: 1. verified-models.ts (for self-hosted deployments) 2. SaaS verified_models database (for app.all-hands.dev) This removes the separate frontend_saas_available field and makes the logic simpler - frontend_support_timestamp represents full frontend availability for all users. Co-authored-by: openhands <openhands@all-hands.dev> * fix: correctly mock subprocess calls in test_search_index_results_success The function calls subprocess.run twice (git log + git show), so the test needs to mock both calls using side_effect. Co-authored-by: openhands <openhands@all-hands.dev> * fix: add frontend_saas_available field to result dictionary Co-authored-by: openhands <openhands@all-hands.dev> * chore: regenerate all_models.json with updated logic Models without SaaS availability now correctly show null frontend_support_timestamp. Co-authored-by: openhands <openhands@all-hands.dev> * docs: add SaaS Admin API documentation to add-new-model skill Document the verified_models Admin API endpoints for adding models to the SaaS database (app.all-hands.dev). This is required for models to appear in the frontend dropdown on the hosted platform. Includes: - POST /api/admin/verified-models (create) - GET /api/admin/verified-models (list) - PUT /api/admin/verified-models/{provider}/{model_name} (update) - DELETE /api/admin/verified-models/{provider}/{model_name} (delete) - Authentication requirements (@all-hands.dev email domain) - Explanation of frontend_saas_available field Co-authored-by: openhands <openhands@all-hands.dev> * Change 'SaaS Database' to 'Cloud Database' in SKILL.md Updated references from 'SaaS Database' to 'Cloud Database' in SKILL.md. --------- Co-authored-by: openhands <openhands@all-hands.dev>

feat: add database-backed verified models for dynamic model managemen…

50c2327

…t in SaaS without code deployments

bittoby requested review from chuckbutkus, hieptl, malhotra5 and tofarr as code owners February 11, 2026 12:17

all-hands-bot reviewed Feb 11, 2026

View reviewed changes

bittoby marked this pull request as draft February 11, 2026 12:33

bittoby added 2 commits February 11, 2026 13:26

fix: update authorization and model name

f496db3

fix: fix CI test error

aab9004

bittoby marked this pull request as ready for review February 11, 2026 14:28

all-hands-bot reviewed Feb 11, 2026

View reviewed changes

bittoby added 6 commits February 11, 2026 15:48

Merge branch 'main' of https://github.com/bittoby/OpenHands into feat…

c2d75c8

…/database-backed-verified-models

refactor: add database-backed verified models with admin auth, compou…

7d104a4

…nd unique constraints, input validation, and proper timestamp handling

Apply ruff formatting fixes to enterprise directory for CI compliance

636fc83

Revert "Apply ruff formatting fixes to enterprise directory for CI co…

4eb3e35

…mpliance" This reverts commit 636fc83.

Fix quote style formatting in verified models files

595d6e1

Fix code formatting to use single quotes consistently

1636f2b

Merge branch 'main' of https://github.com/bittoby/OpenHands into feat…

6da3555

…/database-backed-verified-models

bittoby force-pushed the feat/database-backed-verified-models branch from 16d149e to 6da3555 Compare February 18, 2026 01:19

fix: solve CI lint error

7c2fe35

enyst requested a review from neubig February 18, 2026 23:48

bittoby added 2 commits February 19, 2026 12:41

fix: solve merge conflict

4030c21

fix: solve lint error

1dedafc

neubig removed their request for review February 19, 2026 15:57

bittoby added 4 commits February 20, 2026 13:45

Merge branch 'main' of https://github.com/bittoby/OpenHands into feat…

aa24047

…/database-backed-verified-models

Merge branch 'main' of https://github.com/bittoby/OpenHands into feat…

18fde92

…/database-backed-verified-models

feat: add database-backed verified models for SaaS mode so supported …

d6e7c15

…models can be updated without cutting a release

fix: solve Lint error

68eab25

tofarr reviewed Feb 26, 2026

View reviewed changes

Comment thread enterprise/server/routes/verified_models.py

tofarr requested changes Feb 26, 2026

View reviewed changes

statxc and others added 2 commits February 26, 2026 12:11

Merge branch 'main' of https://github.com/bittoby/OpenHands into feat…

7a9e5b1

…/database-backed-verified-models

fix: add paginated response for verified models API

add68a6

bittoby force-pushed the feat/database-backed-verified-models branch from 35a3ab4 to add68a6 Compare February 26, 2026 12:15

bittoby requested a review from tofarr February 26, 2026 12:21

tofarr mentioned this pull request Feb 26, 2026

refactor: use SQL filtering and pagination in VerifiedModelStore #13068

Merged

3 tasks

tofarr approved these changes Feb 26, 2026

View reviewed changes

tofarr merged commit a92bfe6 into OpenHands:main Feb 26, 2026
16 of 17 checks passed

bittoby deleted the feat/database-backed-verified-models branch February 26, 2026 13:18

neubig mentioned this pull request Mar 7, 2026

feat: add frontend_saas_available field to track SaaS model availability OpenHands/llm-support-tracker#39

Merged

Conversation

bittoby commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Solution

Changes

Change Type

Checklist

Fixes

Release Notes

Uh oh!

all-hands-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

all-hands-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bittoby commented Feb 11, 2026

Uh oh!

neubig commented Feb 11, 2026

Uh oh!

openhands-ai Bot commented Feb 11, 2026

Uh oh!

openhands-ai Bot commented Feb 11, 2026

Summary of Review Comments and Their Resolution Status

Critical Issues (All Addressed ✅)

Important Issues (All Addressed ✅)

Suggestions (All Addressed ✅)

Uh oh!

bittoby commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neubig commented Feb 14, 2026

Uh oh!

bittoby commented Feb 18, 2026

Uh oh!

neubig commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bittoby commented Feb 19, 2026

Uh oh!

bittoby commented Feb 21, 2026

Uh oh!

bittoby commented Feb 23, 2026

Uh oh!

bittoby commented Feb 25, 2026

Uh oh!

Uh oh!

tofarr left a comment

Choose a reason for hiding this comment

Uh oh!

bittoby commented Feb 26, 2026

Uh oh!

tofarr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

bittoby commented Feb 11, 2026 •

edited

Loading

bittoby commented Feb 12, 2026 •

edited

Loading

neubig commented Feb 19, 2026 •

edited

Loading