Skip to content

Optimize SQLite JSON Tag Filtering with Deterministic Bind Names & Cached Templates#2333

Merged
crivetimihai merged 8 commits intomainfrom
perf-issue-1833-optimize-sqlite-tagFilter
Jan 25, 2026
Merged

Optimize SQLite JSON Tag Filtering with Deterministic Bind Names & Cached Templates#2333
crivetimihai merged 8 commits intomainfrom
perf-issue-1833-optimize-sqlite-tagFilter

Conversation

@TS0713
Copy link
Copy Markdown
Collaborator

@TS0713 TS0713 commented Jan 22, 2026

This PR address issue #1833

  • It refactors the SQLite implementation of JSON tag filtering to eliminate random UUID-based bind parameters and replace them with deterministic positional bind names (:p0, :p1, …). It also introduces LRU-cached SQL templates to avoid rebuilding SQL strings on every request.
  • Extended list resource templates in main API by adding support for:
    ✅ Tag-based filtering
    ✅ Visibility filtering
    ✅ Include-inactive filtering

@TS0713 TS0713 marked this pull request as ready for review January 23, 2026 08:47
@TS0713 TS0713 requested a review from crivetimihai as a code owner January 23, 2026 08:47
@TS0713 TS0713 marked this pull request as draft January 23, 2026 08:48
@TS0713 TS0713 marked this pull request as ready for review January 23, 2026 09:37
@crivetimihai crivetimihai added this to the Release 1.0.0-RC1 milestone Jan 24, 2026
TS0713 and others added 8 commits January 25, 2026 02:55
…d templates

Signed-off-by: Satya <tsp.0713@gmail.com>
…is(non-template)

Signed-off-by: Satya <tsp.0713@gmail.com>
…resource services

Signed-off-by: Satya <tsp.0713@gmail.com>
Remove debugging print statements that were accidentally left in the
tag filtering code path. These were outputting query details to stdout
which is not appropriate for production code.

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
When multiple json_contains_tag_expr calls are combined in the same
query (e.g., filtering on tags from different columns), the fixed
bind names (:p0, :p1) would collide and overwrite parameters.

This fix adds column-specific prefixes to bind parameter names
(e.g., :tools_tags_p0, :resources_tags_p0) to ensure uniqueness
when composing multiple tag filter predicates.

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
…ilters

Add comprehensive tests for:
- _sanitize_col_prefix helper function
- json_contains_tag_expr for SQLite with match_any and match_all
- Bind parameter collision prevention when combining multiple tag filters
- LRU caching of SQL templates
- New list_resource_templates filtering parameters (tags, visibility,
  include_inactive)

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
Address edge cases where bind parameters could still collide:
1. Same column filtered multiple times in one query
2. Different column refs that sanitize to identical strings
   (e.g., "a_b.c" and "a.b_c" both become "a_b_c")

Replace static column-based prefix with a thread-safe counter that
generates truly unique prefixes per call (e.g., "tools_tags_42_p0").

This removes the LRU caching of templates since each call now has
a unique prefix, but ensures correctness in all edge cases.

Add test for same-column collision scenario.

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
@crivetimihai crivetimihai force-pushed the perf-issue-1833-optimize-sqlite-tagFilter branch from 4e2824a to fa3a5c7 Compare January 25, 2026 03:40
@crivetimihai crivetimihai merged commit e21eac0 into main Jan 25, 2026
53 checks passed
@crivetimihai crivetimihai deleted the perf-issue-1833-optimize-sqlite-tagFilter branch January 25, 2026 03:51
kcostell06 pushed a commit to kcostell06/mcp-context-forge that referenced this pull request Feb 24, 2026
…ched Templates (IBM#2333)

* Optimize SQLite JSON tag filtering with deterministic binds and cached templates

Signed-off-by: Satya <tsp.0713@gmail.com>

* feat: add tag filtering support to list resources template in main apis(non-template)

Signed-off-by: Satya <tsp.0713@gmail.com>

* removed unused fields - page, limit from list resource template from resource services

Signed-off-by: Satya <tsp.0713@gmail.com>

* fix: remove debug print statements from tool_service.py

Remove debugging print statements that were accidentally left in the
tag filtering code path. These were outputting query details to stdout
which is not appropriate for production code.

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

* Lint

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

* fix: use column-specific bind prefixes to prevent parameter collision

When multiple json_contains_tag_expr calls are combined in the same
query (e.g., filtering on tags from different columns), the fixed
bind names (:p0, :p1) would collide and overwrite parameters.

This fix adds column-specific prefixes to bind parameter names
(e.g., :tools_tags_p0, :resources_tags_p0) to ensure uniqueness
when composing multiple tag filter predicates.

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

* test: add coverage for json_contains_tag_expr and resource template filters

Add comprehensive tests for:
- _sanitize_col_prefix helper function
- json_contains_tag_expr for SQLite with match_any and match_all
- Bind parameter collision prevention when combining multiple tag filters
- LRU caching of SQL templates
- New list_resource_templates filtering parameters (tags, visibility,
  include_inactive)

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

* fix: use thread-safe counter for fully unique bind prefixes

Address edge cases where bind parameters could still collide:
1. Same column filtered multiple times in one query
2. Different column refs that sanitize to identical strings
   (e.g., "a_b.c" and "a.b_c" both become "a_b_c")

Replace static column-based prefix with a thread-safe counter that
generates truly unique prefixes per call (e.g., "tools_tags_42_p0").

This removes the LRU caching of templates since each call now has
a unique prefix, but ensures correctness in all edge cases.

Add test for same-column collision scenario.

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

---------

Signed-off-by: Satya <tsp.0713@gmail.com>
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants