Skip to content

Search Optimization Rewrite #97

Merged
tarekio merged 105 commits intomainfrom
search-v4
Aug 3, 2025
Merged

Search Optimization Rewrite #97
tarekio merged 105 commits intomainfrom
search-v4

Conversation

@level09
Copy link
Collaborator

@level09 level09 commented Jun 1, 2025

Implement Cursor-Based Pagination for Bulletins with Performance Enhancements

Overview

Replaces traditional offset-based pagination with cursor-based infinite scroll for bulletins, significantly improving performance for large datasets.

Key Changes

  • Cursor-based pagination: Uses id < cursor for efficient pagination, eliminating expensive COUNT queries
  • Infinite scroll interface: Smooth browsing experience with intersection observer for automatic loading
  • Search enhancements: Added origin ID text input and fixed "Does not Include" field bug (BYNT-1303)
  • Performance indexes: Added foreign key indexes for search optimization across Bulletins, Actors, and Incidents
  • SQLAlchemy 2.x modernization: Updated to modern ORM patterns with single optimized CTE-based queries

Technical Improvements

  • Single query approach replacing multiple union/intersect queries
  • Minimal serialization with essential-only fields
  • Optional total count feature with checkbox toggle
  • Enhanced error handling and user feedback
  • Updated to Vuetify-labs 3.7.6 with virtual data tables

Impact

  • Faster loading times for large datasets
  • Improved user experience with smooth scrolling
  • Better search performance with database indexes
  • Maintains full backward compatibility

Ready for review and testing with production-scale bulletin datasets.

level09 and others added 30 commits January 3, 2025 23:03
- Updated all instances of direct model queries (e.g., Label, Source, Bulletin, Actor, Incident, Location) to utilize db.session.query for improved session management.
This commit introduces a new file `pagination.py` that implements a flexible pagination system for SQLAlchemy queries. It supports both cursor-based and offset-based pagination, allowing for efficient data retrieval. The `PaginationResult` class is defined to encapsulate the results, including items, total count, next cursor, and items per page. The `paginate_query` function handles the logic for pagination based on the provided parameters.
This update simplifies the pagination implementation by removing unnecessary code related to cursor-based pagination and consolidating the logic for both cursor and offset pagination.
…nation prototype):

- Simplified pagination in the `api_bulletins` function by removing the page parameter and utilizing cursor-based pagination exclusively.
- Enhanced the response structure to directly return pagination results using the new `to_dict` method from `PaginationResult`.
- Updated the frontend to handle pagination more effectively, including a new computed property for item length and improved loading behavior.
- Improved error handling and loading state management during data fetching.

This update streamlines the API and enhances the user experience with more efficient data retrieval.
Unified query conditions into a single list for clarity.
Streamlined parameter handling for consistent filters.
Simplified return structure, ensuring compatibility.
Enhances search functionality and supports future updates.
…y handling

- Enhanced the `api_bulletins` function to streamline query execution and pagination logic, utilizing cursor-based pagination exclusively.
- Updated the response structure to include total count, pagination details, and items in a more efficient format.
- Refactored the `SearchUtils` class to simplify nested query handling, allowing for clearer and more maintainable query construction.
- Improved overall performance and clarity in the bulletin search functionality, setting the stage for future enhancements.
…rameters and improved clarity in pagination metadata handling.
…earch

- Enhanced nested queries handling:  combine conditions from multiple queries, allowing for clearer and more maintainable query construction.
- Streamlined the handling of logical operations (`and`/`or`) when processing nested queries.
…model for search-critical foreign keys (assigned_to_id, first_peer_reviewer_id, second_peer_reviewer_id) - Add indexes to Event model for search-critical foreign keys (location_id, eventtype_id) - Create migration script with production-safe CONCURRENTLY indexes - Include junction table indexes for many-to-many search relationships - Based on BulletinSearchBox.js search pattern analysis
…or model for search-critical foreign keys (assigned_to_id, first_peer_reviewer_id, second_peer_reviewer_id, origin_place_id) - Create migration script with production-safe CONCURRENTLY indexes - Include junction table indexes for actor many-to-many search relationships - Based on ActorSearchBox.js search pattern analysis
…Incident model for search-critical foreign keys (assigned_to_id, first_peer_reviewer_id, second_peer_reviewer_id) - Create migration script with production-safe CONCURRENTLY indexes - Include junction table indexes for incident many-to-many search relationships - Based on IncidentSearchBox.js search pattern analysis
cango91 and others added 25 commits July 10, 2025 19:12
## Jira Issue
1. [BYNT-1381](https://syriajustice.atlassian.net/browse/BYNT-1381)

## Description
Implements comprehensive migration for Actors and Incidents, covering
SearchUtils refactoring from (queries, ops) tuple to unified query
objects, cursor-based pagination with dual query strategy, complete
database index audit (all 50 indexes introduced in branch `search-v4`
migrations are also defined on ORM-level), and minimal serialization
patterns

## Checklist
- [x] Tests added/updated
- [x] Documentation updated (if needed)
- [ ] New strings prepared for translations

## API Changes (if applicable)
- [ ] Permissions checked
- [ ] Endpoint tests added

## Additional Notes
[Any other relevant information]


[BYNT-1381]:
https://syriajustice.atlassian.net/browse/BYNT-1381?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: tarekio <7659394+tarekio@users.noreply.github.com>
Add permission checks to search endpoints before returning item data.
Users without access now get restricted_json() instead of full details.

Fixes issue where users could see titles, assigned users, and status
of restricted bulletins, actors, and incidents in search results.
Replace NOT ILIKE ALL() with individual indexed searches to resolve
21-42 second timeouts. Uses subquery approach enabling GIN trigram
index usage for better performance.
Use direct NOT ILIKE conditions instead of NOT IN subquery
to avoid correlated subquery performance issues. Generates
WHERE NOT (search ILIKE '%term1%') AND NOT (search ILIKE '%term2%')
instead of WHERE id NOT IN (SELECT id FROM ... OR ...).
…cessary direct NOT conditions and optimized the exclude logic for better performance.
Replace slow NOT IN subquery pattern with direct NOT ILIKE ALL
array syntax to leverage GIN trigram index efficiently.

Performance improvement: 34 seconds -> ~400ms for complex exclusions
Use fast array containment (@>) for exact tag matches (0.06ms vs 1.8s)
while preserving ILIKE wildcards for partial searches when needed.

Performance improvement: 30,000x faster for exact tag searches
- Exact Match checked: Use fast array containment (@>) for 9533x performance improvement
- Exact Match unchecked: Use ILIKE for partial/wildcard matching behavior
- Preserve Any checkbox functionality for OR vs AND logic

Performance: 572ms -> 0.06ms for exact tag searches
@tarekio tarekio merged commit 1ce18f7 into main Aug 3, 2025
6 of 7 checks passed
@tarekio tarekio deleted the search-v4 branch August 3, 2025 18:09
tarekio pushed a commit that referenced this pull request Aug 25, 2025
Due to recent changes in #97 an update was made to the latest vuetify
version that included [some
changes](https://vuetifyjs.com/en/getting-started/release-notes/?version=v3.7.4),
that caused the dialog to ignore custom sizes when the prop fullscreen
was used. This update changes the width through the content styles
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants