Skip to content

1892 get member count n plus one#1905

Merged
crivetimihai merged 3 commits intomainfrom
1892-get-member-count-n-plus-one
Jan 5, 2026
Merged

1892 get member count n plus one#1905
crivetimihai merged 3 commits intomainfrom
1892-get-member-count-n-plus-one

Conversation

@crivetimihai
Copy link
Copy Markdown
Member

@crivetimihai crivetimihai commented Jan 5, 2026

PR: Fix N+1 query patterns in team member counts

Summary

  • Adds batch query methods to eliminate N+1 patterns in admin UI team listings
  • Implements Redis caching for team member counts with configurable TTL
  • Adds partial database index for optimized COUNT queries
  • Reduces /admin/ P95 latency from 14s to <500ms under load

Problem

Load testing with 3000 concurrent users revealed critical performance issues:

  • 11,429+ individual COUNT queries for team member counts
  • 60+ idle-in-transaction connections causing timeouts
  • 7.8 second average response time for /admin/ endpoint
  • 21 idle transaction timeout errors during test runs

Root cause: EmailTeam.get_member_count() executes a separate COUNT query for each team, creating an N+1 pattern when listing teams.

Solution

Phase 1: Batch Query Methods

Three new methods in TeamManagementService:

  • get_member_counts_batch() - batch fetch member counts
  • get_user_roles_batch() - batch fetch user roles
  • get_pending_join_requests_batch() - batch fetch pending join requests

Phase 2: Call Site Updates

Updated 6 call sites to use batch methods:

  • admin.py:admin_ui()
  • admin.py:_render_unified_teams_view()
  • admin.py:admin_list_teams()
  • routers/teams.py:list_teams()
  • routers/teams.py:discover_teams()

Phase 3: Database Index

Added partial index for PostgreSQL:

CREATE INDEX idx_email_team_members_team_active_partial
ON email_team_members(team_id) WHERE is_active = true;

Phase 4: Redis Caching

  • get_member_counts_batch_cached() with configurable TTL
  • Automatic cache invalidation on membership changes
  • Graceful fallback when Redis unavailable

Configuration

New settings (all optional, sensible defaults):

TEAM_MEMBER_COUNT_CACHE_ENABLED=true   # Enable/disable caching
TEAM_MEMBER_COUNT_CACHE_TTL=300        # TTL in seconds (30-3600)

Performance Impact

Metric Before After Improvement
Queries per page 3N+2 4-5 ~60× reduction (N=100)
/admin/ P95 latency 14,000 ms <500 ms 28× faster
Idle-in-transaction 60+ <5 12× reduction
Database connections 100+ ~20 80% reduction

Files Changed

Configuration

  • mcpgateway/config.py - New cache settings
  • .env.example - Documentation
  • docker-compose.yml - Default values
  • charts/mcp-stack/values.yaml - Helm chart values
  • charts/mcp-stack/values.schema.json - Schema validation
  • docs/docs/manage/configuration.md - User documentation

Implementation

  • mcpgateway/services/team_management_service.py - Batch methods + caching
  • mcpgateway/admin.py - Updated call sites
  • mcpgateway/routers/teams.py - Updated call sites
  • mcpgateway/alembic/versions/ - Index migration

Tests

  • tests/unit/mcpgateway/services/test_team_management_service.py
  • tests/integration/ - Cache invalidation tests

Closes

Closes #1892
Closes #1893
Contributes to #1894
Contributes to #1862

Testing

  • Unit tests for all 3 batch methods
  • Unit tests for cached method (enabled/disabled/Redis unavailable)
  • Unit tests for cache invalidation
  • Integration tests for membership change invalidation
  • Load test confirms N+1 elimination
  • Load test confirms cache hit rate >80%
  • Helm chart deploys successfully

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
@crivetimihai crivetimihai marked this pull request as ready for review January 5, 2026 00:10
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
@crivetimihai crivetimihai merged commit ed0d93d into main Jan 5, 2026
50 checks passed
@crivetimihai crivetimihai deleted the 1892-get-member-count-n-plus-one branch January 5, 2026 00:57
kcostell06 pushed a commit to kcostell06/mcp-context-forge that referenced this pull request Feb 24, 2026
* [PERFORMANCE]: N+1 query pattern in EmailTeam.get_member_count() IBM#1892

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

* [PERFORMANCE]: N+1 query pattern in EmailTeam.get_member_count() IBM#1892

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

* [PERFORMANCE]: N+1 query pattern in EmailTeam.get_member_count() IBM#1892

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>

---------

Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[PERFORMANCE]: Add partial index for team member count queries [PERFORMANCE]: N+1 query pattern in EmailTeam.get_member_count()

1 participant