Summary

Provide full tool list/spec refresh support for already-registered MCP gateways, including periodic polling, manual refresh APIs (for Orchestrate), and event-driven refresh when servers signal tools/list_changed. This should update tool specs (schemas/metadata) and handle add/update/remove without requiring gateway re-registration.

Background

PR #1950 adds AUTO_REFRESH_SERVERS to sync tools/prompts/resources during health checks. That provides a global, health-check-driven refresh but does not cover:

• Manual refresh API for Orchestrate
• Per-gateway refresh interval configuration
• Event-driven refresh via notifications/tools/list_changed
• Full tool spec synchronization (e.g., annotations/outputSchema)
• Concurrency/locking requirements for overlapping refreshes

This issue completes REQ-002 beyond the health-check auto-refresh path.

Requirements

1. Polling refresh

• Periodically poll remote MCP servers for tools/list (and optionally prompts/resources) and sync with DB.
• Allow per-gateway interval; fall back to a global default.
• Avoid refresh when a gateway is disabled/unreachable, but refresh after it becomes healthy again.

2. Manual refresh API (Orchestrate)

• POST /gateways/{id}/tools/refresh to re-discover tools immediately.
• Return counts: added/updated/removed; include validation errors if any.
• Support optional include_resources / include_prompts flags.

3. Event-driven refresh

• If server advertises capability and sends notifications/tools/list_changed, enqueue a refresh for that gateway.
• Ensure notifications don’t trigger refresh storms (debounce).

4. Tool spec synchronization

• Update all relevant tool fields (not just list membership):
  • description, request_type, headers, input_schema, output_schema (if supported), jsonpath_filter, annotations, visibility/auth metadata as required.
• Ensure stale tools are removed or deactivated per policy.

5. Concurrency & safety

• Only one refresh per gateway at a time (per-gateway lock).
• No long DB sessions during network calls; use fresh sessions for writes.
• Metrics + logs for refresh duration, changes, and failures.

Acceptance Criteria

• A tool added/removed/modified on the MCP server is reflected in Context Forge without deregistering the gateway.
• Manual refresh endpoint updates tools within one call and returns counts.
• tools/list_changed notifications trigger a refresh within a bounded time window.
• Tool spec changes (including annotations/outputSchema) are persisted.
• Refreshes do not overlap for a single gateway and do not block health checks.

Notes / References

• Spec: https://modelcontextprotocol.io/specification/2025-06-18/basic/utilities/cancellation
• Spec: https://modelcontextprotocol.io/specification/2025-06-18/basic/utilities/progress
• MCP notifications/tools/list_changed is already modeled in mcpgateway/common/models.py.
• PR configurable auto refresh for automatic tools/prompts/resources sync during health checks #1950 provides baseline health-check auto-refresh; this issue extends it.