Background

We currently have three different mechanisms for optionally enabling or disabling services in our Docker Compose setup:

1. Replica count toggling (0/1)

   • Example: Garbage Collector
   • Controlled via env var
   • deploy.replicas = 1 or 0

2. Profiles

   • Example: MCP server
   • Enabled via --profile=mcp
   • Requires controller logic to inject flags at startup

3. Separate overlay compose files

   • Example: Query services (and soon compression services via spider integration)
   • Additional docker-compose-XYZ.yaml extending the base
   • Enabled by passing -f docker-compose-XYZ.yaml at runtime
   • Used for toggling groups of services when the composition varies significantly

Each mechanism serves a slightly different use case, but the result is fragmented enable/disable logic, increasing maintenance burden and making controller code inconsistent.

Problem

Because we use multiple patterns to control optional service activation:

1. The controller code becomes inconsistent.
   • Replica toggling uses environment setup functions (CLP_GARBAGE_COLLECTOR_ENABLED inside _set_up_env_for_garbage_collector()).
   • Profiles require injecting CLI flags in the start() path (cmd += ["--profile", "mcp"]), which is less isolated.
2. It's harder to reason about which services support which activation pattern.
3. Adding new optional services may copy-paste the existing inconsistency.

Proposal

Adopt (1) and (3) as the two primary supported mechanisms, and phase out (2) profiles.

• Use (1) replica count toggling for individual services that are simply enabled or disabled.
• Use (3) overlay compose files for groups of services whose compositions vary significantly depending on the configuration (e.g., different schedulers with different service sets).

Remove or migrate existing usage of (2) profiles, since it introduces controller-side CLI logic and is less maintainable. Specifically, remove the MCP server's profile-based activation and migrate it to mechanism (1) using an environment-variable-controlled replica count (e.g., CLP_MCP_ENABLED -> replicas 0 or 1).

Related PR: #1481
Comment: #1481 (comment)
Requested by: @junhaoliao