Description

This issue tracks the work needed to fully support using external 3rd-party services (database, queue, redis, results_cache) instead of bundled ones.

Background

PR #1648 introduced the bundled configuration option to allow users to specify which 3rd-party services to start. However, additional changes are needed to fully support external services:

1. Conditional transform_for_container() calls: The transform_for_container() method should only be called for 3rd-party service components that are bundled, not for external ones.

2. Remove hardcoded hostnames: Several hardcoded service hostnames (e.g., results_cache, queue, redis) in docker-compose-all.yaml need to be replaced with environment variables to allow connecting to external services.

Context

• Discussed in: feat(clp-package): Support selectively enabling 3rd-party services via bundled config (resolves #1623). #1648 (comment)
• Requested by: @junhaoliao

Requirements

• Conditionally call .transform_for_container() only for bundled services in components/clp-py-utils/clp_py_utils/clp_config.py
• Replace hardcoded hostnames with environment variables in tools/deployment/package/docker-compose-all.yaml for:
  • results-cache-init (results_cache hostname)
  • compression-scheduler (queue, redis hostnames)
  • compression-worker (queue, redis hostnames)
  • query-scheduler (queue, redis hostnames)
  • query-worker (queue, redis hostnames)

This will enable users to set bundled: [] in clp-config.yaml and configure external service endpoints without manual modifications to the generated configuration.