Background

Currently, the CLP Pacakge lacks a mechanism for users to control whether non-CLP (3rd-party) services are started automatically, or whether external services (e.g., AWS RDS or locally installed databases) are used. This can lead to inconsistencies and unnecessary service deployments being started by the CLP Package, which may conflict with user's service deployments.

Additionally, the database deployment setup is tightly coupled with environment configuration:

• _set_up_env_for_database both creates deployment directories and sets CLP database environment variables.
• _set_up_env_for_spider_db only sets Spider database environment variables.

This makes it difficult to manage deployment logic and environment configuration independently.

Proposed Enhancement

Allow users to explicitly specify in clp-config.yaml which 3rd-party services they want start-clp.sh to start. This would provide better control and consistency.

Suggested Approach

1. Add a bundled array to clp-config.yaml to explicitly define which 3rd-party services should be started, e.g.:

   bundled: ["database", "queue", "redis", "results_cache"]

2. Create dedicated setup functions for each 3rd-party service deployment so that any deployment-specific initialisation logic is cleanly separated.

Context

• PR: feat(clp-package)!: Add Spider config interfaces; Integrate Spider compression orchestration into the Docker Compose project. #1606
• Comment: feat(clp-package)!: Add Spider config interfaces; Integrate Spider compression orchestration into the Docker Compose project. #1606 (comment)
• Requested by: @junhaoliao