WIP - feat(deployment): Add k8s Helm chart for CLP package deployment.#1603
WIP - feat(deployment): Add k8s Helm chart for CLP package deployment.#1603junhaoliao wants to merge 175 commits into
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| ports: | ||
| - port: 3306 | ||
| targetPort: 3306 | ||
| --- |
There was a problem hiding this comment.
fine to allow --- in helm yamls?
| app.kubernetes.io/component: "database" | ||
| ports: | ||
| - port: 3306 | ||
| nodePort: 30306 |
There was a problem hiding this comment.
shall we expose the port using NodePort, or ask users to run kubectl port-forward to expose ports explicitly for databases?
| ports: | ||
| - port: 4000 | ||
| targetPort: 4000 | ||
| nodePort: 30400 |
There was a problem hiding this comment.
shall we set up nginx ingress instead of this?
| @@ -0,0 +1,17 @@ | |||
| apiVersion: "v2" | |||
| name: "clp-package" | |||
There was a problem hiding this comment.
- is this name fine?
- is the
tools/deployment/package-helmname fine?
| apiVersion: "v1" | ||
| kind: "ConfigMap" | ||
| metadata: | ||
| name: {{ include "clp-package.fullname" . }}-config |
There was a problem hiding this comment.
maybe -generated-config?
|
|
||
| storage: | ||
| # Base directory on host when using local PVs. | ||
| localPathBase: "/tmp/clp" |
There was a problem hiding this comment.
maybe just a clpHome key at the root? or we enforce users to specify absolute paths in the clpConfig
There was a problem hiding this comment.
i believe it's better to enforce absolute paths in clpConfig, so that we don't have to do path manipulations with go templates. though that means users can't just directly copy over an already working clp-config.yml and paste into clpConfig - they will have to adjust all paths to use absolute paths
…ix query scheduler hostname binding issue.
…r, queue, and redis.
| port: 6379 | ||
| query_backend_database: {{ .Values.clpConfig.redis.query_backend_database }} | ||
| reducer: | ||
| base_port: {{ .Values.clpConfig.reducer.base_port }} |
There was a problem hiding this comment.
technically we can hardcode this since we dont need to expose the ports outside of the cluster?
…s for staged-streams and staged-archives
…o ensure proper YAML formatting
…eams in API server deployment
…y in configmap template
…er deployment template
…onals in deployment templates
… to ensure proper YAML formatting
…and streams in worker deployments
…t templates for consistency
…for Kubernetes, enhance configuration details, and improve orchestration documentation
…tinode worker deployments
…templates, add distributed mode support, and enhance worker scheduling options.
…duling examples and cases Add revised examples for scheduling workers with separate node pools or shared nodes, including updated Helm configurations. Introduce a test script for multi-node deployment with dedicated workers and modify the shared worker test script accordingly.
…ver deployments to match docker compose
|
This prototype PR can now be closed, as all of its contents were checked in much earlier. |
this PR is meant to be reference only and should not be merged.
Summary - PR Checklist
database.namewithdatabase.names.clp(fixes #1811). #1814 (0.1.2-dev.8) - Refactordatabase.nametodatabase.names.clpquery_scheduler.portfromvalues.yaml(resolves #1812). #1815 (0.1.2-dev.9) - Remove unusedquery_scheduler.port0.1.2-dev.10) - Add query-scheduler probes0.1.2-dev.11) - Add garbage_collector deployment (incl. retention_period)0.1.2-dev.12) - Add api_server deployment0.1.2-dev.13) - Add mcp_server deployment0.1.2-dev.14) - Template formatting improvements0.1.2-dev.15) - Add missing database.names.spider which prevented the API server from starting.0.1.2-dev.16) - S3/AWS storage support (depends on feat(helm): Add garbage collector deployment with retention period configuration. #1817, feat(helm): Add API server deployment. #1818)log-ingestordeployment. #1856 (0.1.2-dev.17) - Add log-ingestor for S3 (depends on feat(helm): Add S3 storage support for archives, streams, and log inputs. #1825)0.1.2-dev.18) - Add node affinity and topology spread for multinode worker deployments (depends on feat(helm): Add S3 storage support for archives, streams, and log inputs. #1825)log_ingestoris explicitly null in configmap when disabled (fixes #1881). #1882 (0.1.2-dev.19) - fix(helm): Ensure log_ingestor is explicitly null in configmap when disabled (fixes Fix log_ingestor configuration fallback to defaults when set to null in Helm values.yaml #1881).log_ingestorby default (fixes #1912); gatelog_ingestortemplates to S3logs_inputonly. #1913 (0.1.3-dev.2) - fix(helm): Enable log_ingestor by default (fixes Enable log_ingestor by default in Helm values.yaml #1912); gate log_ingestor templates to S3 logs_input only.0.1.3-dev.3) - Use tmpfs for temporary directory mounts in Docker Compose.0.1.2-dev.21) - Spider support (code + docs, de-prioritized)