feat: non-privileged base cluster with new kustomize#4215
Conversation
| gcloud container clusters get-credentials cloud --zone us-central1-f --project sourcegraph-dev | ||
|
|
||
| kubectl apply --dry-run --validate --recursive -f base/ --context=gke_sourcegraph-dev_us-central1-f_cloud | ||
| kustomize build ./base/. | kubectl apply --dry-run --validate --context=gke_sourcegraph-dev_us-central1-f_cloud -f - |
There was a problem hiding this comment.
Why the change in the CI for the base cluster? I assume this is technically equivalent to before, but I could see the motivation be that we want to test a command that is more directly what we ask users to run.
Would it also make sense to add CI commands for new/base?
There was a problem hiding this comment.
Why the change in the CI for the base cluster?
this is due to the kustomization files added to the original base that requires kustomize to run, but since we will be setting up a separate repo for the new base, this change will not be necessary.
Would it also make sense to add CI commands for new/base?
It makes sense to me! I've created a new issue for creating tests for the new base https://github.com/orgs/sourcegraph/projects/290?pane=issue&itemId=18056795
| # Apply the base Soucegraph deployment | ||
| # shellcheck disable=SC2068 | ||
| kubectl apply --prune -l deploy=sourcegraph -f base --recursive $@ | ||
| echo "Printing diff" |
There was a problem hiding this comment.
Do we need to change the commands for the old base cluster? These seem roughly equivalent, but if that's the case, I would prefer not changing them in case there are subtle differences or if changing this file causes a merge conflict for anyone who has forked this repository.
New default cluster with improved deployment setup
❌ DO NOT MERGE UNTIL #4215 IS CLOSED ❌
RE: #4213, job fair item
Related PRs: https://github.com/sourcegraph/sourcegraph/pull/46227 https://github.com/sourcegraph/sourcegraph/pull/45862
Introduction
This PR serves to introduce a new base cluster for Sourcegraph, along with a new Kustomize setup for deploying Sourcegraph that aligns with industry standards and provides a better user experience.
Background
The current base cluster includes all resources that may not be suitable for all user environments, resulting in permission issues during deployments and causing delays and a poor customer experience. In an effort to address these issues, we have developed a new base cluster that is tailored to the scaling and security requirements of enterprise customer environments.
New base cluster
The new base cluster is made up of two different stacks:
Notable changes and improvements
New Kustomize setup
The new setup introduces a better deployment experience for everyone, while addressing the following common issues caused by our current Kubernetes deployment method:
Benefits
The changes we are introducing provide users with higher flexibility to customize their Sourcegraph deployment using the components that are pre-configured by Sourcegraph. This in turn provides us with the ability to control and oversee the changes customers make and prevent incidents where clusters are set up with service-breaking configurations.
It also allows different services to be deployed separately. No one should be blocked due to permission issue
Expectations
The new changes shouldNOT affect current users and existing instances as the new setup uses the same tools that are currently used by our customers who are deploying with the custom scripts (which use Kustomize under the hood). It is also built into kubectl, a tool that is required to perform all Kubernetes deployments.
We hope this new base cluster and deployment method will improve the user experience and provide a more industry-standard approach to deploying Sourcegraph, and more importantly, make deploying Sourcegraph easier for everyone
Decision flow
For users with RBAC-enabled clusters
For users without RBAC-enabled cluster
Progress Updates / Notes
TODO:
[x] Combine current non-privilege and non-privilege-create-cluster overlays into one
[x] Replace searcher deployment and symbols deployment with StatefulSets
[x] Add
fsGroupPolicyto replace the oldnon-privileged-create-clusteroverlay[x] Add commonly-used kustomize overlays and components for the new base cluster
[x] Add component and overlay for k3s deployment
[x] Move all monitoring-related resources into a separate directory
[x] Add component and overlay for instance size - XS
[x] Add component and overlay for instance size - S
[x] Add component and overlay for instance size - M
[x] Add component and overlay for instance size - L
[x] Add component and overlay for instance size - XL
[x] Test searcher and symbols as StatefulSets
[x] Remove all rbac resources from frontend
[x] Remove rolebinding for frontend
Issues:
Checklist
Test plan
How to test
Instructions to build / deploy the new cluster using the new Kustomize set ups.
WIP: plugin to generate endpoints at build time
# NOT READY FOR TESTING kustomize build --enable-alpha-plugins --enable-exec new/plugins/example -o new/generated-cluster.yaml