GitLab Provider: Validation of SecretStore failed with "remote error: tls: internal error" #1881
Description
Describe the solution you'd like
I want to be able to use GitLab as my SecretStore.
What is the added value?
The GitLab Provider will work as intended.
Observations (Constraints, Context, etc):
Context: I have two similar k3s clusters running and both are using ESO with the GitLab Provider and with the exact same configuration. Both clusters are running the latest stable version of k3s, which currently is v1.25.5+k3s1. Both clusters are running on machines on Ubuntu 22.04. The following error occurred with ESO 0.7.0 but also occurs after an update to 0.7.1.
Yesterday I was updating the VM one of the clusters is running on (it's single node) from Ubuntu 20.04 to 22.04. I am not sure whether this caused the problem but later I saw that the GitLab ClusterSecretStore I was already successfully using for almost a year stopped working with the following error:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ValidationFailed 15m (x51 over 13h) cluster-secret-store could not verify whether the gilabClient is valid: Get "https://gitlab.com/api/v4/pro
jects/<my-project-id>/variables": remote error: tls: internal error
The external-secrets pod is giving me the following logs periodically:
{
"level": "error",
"ts": 1673091831.7901502,
"logger": "controllers.ClusterSecretStore",
"msg": "unable to validate store",
"clustersecretstore": "/gitlab-secret-store",
"error": "could not validate provider: could not verify whether the gilabClient is valid: Get \"https://gitlab.com/api/v4/projects/<my-project-id>/variables\":remote error: tls: internal error",
"stacktrace": "github.com/external-secrets/external-secrets/pkg/controllers/secretstore.reconcile\n\t/home/runner/work/external-secrets/external-secrets/pkg/controllers/secretstore/common.go: 67\ngithub.com/external-secrets/external-secrets/pkg/controllers/secretstore.(*ClusterStoreReconciler).Reconcile\n\t/home/runner/work/external-secrets/external-secrets/pkg/controllers/secretstore/clustersecretstore_controller.go: 54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go: 122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go: 323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go: 274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go: 235"
},
{
"level": "error",
"ts": 1673091831.8024669,
"msg": "Reconciler error",
"controller": "clustersecretstore",
"controllerGroup": "external-secrets.io",
"controllerKind": "ClusterSecretStore",
"ClusterSecretStore": {
"name": "gitlab-secret-store"
},
"namespace": "",
"name": "gitlab-secret-store",
"reconcileID": "57ca9df3-68cb-4266-b51b-a851f074a44f",
"error": "could not validate provider: could not verify whether the gilabClient is valid: Get \"https://gitlab.com/api/v4/projects/<my-project-id>/variables\": remote error: tls: internal error",
"stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go: 329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go: 274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go: 235"
}
The cert-controller and webhook pods don't log anything suspicious.
- When I curl
https://gitlab.com/api/v4/projects/<my-project-id>/variablesfrom the machines k3s is running on and from containers I spawn, I get the expected response with the variables. - As I said: I have two clusters and the other cluster is running with exactly the same configuration and it's working there.
- I completely re-installed ESO with no success
- I completely re-installed the whole cluster - still the same problem
Now I've run out of ideas how to troubleshoot this and am searching for help here. If I can provide you more information I am happy to do so. And I am happy for every help I get.