Skip to content

Allow the E2E tests to run on clusters that have not been prepared by the Makefile#6387

Merged
jetstack-bot merged 1 commit intocert-manager:masterfrom
wallrj:portable-e2e-test-binary
Oct 4, 2023
Merged

Allow the E2E tests to run on clusters that have not been prepared by the Makefile#6387
jetstack-bot merged 1 commit intocert-manager:masterfrom
wallrj:portable-e2e-test-binary

Conversation

@wallrj
Copy link
Copy Markdown
Member

@wallrj wallrj commented Oct 4, 2023 •
edited
Loading

In cert-manager/cert-manager-olm#99 I attempted to test the cert-manager OLM packages by runing make e2e-build to create an e2e.test binary and I upload that to a virtual machine where I'm running an OpenShift test cluster with cert-manager installed via OperatorHub.

But the tests won't run because they first deploy a Vault server where the Deployment refers to a Docker image called local/vault:local.
That image is normally pre-loaded into the cluster by the makefile, but in my case I'm running the tests on a cluster where the make e2e-setup script has not (and cannot) be run.

This PR changes the E2E test so that it only uses the local/vault image if it detects that it is being executed by make.

This problem was caused by #5950 where @inteon changed the E2E tests to always install a single instance of Vault before the tests run, for performance reasons.
Previously the Vault addon would only be installed and uninstalled if the "Vault Issuer" tests were run.
So previously, I was able to compile the E2E binary and run only the "CA Issuer" tests without triggering the installation of Vault.

Testing

# Create a cluster
kind create cluster

# Install cert-manager from OperatorHub.io
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.25.0/install.sh | bash -s v0.25.0
kubectl create -f https://operatorhub.io/install/cert-manager.yaml

# Compile the E2E tests
make e2e-build

# Execute the Vault Issuer tests
./_bin/test/e2e.test --repo-root=/dev/null --ginkgo.focus="Vault\ Issuer" --ginkgo.skip="Gateway"

You see that the Vault server is using the standard Docker image

$ kubectl -n e2e-vault describe pod chart-vault-vault-0
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  10s   default-scheduler  Successfully assigned e2e-vault/chart-vault-vault-0 to kind-control-plane
  Normal  Pulling    10s   kubelet            Pulling image "hashicorp/vault:1.13.1"

And as a result I can now run the Vault Issuer E2E tests on my personal cluster.
There is a warning due to some bug in the way we set up the logging, which can be addressed in another PR.
And there is a bug where the service account token used by one of the tests is assuming that cert-manager is installed in the cert-manager namespace, but operatorhub.io installs in an operators namespace by default.

$ ./_bin/test/e2e.test --repo-root=/dev/null --ginkgo.focus="Vault\ Issuer" --ginkgo.skip="Gateway"
Running Suite: cert-manager e2e suite - /home/richard/projects/cert-manager/cert-manager
========================================================================================
Random Seed: 1696435538

Will run 25 of 777 specs
"hashicorp" already exists with the same configuration, skipping
Release "chart-vault-vault" does not exist. Installing it now.
NAME: chart-vault-vault
LAST DEPLOYED: Wed Oct  4 17:05:40 2023
NAMESPACE: e2e-vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/


Your release is named chart-vault-vault. To learn more about the release, try:

  $ helm status chart-vault-vault
  $ helm get manifest chart-vault-vault
SSSSSSSSSS•••••SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•••••SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS••[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
        >  goroutine 747 [running]:
        >  runtime/debug.Stack()
        >       runtime/debug/stack.go:24 +0x5e
        >  sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
        >       sigs.k8s.io/controller-runtime@v0.16.2/pkg/log/log.go:60 +0xcd
        >  sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).WithName(0xc000232040, {0x1da69ee, 0x14})
        >       sigs.k8s.io/controller-runtime@v0.16.2/pkg/log/deleg.go:147 +0x45
        >  github.com/go-logr/logr.Logger.WithName({{0x2070d50, 0xc000232040}, 0x0}, {0x1da69ee?, 0xc000e3fd98?})
        >       github.com/go-logr/logr@v1.2.4/logr.go:336 +0x3d
        >  sigs.k8s.io/controller-runtime/pkg/client.newClient(0x0?, {0x0, 0xc00020f0a0, {0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0})
        >       sigs.k8s.io/controller-runtime@v0.16.2/pkg/client/client.go:122 +0xec
        >  sigs.k8s.io/controller-runtime/pkg/client.New(0x1dc1036?, {0x0, 0xc00020f0a0, {0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0})
        >       sigs.k8s.io/controller-runtime@v0.16.2/pkg/client/client.go:103 +0x7d
        >  github.com/cert-manager/cert-manager/e2e-tests/framework.(*Framework).BeforeEach(0xc0006392c0)
        >       github.com/cert-manager/cert-manager/e2e-tests/framework/framework.go:145 +0x33c
        >  github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0xa01938, 0xc000ba2180})
        >       github.com/onsi/ginkgo/v2@v2.12.0/internal/node.go:463 +0x13
        >  github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
        >       github.com/onsi/ginkgo/v2@v2.12.0/internal/suite.go:865 +0x8d
        >  created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 66
        >       github.com/onsi/ginkgo/v2@v2.12.0/internal/suite.go:852 +0xd7b
•••SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•••••••••
------------------------------
• [FAILED] [60.129 seconds]
[cert-manager] Vault Issuer [It] should be ready with a valid serviceAccountRef
github.com/cert-manager/cert-manager/e2e-tests/suite/issuers/vault/issuer.go:346

  Timeline >>
  STEP: Creating a kubernetes client @ 10/04/23 17:06:33.153
  STEP: Creating an API extensions client @ 10/04/23 17:06:33.154
  STEP: Creating a cert manager client @ 10/04/23 17:06:33.154
  STEP: Creating a controller-runtime client @ 10/04/23 17:06:33.154
  STEP: Creating a gateway-api client @ 10/04/23 17:06:33.154
  STEP: Building a namespace api object @ 10/04/23 17:06:33.154
  STEP: Using the namespace e2e-tests-create-vault-issuer-xxpht @ 10/04/23 17:06:33.159
  STEP: Building a ResourceQuota api object @ 10/04/23 17:06:33.159
  STEP: Configuring the Vault server @ 10/04/23 17:06:33.165
  STEP: creating a service account for Vault authentication @ 10/04/23 17:06:33.218
  STEP: Creating the Role and RoleBinding to let cert-manager use TokenRequest for the ServiceAccount @ 10/04/23 17:06:33.225
  STEP: Creating an Issuer @ 10/04/23 17:06:33.231
  STEP: Waiting for Issuer to become Ready @ 10/04/23 17:06:33.239
  Oct  4 17:06:33.239: INFO: Waiting for issuer test-vault-issuer condition v1.IssuerCondition{Type:"Ready", Status:"True", LastTransitionTime:<nil>, Reason:"", Message:"", ObservedGeneration:0}
  Oct  4 17:06:38.245: INFO: Waiting for issuer test-vault-issuer condition v1.IssuerCondition{Type:"Ready", Status:"True", LastTransitionTime:<nil>, Reason:"", Message:"", ObservedGeneration:0}
  Oct  4 17:06:44.745: INFO: Waiting for issuer test-vault-issuer condition v1.IssuerCondition{Type:"Ready", Status:"True", LastTransitionTime:<nil>, Reason:"", Message:"", ObservedGeneration:0}
  Oct  4 17:06:52.245: INFO: Waiting for issuer test-vault-issuer condition v1.IssuerCondition{Type:"Ready", Status:"True", LastTransitionTime:<nil>, Reason:"", Message:"", ObservedGeneration:0}
  Oct  4 17:07:01.245: INFO: Waiting for issuer test-vault-issuer condition v1.IssuerCondition{Type:"Ready", Status:"True", LastTransitionTime:<nil>, Reason:"", Message:"", ObservedGeneration:0}
  Oct  4 17:07:11.745: INFO: Waiting for issuer test-vault-issuer condition v1.IssuerCondition{Type:"Ready", Status:"True", LastTransitionTime:<nil>, Reason:"", Message:"", ObservedGeneration:0}
  Oct  4 17:07:24.246: INFO: Waiting for issuer test-vault-issuer condition v1.IssuerCondition{Type:"Ready", Status:"True", LastTransitionTime:<nil>, Reason:"", Message:"", ObservedGeneration:0}
  Oct  4 17:07:33.246: INFO: Waiting for issuer test-vault-issuer condition v1.IssuerCondition{Type:"Ready", Status:"True", LastTransitionTime:<nil>, Reason:"", Message:"", ObservedGeneration:0} (took 1m0s)
  [FAILED] in [It] - github.com/cert-manager/cert-manager/e2e-tests/suite/issuers/vault/issuer.go:370 @ 10/04/23 17:07:33.264
  STEP: Cleaning up AppRole @ 10/04/23 17:07:33.264
  STEP: Cleaning up Kubernetes @ 10/04/23 17:07:33.272
  STEP: Cleaning up Vault @ 10/04/23 17:07:33.274
  STEP: Deleting test namespace @ 10/04/23 17:07:33.278
  << Timeline

  [FAILED] Unexpected error:
      <*errors.errorString | 0xc0011de1a0>:
      context deadline exceeded: Last Status: 'False' Reason: 'VaultError', Message: 'Failed to initialize Vault client: while requesting a Vault token using the Kubernetes auth: while requesting a token for the service account e2e-tests-create-vault-issuer-xxpht/vault-serviceaccount: serviceaccounts "vault-serviceaccount" is forbidden: User "system:serviceaccount:operators:cert-manager" cannot create resource "serviceaccounts/token" in API group "" in the namespace "e2e-tests-create-vault-issuer-xxpht"'
      {
          s: "context deadline exceeded: Last Status: 'False' Reason: 'VaultError', Message: 'Failed to initialize Vault client: while requesting a Vault token using the Kubernetes auth: while requesting a token for the service account e2e-tests-create-vault-issuer-xxpht/vault-serviceaccount: serviceaccounts \"vault-serviceaccount\" is forbidden: User \"system:serviceaccount:operators:cert-manager\" cannot create resource \"serviceaccounts/token\" in API group \"\" in the namespace \"e2e-tests-create-vault-issuer-xxpht\"'",
      }
  occurred
  In [It] at: github.com/cert-manager/cert-manager/e2e-tests/suite/issuers/vault/issuer.go:370 @ 10/04/23 17:07:33.264
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSproxy logs:


Summarizing 1 Failure:
  [FAIL] [cert-manager] Vault Issuer [It] should be ready with a valid serviceAccountRef
  github.com/cert-manager/cert-manager/e2e-tests/suite/issuers/vault/issuer.go:370

Ran 25 of 777 Specs in 115.215 seconds
FAIL! -- 24 Passed | 1 Failed | 0 Pending | 752 Skipped
--- FAIL: TestE2E (115.23s)
FAIL

The CA Issuer tests all pass, which is the important thing for my OperatorHub package testing

r$ ./_bin/test/e2e.test --repo-root=/dev/null --ginkgo.focus="CA\ Issuer" --ginkgo.skip="Gateway"
Running Suite: cert-manager e2e suite - /home/richard/projects/cert-manager/cert-manager
========================================================================================
Random Seed: 1696435730

Will run 44 of 777 specs
"hashicorp" already exists with the same configuration, skipping
Release "chart-vault-vault" does not exist. Installing it now.
NAME: chart-vault-vault
LAST DEPLOYED: Wed Oct  4 17:08:52 2023
NAMESPACE: e2e-vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/


Your release is named chart-vault-vault. To learn more about the release, try:

  $ helm status chart-vault-vault
  $ helm get manifest chart-vault-vault
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS••••SSS••••••[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
        >  goroutine 482 [running]:
        >  runtime/debug.Stack()
        >       runtime/debug/stack.go:24 +0x5e
        >  sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
        >       sigs.k8s.io/controller-runtime@v0.16.2/pkg/log/log.go:60 +0xcd
        >  sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).WithName(0xc000234040, {0x1da69ee, 0x14})
        >       sigs.k8s.io/controller-runtime@v0.16.2/pkg/log/deleg.go:147 +0x45
        >  github.com/go-logr/logr.Logger.WithName({{0x2070d50, 0xc000234040}, 0x0}, {0x1da69ee?, 0xc000917d98?})
        >       github.com/go-logr/logr@v1.2.4/logr.go:336 +0x3d
        >  sigs.k8s.io/controller-runtime/pkg/client.newClient(0x0?, {0x0, 0xc00022a620, {0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0})
        >       sigs.k8s.io/controller-runtime@v0.16.2/pkg/client/client.go:122 +0xec
        >  sigs.k8s.io/controller-runtime/pkg/client.New(0x1dc1036?, {0x0, 0xc00022a620, {0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0})
        >       sigs.k8s.io/controller-runtime@v0.16.2/pkg/client/client.go:103 +0x7d
        >  github.com/cert-manager/cert-manager/e2e-tests/framework.(*Framework).BeforeEach(0xc000280960)
        >       github.com/cert-manager/cert-manager/e2e-tests/framework/framework.go:145 +0x33c
        >  github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0xa01948, 0xc000660180})
        >       github.com/onsi/ginkgo/v2@v2.12.0/internal/node.go:463 +0x13
        >  github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
        >       github.com/onsi/ginkgo/v2@v2.12.0/internal/suite.go:865 +0x8d
        >  created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 11
        >       github.com/onsi/ginkgo/v2@v2.12.0/internal/suite.go:852 +0xd7b
••••••••••••SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSproxy logs:


Ran 23 of 777 Specs in 56.241 seconds
SUCCESS! -- 23 Passed | 0 Failed | 0 Pending | 754 Skipped
PASS
NONE

@wallrj wallrj self-assigned this Oct 4, 2023
@jetstack-bot jetstack-bot added release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Oct 4, 2023
@wallrj wallrj requested a review from inteon October 4, 2023 15:58
@jetstack-bot jetstack-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. area/testing Issues relating to testing labels Oct 4, 2023
@wallrj
Allow the E2E tests to run on clusters that have not been prepared by…
0b7f36a 
… the Makefile

Signed-off-by: Richard Wall <richard.wall@venafi.com>
@wallrj wallrj force-pushed the portable-e2e-test-binary branch from 7323c21 to 0b7f36a Compare October 4, 2023 15:58
@wallrj
Copy link
Copy Markdown
Member Author

wallrj commented Oct 4, 2023

/retest

@wallrj wallrj mentioned this pull request Oct 4, 2023
Closed
@inteon
Copy link
Copy Markdown
Member

inteon commented Oct 4, 2023

Looks good!
/lgtm
/approve

@jetstack-bot jetstack-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 4, 2023
@jetstack-bot
Copy link
Copy Markdown
Contributor

jetstack-bot commented Oct 4, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inteon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jetstack-bot jetstack-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 4, 2023
@jetstack-bot jetstack-bot merged commit 3ac37ba into cert-manager:master Oct 4, 2023
@wallrj wallrj deleted the portable-e2e-test-binary branch October 5, 2023 06:03
@wallrj wallrj mentioned this pull request Oct 5, 2023
Merged
@wallrj wallrj mentioned this pull request Feb 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@inteon inteon Awaiting requested review from inteon

Assignees

@wallrj wallrj

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/testing Issues relating to testing dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. lgtm Indicates that a PR is ready to be merged. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wallrj @inteon @jetstack-bot