Skip to content

Use single vault instance in e2e tests#5950

Merged
jetstack-bot merged 4 commits intocert-manager:masterfrom
inteon:use_single_vault_instance
May 2, 2023
Merged

Use single vault instance in e2e tests#5950
jetstack-bot merged 4 commits intocert-manager:masterfrom
inteon:use_single_vault_instance

Conversation

@inteon
Copy link
Copy Markdown
Member

@inteon inteon commented Apr 13, 2023

Goal: reduce e2e test time & reduce flakes

NOTE: I removed the approle_custom_mount tests, because now all tests use custom mounts (a unique mount point name per test)

Kind

/kind cleanup

Release Note

NONE

@jetstack-bot jetstack-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. kind/bug Categorizes issue or PR as related to a bug. kind/feature Categorizes issue or PR as related to a new feature. area/testing Issues relating to testing kind/documentation Categorizes issue or PR as related to documentation. kind/design Categorizes issue or PR as related to design. approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 13, 2023
@inteon inteon force-pushed the use_single_vault_instance branch from cee4f2f to fa83db5 Compare April 13, 2023 18:24
@jetstack-bot jetstack-bot added dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. and removed dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. labels Apr 13, 2023
@inteon inteon force-pushed the use_single_vault_instance branch 2 times, most recently from 0df1a11 to e39d39b Compare April 13, 2023 18:39
@cert-manager cert-manager deleted a comment from jetstack-bot Apr 13, 2023
@inteon inteon removed kind/bug Categorizes issue or PR as related to a bug. kind/design Categorizes issue or PR as related to design. kind/documentation Categorizes issue or PR as related to documentation. kind/feature Categorizes issue or PR as related to a new feature. labels Apr 13, 2023
@jetstack-bot jetstack-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 13, 2023
@inteon inteon force-pushed the use_single_vault_instance branch 4 times, most recently from 58b081e to 087de80 Compare April 13, 2023 20:51
@maelvls
Copy link
Copy Markdown
Member

maelvls commented Apr 14, 2023

Awesome initiative. Running helm install for every single Vault test is a massive waste of time.

@inteon inteon force-pushed the use_single_vault_instance branch 2 times, most recently from 6810e17 to 1abbb20 Compare April 15, 2023 07:41
@inteon inteon force-pushed the use_single_vault_instance branch 8 times, most recently from 04d1aad to 9b6290b Compare April 26, 2023 15:09
@inteon inteon changed the title WIP: Use single vault instance Use single vault instance Apr 26, 2023
@jetstack-bot jetstack-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 26, 2023
@inteon inteon requested a review from irbekrm April 27, 2023 07:48
@inteon inteon mentioned this pull request Apr 27, 2023
@irbekrm
Copy link
Copy Markdown
Contributor

irbekrm commented Apr 27, 2023

/test pull-cert-manager-master-e2e-v1-26
/test pull-cert-manager-master-e2e-v1-22
/test pull-cert-manager-master-e2e-v1-23
/test pull-cert-manager-master-e2e-v1-24
/test pull-cert-manager-master-e2e-v1-25

@inteon
Copy link
Copy Markdown
Member Author

inteon commented Apr 27, 2023

@irbekrm there was still one remaining bug. I fixed it in the last commit.

@inteon inteon changed the title Use single vault instance Use single vault instance in e2e tests Apr 28, 2023
@inteon inteon force-pushed the use_single_vault_instance branch from 768f3ef to 512b7ac Compare April 28, 2023 07:38
Copy link
Copy Markdown
Contributor

@irbekrm irbekrm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Tim, I have read through part of the code.

I would be interested to see some data as to how much improvement this PR actually makes.

At the moment after looking at the code I think that having a shared instance adds too much complexity for the potential gain and makes the already too complicated e2e test setup even more complicated- I am worried that because of that the code will not be maintainable and also we could end up not testing some of the Vault setup methods because of it being hard to reason about what the code does.

if !g.SupportsGlobal() {
return nil, fmt.Errorf("requested global plugin does not support shared mode with current configuration")
}
toBeTransferred[idx] = data
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make []interface{} into map[AddonName]interface{} and use addon name as a key? (Add a Name method to addons)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is easier to keep this as a statically allocated array, otherwise we have to add logic for keys which are not that useful.

inteon added 3 commits April 28, 2023 13:05
Signed-off-by: Tim Ramlot <42113979+inteon@users.noreply.github.com>
Signed-off-by: Tim Ramlot <42113979+inteon@users.noreply.github.com>
Signed-off-by: Tim Ramlot <42113979+inteon@users.noreply.github.com>
@inteon inteon force-pushed the use_single_vault_instance branch from 512b7ac to d3ba8be Compare April 28, 2023 11:50
@inteon
Copy link
Copy Markdown
Member Author

inteon commented Apr 28, 2023

@irbekrm Thank you for the feedback, I improved the comments that explain how data is passed between ginkgo processes. I'll now gather some data on stability & speed and update this comment with that data.

9m45s: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-22/1651941452378279936
9m15s: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-23/1651941452453777408
10m35s: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-24/1651941452516691968
9m32s: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-25/1651941452613160960
9m16s: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-26/1651941452697047040
9m23s: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-27/1651936266138161152

8m50s: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-22/1651954088226590720
8m55s: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-23/1651954088323059712
10m13s: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-24/1651954088423723008
9m16s: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-25/1651954088520192000
8m20s: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-26/1651954088604078080
pebble failure: https://prow.build-infra.jetstack.net/view/gs/jetstack-logs/pr-logs/pull/cert-manager_cert-manager/5950/pull-cert-manager-master-e2e-v1-27/1651954163992498176

Signed-off-by: Tim Ramlot <42113979+inteon@users.noreply.github.com>
@inteon
Copy link
Copy Markdown
Member Author

inteon commented Apr 28, 2023

/test pull-cert-manager-master-e2e-v1-27
/test pull-cert-manager-master-e2e-v1-26
/test pull-cert-manager-master-e2e-v1-22
/test pull-cert-manager-master-e2e-v1-23
/test pull-cert-manager-master-e2e-v1-24
/test pull-cert-manager-master-e2e-v1-25

Copy link
Copy Markdown
Contributor

@irbekrm irbekrm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the changes Tim, it's a lot clearer.
I had another read through this and through Ginkgo docs this morning.

Can we not keep the more standard structure and make it so that we use the Ginkgo's existing mechanism for co-ordinating between first and all processes by putting all the code that needs to run just once into the first function argument and making all the setup code in the second function arg such that it can run on any process (as we have it currently)?
See the example in https://onsi.github.io/ginkgo/#parallel-suite-setup-and-cleanup-synchronizedbeforesuite-and-synchronizedaftersuite:~:text=We%20can%20use%20this%20behavior%20to%20set%20up%20shared%20external%20resources%20like%20so%3A
Or, is this not possible? I think this would be preferable instead of building our own mechanism (the isGinkgoProcessNumberOne variable) on top of what Ginkgo already has.

If this is not possible due to addons structure, happy to approve this PR as is, but perhaps add a note as to why or if it could be done in future by changing the addons structure.

@inteon
Copy link
Copy Markdown
Member Author

inteon commented May 2, 2023

Thanks for making the changes Tim, it's a lot clearer. I had another read through this and through Ginkgo docs this morning.

Can we not keep the more standard structure and make it so that we use the Ginkgo's existing mechanism for co-ordinating between first and all processes by putting all the code that needs to run just once into the first function argument and making all the setup code in the second function arg such that it can run on any process (as we have it currently)? See the example in https://onsi.github.io/ginkgo/#parallel-suite-setup-and-cleanup-synchronizedbeforesuite-and-synchronizedaftersuite:~:text=We%20can%20use%20this%20behavior%20to%20set%20up%20shared%20external%20resources%20like%20so%3A Or, is this not possible? I think this would be preferable instead of building our own mechanism (the isGinkgoProcessNumberOne variable) on top of what Ginkgo already has.

If this is not possible due to addons structure, happy to approve this PR as is, but perhaps add a note as to why or if it could be done in future by changing the addons structure.

The reason that we use isGinkgoProcessNumberOne is that we run ProvisionGlobals on process # 1 in parallel with running Setup on the other processes & starting their test suites. This allows us to start non-vault tests before Vault has been setup.
We could move the isGinkgoProcessNumberOne logic to the Addon implementation instead and set a boolean on the addon struct during Setup that gates whether Provision should do something/ nothing ( not sure if that is better than what we have now, it would just be a style change ).

@irbekrm
Copy link
Copy Markdown
Contributor

irbekrm commented May 2, 2023

The reason that we use isGinkgoProcessNumberOne is that we run ProvisionGlobals on process # 1 in parallel with running Setup on the other processes & starting their test suites. This allows us to start non-vault tests before vault has been setup.
We could move the isGinkgoProcessNumberOne logic to the Addon implementation instead and set a boolean on the addon struct during Setup that gates whether Provision should do something/ nothing ( not sure if that is better than what we have now, it would just be a style change ).

Thanks for the reply. Yeah I don't think a boolean would be better.

/lgtm
/approve

@jetstack-bot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inteon, irbekrm

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/acme/dns01 Indicates a PR modifies ACME DNS01 provider code area/acme Indicates a PR directly modifies the ACME Issuer code area/testing Issues relating to testing dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm Indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants