Skip to content

Commit 07a83d6

Browse files
MadameSheemaMadameSheema
authored
[Security Solution] Adds serverlessQA tag to the Cypress tests (#179737)
## Summary We want to start integrating our Cypress tests with the serverless Kibana quality gate. However, not all the teams feel comfortable enabling all the tests, to facilitate the effort of enabling tests in the quality gate we are adding the `@serverlessQA` tag. Once this PR is merged, the behavior will be the following: **Regular PR flow** All the tests tagged as `@serverless` will be executed as part of the PR validation process using the serverless FTR environment (not a real one). If you want to skip a test to be executed in this flow use `@brokenInServerless` or `@skipInServerless`. **Periodic pipeline** All the tests tagged as `@serverless` will be executed as part of the periodic pipeline using a real serverless project. QA environment is used to do so using the latest available commit in main at the time of the execution. If you want to skip a test to be executed in this flow use `@brokenInServerlessQA`. **Kibana second quality gate** All the tests tagged as `@serverlessQA` will be executed as part of the kibana release process using a real serverless project with the latest image available in the QA environment. If you want to skip a test to be executed in this flow use `@brokenInServerlessQA`.
1 parent 0527f48 commit 07a83d6

43 files changed

Lines changed: 259 additions & 261 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -459,10 +459,16 @@ ${JSON.stringify(argv, null, 2)}
459459
`);
460460

461461
const isOpen = argv._.includes('open');
462-
463462
const cypressConfigFilePath = require.resolve(`../../${argv.configFile}`) as string;
464463
const cypressConfigFile = await import(cypressConfigFilePath);
465-
464+
// KIBANA_MKI_USE_LATEST_COMMIT === 1 means that we are overriding the image for the periodic pipeline execution.
465+
// We don't override the image when executing the tests on the second quality gate.
466+
if (
467+
!process.env.KIBANA_MKI_USE_LATEST_COMMIT ||
468+
process.env.KIBANA_MKI_USE_LATEST_COMMIT !== '1'
469+
) {
470+
cypressConfigFile.env.grepTags = '@serverlessQA --@skipInServerless';
471+
}
466472
const tier: string = argv.tier;
467473
const endpointAddon: boolean = argv.endpointAddon;
468474
const cloudAddon: boolean = argv.cloudAddon;

‎x-pack/test/security_solution_cypress/cypress/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -44,11 +44,11 @@ Please, before opening a PR with the new test, please make sure that the test fa
4444

4545
Note that we use tags in order to select which tests we want to execute:
4646

47-
- `@serverless` includes a test in the Serverless test suite for PRs (the so-called first quality gate) and QA environemnt (the so-called second quality gate). You need to explicitly add this tag to any test you want to run in CI for serverless.
47+
- `@serverless` includes a test in the Serverless test suite for PRs (the so-called first quality gate) and QA environment for the periodic pipeline. You need to explicitly add this tag to any test you want to run in CI for serverless.
48+
- `@serverlessQA` includes a test in the Serverless test suite for the Kibana release process of serverless. You need to explicitly add this tag to any test you want yo run in CI for the second quality gate. These tests should be stable, otherviswe they will be blocking the release pipeline. They should be alsy critical enough, so that when they fail, there's a high chance of an SDH or blocker issue to be reported.
4849
- `@ess` includes a test in the normal, non-Serverless test suite. You need to explicitly add this tag to any test you want to run against a non-Serverless environment.
49-
- `@brokenInServerless` excludes a test from the Serverless test suite (even if it's tagged as `@serverless`). Indicates that a test should run in Serverless, but currently is broken.
50-
- `@brokenInServerlessQA` excludes a test form the Serverless QA enviornment (second quality gate). Indicates that a test should run on it, but currently is broken.
51-
- `@skipInServerless` excludes a test from the Serverless test suite (even if it's tagged as `@serverless`). Could indicate many things, e.g. "the test is flaky in Serverless", "the test is Flaky in any type of environemnt", "the test has been temporarily excluded, see the comment above why".
50+
- `@skipInEss` excludes a test from the non-Serverless test suite. The test will not be executed as part for the PR process. All the skipped tests should have a link to a ticket describing the reason why the test got skipped.
51+
- `@skipInServerless` excludes a test from the Serverless test suite and Serverless QA environment for both, periodic pipeline and second quality gate (even if it's tagged as `@serverless`). Could indicate many things, e.g. "the test is flaky in Serverless", "the test is Flaky in any type of environment", "the test has been temporarily excluded, see the comment above why". All the skipped tests should have a link to a ticket describing the reason why the test got skipped.
5252

5353
Please, before opening a PR with a new test, make sure that the test fails. If you never see your test fail you don’t know if your test is actually testing the right thing, or testing anything at all.
5454

‎x-pack/test/security_solution_cypress/cypress/cypress.config.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ export default defineCypressConfig({
1212
defaultCommandTimeout: 60000,
1313
env: {
1414
grepFilterSpecs: true,
15-
grepTags: '@ess',
15+
grepTags: '@ess --@skipInEss',
1616
},
1717
execTimeout: 60000,
1818
pageLoadTimeout: 60000,

‎x-pack/test/security_solution_cypress/cypress/cypress_ci.config.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ export default defineCypressConfig({
1818
env: {
1919
grepFilterSpecs: true,
2020
grepOmitFiltered: true,
21-
grepTags: '@ess',
21+
grepTags: '@ess --@skipInEss',
2222
},
2323
execTimeout: 150000,
2424
pageLoadTimeout: 150000,

‎x-pack/test/security_solution_cypress/cypress/cypress_ci_serverless.config.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ export default defineCypressConfig({
1919
env: {
2020
grepFilterSpecs: true,
2121
grepOmitFiltered: true,
22-
grepTags: '@serverless --@brokenInServerless --@skipInServerless',
22+
grepTags: '@serverless --@skipInServerless',
2323
},
2424
execTimeout: 150000,
2525
pageLoadTimeout: 150000,

‎x-pack/test/security_solution_cypress/cypress/cypress_ci_serverless_qa.config.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ export default defineCypressConfig({
1919
env: {
2020
grepFilterSpecs: true,
2121
grepOmitFiltered: true,
22-
grepTags: '@serverless --@brokenInServerless --@skipInServerless --@brokenInServerlessQA',
22+
grepTags: '@serverless --@skipInServerless',
2323
},
2424
execTimeout: 300000,
2525
pageLoadTimeout: 300000,

‎x-pack/test/security_solution_cypress/cypress/cypress_serverless.config.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ export default defineCypressConfig({
2424
numTestsKeptInMemory: 10,
2525
env: {
2626
grepFilterSpecs: true,
27-
grepTags: '@serverless --@brokenInServerless --@skipInServerless',
27+
grepTags: '@serverless --@skipInServerless',
2828
},
2929
e2e: {
3030
experimentalCspAllowList: ['default-src', 'script-src', 'script-src-elem'],

‎x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/detection_alerts/callouts/missing_privileges_callout.cy.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ const waitForPageTitleToBeShown = () => {
4747
// FLAKY: https://github.com/elastic/kibana/issues/178176
4848
describe.skip(
4949
'Detections > Callouts',
50-
{ tags: ['@ess', '@serverless', '@brokenInServerlessQA'] },
50+
{ tags: ['@ess', '@serverless', '@skipInServerless'] },
5151
() => {
5252
before(() => {
5353
// First, we have to open the app on behalf of a privileged user in order to initialize it.

0 commit comments

Comments
 (0)