Skip to content

[EDR Workflows] Fix and unskip Unable to create Package Policy via Fleet! FTR api tests#250075

Merged
gergoabraham merged 7 commits intoelastic:mainfrom
gergoabraham:fix-unskip-cannot-create-package-policy-ftr-api-tests
Jan 26, 2026
Merged

[EDR Workflows] Fix and unskip Unable to create Package Policy via Fleet! FTR api tests#250075
gergoabraham merged 7 commits intoelastic:mainfrom
gergoabraham:fix-unskip-cannot-create-package-policy-ftr-api-tests

Conversation

@gergoabraham
Copy link
Copy Markdown
Contributor

@gergoabraham gergoabraham commented Jan 22, 2026

Summary

This PR fixes the flakiness around creating an Endpoint package policy using endpointPolicyTestResources.createPolicy().

When an Endpoint package policy is created, the default rules are installed in the background, which process can be flaky, or, together with other functionalities, simply too slow (i.e. taking more than 2 minutes, causing socket hang up error). Therefore, there's a workaround by the Detections team to install a mock security_detection_engine package. This fix workaround is applied in the EDR API tests as well.

For more info, see the original workaround PR:

Flaky runner: 25/25 ✅ for all 6 test groups:

  • artifacts ESS + serverless
  • artifacts with Endpoint exceptions moved FF enabled ESS + serverless
  • spaces ESS + serverless

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/10576/steps/canvas

fix #249780
fix #249463
fix #249468
fix #248923
fix #248759
fix #248578
fix #248554
fix #247416
fix #247211
fix #246564
fix #246341
fix #246450
fix #246356
fix #246207

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

@gergoabraham gergoabraham self-assigned this Jan 22, 2026
@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#10572

[❌] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/configs/endpoint_exceptions_moved_ff.ess.config.ts: 0/25 tests passed.
[❌] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/configs/endpoint_exceptions_moved_ff.serverless.config.ts: 24/25 tests passed.
[❌] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/configs/ess.config.ts: 0/25 tests passed.
[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/configs/serverless.config.ts: 25/25 tests passed.
[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/policy/trial_license_complete_tier/configs/ess.config.ts: 25/25 tests passed.
[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/policy/trial_license_complete_tier/configs/serverless.config.ts: 25/25 tests passed.

see run history

@kibanamachine
Copy link
Copy Markdown
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#10576

[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/configs/endpoint_exceptions_moved_ff.ess.config.ts: 25/25 tests passed.
[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/configs/endpoint_exceptions_moved_ff.serverless.config.ts: 25/25 tests passed.
[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/configs/ess.config.ts: 25/25 tests passed.
[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/configs/serverless.config.ts: 25/25 tests passed.
[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/policy/trial_license_complete_tier/configs/ess.config.ts: 25/25 tests passed.
[✅] x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/policy/trial_license_complete_tier/configs/serverless.config.ts: 25/25 tests passed.

see run history

@gergoabraham gergoabraham added release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution backport:all-open Backport to all branches that could still receive a release labels Jan 23, 2026
@gergoabraham gergoabraham marked this pull request as ready for review January 23, 2026 13:19
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

cc @gergoabraham

@gergoabraham gergoabraham merged commit d36d440 into elastic:main Jan 26, 2026
16 checks passed
@kibanamachine
Copy link
Copy Markdown
Contributor

Starting backport for target branches: 8.19, 9.1, 9.2, 9.3

https://github.com/elastic/kibana/actions/runs/21363943112

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jan 26, 2026
…leet!` FTR api tests (elastic#250075)

## Summary

This PR fixes the flakiness around creating an Endpoint package policy
using `endpointPolicyTestResources.createPolicy()`.

When an Endpoint package policy is created, the default rules are
installed in the background, which process can be flaky, or, together
with other functionalities, simply too slow (i.e. taking more than 2
minutes, causing `socket hang up` error). Therefore, there's a
workaround by the Detections team to install a mock
`security_detection_engine` package. This fix workaround is applied in
the EDR API tests as well.

For more info, see the original workaround PR:
- elastic#227689

Flaky runner: 25/25 ✅ for all 6 test groups:
- artifacts ESS + serverless
- artifacts with Endpoint exceptions moved FF enabled ESS + serverless
- spaces ESS + serverless

https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/10576/steps/canvas

fix elastic#249780
fix elastic#249463
fix elastic#249468
fix elastic#248923
fix elastic#248759
fix elastic#248578
fix elastic#248554
fix elastic#247416
fix elastic#247211
fix elastic#246564
fix elastic#246341
fix elastic#246450
fix elastic#246356
fix elastic#246207

### Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

(cherry picked from commit d36d440)
@kibanamachine
Copy link
Copy Markdown
Contributor

💔 Some backports could not be created

Status Branch Result
8.19 Backport failed because of merge conflicts
9.1 Backport failed because of merge conflicts
9.2 Backport failed because of merge conflicts
9.3

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 250075

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Jan 26, 2026
… via Fleet!` FTR api tests (#250075) (#250450)

# Backport

This will backport the following commits from `main` to `9.3`:
- [[EDR Workflows] Fix and unskip `Unable to create Package Policy via
Fleet!` FTR api tests
(#250075)](#250075)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Gergő
Ábrahám","email":"gergo.abraham@elastic.co"},"sourceCommit":{"committedDate":"2026-01-26T15:46:47Z","message":"[EDR
Workflows] Fix and unskip `Unable to create Package Policy via Fleet!`
FTR api tests (#250075)\n\n## Summary\n\nThis PR fixes the flakiness
around creating an Endpoint package policy\nusing
`endpointPolicyTestResources.createPolicy()`.\n\nWhen an Endpoint
package policy is created, the default rules are\ninstalled in the
background, which process can be flaky, or, together\nwith other
functionalities, simply too slow (i.e. taking more than 2\nminutes,
causing `socket hang up` error). Therefore, there's a\nworkaround by the
Detections team to install a mock\n`security_detection_engine` package.
This fix workaround is applied in\nthe EDR API tests as well.\n\nFor
more info, see the original workaround PR:\n-
https://github.com/elastic/kibana/pull/227689\n\nFlaky runner: 25/25 ✅
for all 6 test groups:\n- artifacts ESS + serverless\n- artifacts with
Endpoint exceptions moved FF enabled ESS + serverless\n- spaces ESS +
serverless\n\n\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/10576/steps/canvas\n\nfix
#249780\nfix #249463\nfix #249468\nfix #248923\nfix #248759\nfix
#248578\nfix #248554\nfix #247416\nfix #247211\nfix #246564\nfix
#246341\nfix #246450\nfix #246356\nfix #246207\n\n### Checklist\n\nCheck
the PR satisfies following conditions. \n\nReviewers should verify this
PR satisfies this list as well.\n\n- [x] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests
changed","sha":"d36d440e1dff84f0ca2306c97dd08fed16259c22","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Defend
Workflows","backport:all-open","v9.4.0"],"title":"[EDR Workflows] Fix
and unskip `Unable to create Package Policy via Fleet!` FTR api
tests","number":250075,"url":"https://github.com/elastic/kibana/pull/250075","mergeCommit":{"message":"[EDR
Workflows] Fix and unskip `Unable to create Package Policy via Fleet!`
FTR api tests (#250075)\n\n## Summary\n\nThis PR fixes the flakiness
around creating an Endpoint package policy\nusing
`endpointPolicyTestResources.createPolicy()`.\n\nWhen an Endpoint
package policy is created, the default rules are\ninstalled in the
background, which process can be flaky, or, together\nwith other
functionalities, simply too slow (i.e. taking more than 2\nminutes,
causing `socket hang up` error). Therefore, there's a\nworkaround by the
Detections team to install a mock\n`security_detection_engine` package.
This fix workaround is applied in\nthe EDR API tests as well.\n\nFor
more info, see the original workaround PR:\n-
https://github.com/elastic/kibana/pull/227689\n\nFlaky runner: 25/25 ✅
for all 6 test groups:\n- artifacts ESS + serverless\n- artifacts with
Endpoint exceptions moved FF enabled ESS + serverless\n- spaces ESS +
serverless\n\n\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/10576/steps/canvas\n\nfix
#249780\nfix #249463\nfix #249468\nfix #248923\nfix #248759\nfix
#248578\nfix #248554\nfix #247416\nfix #247211\nfix #246564\nfix
#246341\nfix #246450\nfix #246356\nfix #246207\n\n### Checklist\n\nCheck
the PR satisfies following conditions. \n\nReviewers should verify this
PR satisfies this list as well.\n\n- [x] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests
changed","sha":"d36d440e1dff84f0ca2306c97dd08fed16259c22"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/250075","number":250075,"mergeCommit":{"message":"[EDR
Workflows] Fix and unskip `Unable to create Package Policy via Fleet!`
FTR api tests (#250075)\n\n## Summary\n\nThis PR fixes the flakiness
around creating an Endpoint package policy\nusing
`endpointPolicyTestResources.createPolicy()`.\n\nWhen an Endpoint
package policy is created, the default rules are\ninstalled in the
background, which process can be flaky, or, together\nwith other
functionalities, simply too slow (i.e. taking more than 2\nminutes,
causing `socket hang up` error). Therefore, there's a\nworkaround by the
Detections team to install a mock\n`security_detection_engine` package.
This fix workaround is applied in\nthe EDR API tests as well.\n\nFor
more info, see the original workaround PR:\n-
https://github.com/elastic/kibana/pull/227689\n\nFlaky runner: 25/25 ✅
for all 6 test groups:\n- artifacts ESS + serverless\n- artifacts with
Endpoint exceptions moved FF enabled ESS + serverless\n- spaces ESS +
serverless\n\n\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/10576/steps/canvas\n\nfix
#249780\nfix #249463\nfix #249468\nfix #248923\nfix #248759\nfix
#248578\nfix #248554\nfix #247416\nfix #247211\nfix #246564\nfix
#246341\nfix #246450\nfix #246356\nfix #246207\n\n### Checklist\n\nCheck
the PR satisfies following conditions. \n\nReviewers should verify this
PR satisfies this list as well.\n\n- [x] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests
changed","sha":"d36d440e1dff84f0ca2306c97dd08fed16259c22"}}]}]
BACKPORT-->

Co-authored-by: Gergő Ábrahám <gergo.abraham@elastic.co>
@gergoabraham
Copy link
Copy Markdown
Contributor Author

💔 Some backports could not be created

Status Branch Result
❌ 8.19 Backport failed because of merge conflicts
❌ 9.1 Backport failed because of merge conflicts
❌ 9.2 Backport failed because of merge conflicts
✅ 9.3
Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 250075

Questions ?

Please refer to the Backport tool documentation

Note

all tests are enabled and non-flaky on 8.19/9.2, so no need to backport to those versions ✅

@gergoabraham gergoabraham deleted the fix-unskip-cannot-create-package-policy-ftr-api-tests branch February 2, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:all-open Backport to all branches that could still receive a release release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v9.3.0 v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Failing test: EDR Workflows API - Policy Tests - ESS Env - Trial License.x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/policy/trial_license_complete_tier/datastream_index_creation·ts - Endpoint Policy @ess @serverless Creation of DOT indices for elastic defend policies "before each" hook for "should create indices when endpoint integration policy is created" Failing test: EDR Workflows - Artifacts Integration Tests - ESS Env - Trial License.x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/trusted_apps·ts - Endpoint plugin @ess @serverless @skipInServerlessMKI Endpoint artifacts (via lists plugin): Trusted Applications "before all" hook in "@ess @serverless @skipInServerlessMKI Endpoint artifacts (via lists plugin): Trusted Applications" Failing test: EDR Workflows - Artifacts Integration Tests - Serverless Env - Complete.x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/trusted_apps·ts - Endpoint plugin @ess @serverless @skipInServerlessMKI Endpoint artifacts (via lists plugin): Trusted Applications "before all" hook in "@ess @serverless @skipInServerlessMKI Endpoint artifacts (via lists plugin): Trusted Applications" Failing test: EDR Workflows API - Policy Tests - Serverless Env - Complete.x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/policy/trial_license_complete_tier/datastream_index_creation·ts - Endpoint Policy @ess @serverless Creation of DOT indices for elastic defend policies "before each" hook for "should create indices when endpoint integration policy is created" Failing test: EDR Workflows - Endpoint Exceptions Integration Tests - Serverless Env - Complete.x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/endpoint_exceptions·ts - Endpoint Exceptions with feature flag enabled @ess @serverless @skipInServerlessMKI Endpoint artifacts (via lists plugin): Endpoint Exceptions "before all" hook in "@ess @serverless @skipInServerlessMKI Endpoint artifacts (via lists plugin): Endpoint Exceptions"
Failing test: EDR Workflows - Endpoint Exceptions Integration Tests - Serverless Env - Complete.x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/endpoint_list_api_rbac·ts - Endpoint Exceptions with feature flag enabled @ess @serverless @skipInServerlessMKI Endpoint List API (deprecated): RBAC and Validation "before all" hook in "@ess @serverless @skipInServerlessMKI Endpoint List API (deprecated): RBAC and Validation" Failing test: EDR Workflows - Endpoint Exceptions Integration Tests - ESS Env - Trial License.x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/endpoint_list_api_rbac·ts - Endpoint Exceptions with feature flag enabled @ess @serverless @skipInServerlessMKI Endpoint List API (deprecated): RBAC and Validation "before all" hook in "@ess @serverless @skipInServerlessMKI Endpoint List API (deprecated): RBAC and Validation" Failing test: EDR Workflows - Artifacts Integration Tests - ESS Env - Trial License.x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/trusted_devices·ts - Endpoint plugin @ess @serverless @skipInServerlessMKI Endpoint artifacts (via lists plugin): Trusted Devices "before all" hook in "@ess @serverless @skipInServerlessMKI Endpoint artifacts (via lists plugin): Trusted Devices" Failing test: EDR Workflows - Artifacts Integration Tests - Serverless Env - Complete.x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/trusted_devices·ts - Endpoint plugin @ess @serverless @skipInServerlessMKI Endpoint artifacts (via lists plugin): Trusted Devices "before all" hook in "@ess @serverless @skipInServerlessMKI Endpoint artifacts (via lists plugin): Trusted Devices" Failing test: EDR Workflows - Artifacts Integration Tests - ESS Env - Trial License.x-pack/solutions/security/test/security_solution_api_integration/test_suites/edr_workflows/artifacts/trial_license_complete_tier/event_filters·ts - Endpoint plugin @ess @serverless @skipInServerlessMKI Endpoint artifacts (via lists plugin): Event Filters "before all" hook for "should return 400 for import of endpoint exceptions"

5 participants