Skip to content

[FTR] Add integration tests to verify FTR exit code preservation#243630

Merged
csr merged 2 commits intoelastic:mainfrom
csr:add-ftr-exist-code-integration-test
Nov 20, 2025
Merged

[FTR] Add integration tests to verify FTR exit code preservation#243630
csr merged 2 commits intoelastic:mainfrom
csr:add-ftr-exist-code-integration-test

Conversation

@csr
Copy link
Copy Markdown
Member

@csr csr commented Nov 20, 2025 •
edited by kibanamachine
Loading

This PR adds two integration tests to verify that FTR preserves exit codes correctly when cleanup handlers are registered via @kbn/cleanup-before-exit.

This test would have caught the bug fixed in #243499, where cleanupBeforeExit was accidentally resetting the exit code to 0 when tests failed.

@csr csr requested a review from dgieselaar November 20, 2025 11:35
@csr csr self-assigned this Nov 20, 2025
@csr csr added release_note:skip Skip the PR/issue when compiling release notes backport:all-open Backport to all branches that could still receive a release labels Nov 20, 2025
@csr csr marked this pull request as ready for review November 20, 2025 11:35
@csr csr requested a review from a team as a code owner November 20, 2025 11:35
@csr csr requested a review from a team November 20, 2025 11:39
@csr csr changed the title [FTR] Add integration test for exit code preservation [FTR] Add integration tests to verify FTR exit code preservation Nov 20, 2025
fake-haris
fake-haris reviewed Nov 20, 2025
View reviewed changes
...bn-test/src/functional_test_runner/integration_tests/__fixtures__/exit_code/tests/failing.js
export default function () {
describe('exit code test suite', () => {
it('intentionally fails to verify exit code is preserved', () => {
throw new Error('intentional failure for exit code test');
Copy link
Copy Markdown
Contributor

@fake-haris fake-haris Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe instead of throwing an error, a failing assertion is needed in order to simulate the behavior caused the status code issue.

Copy link
Copy Markdown
Member Author

@csr csr Nov 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fake-haris - while your suggestion will also work, Mocha treats both failed assertions and thrown exceptions in the same way and will consider the test as failing. Either will cause the test to fail and set the exit code to 1, which is what we're testing.

dgieselaar
dgieselaar reviewed Nov 20, 2025
View reviewed changes
...form/packages/shared/kbn-test/src/functional_test_runner/integration_tests/exit_code.test.js
// The cleanup handler must not override this exit code.
it('exits with code 0 when tests pass', () => {
const proc = spawnSync(process.execPath, [SCRIPT, '--config', PASSING_CONFIG], {
env: { ...process.env, SCOUT_REPORTER_ENABLED: '0' },
Copy link
Copy Markdown
Contributor

@dgieselaar dgieselaar Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have you considered setting TEST_ES_DISABLE_STARTUP, and adding/setting TEST_KIBANA_DISABLE_STARTUP, to speed up the test? or does this config already ensure no services are started?

Copy link
Copy Markdown
Member Author

@csr csr Nov 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dgieselaar - the current configs (config_passing.js and config_failing.js) do not start any services. They're vary barebones, they just set a dummy Elasticsearch port under servers (which is required).

fake-haris
fake-haris approved these changes Nov 20, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@fake-haris fake-haris left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@csr csr enabled auto-merge (squash) November 20, 2025 12:15
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Nov 20, 2025 •
edited
Loading

💔 Build Failed

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #21 / Agent Builder agents Edit agent should edit agent name
  • [job] [logs] FTR Configs #52 / Entity Analytics - Privilege Monitoring @ess @serverless @skipInServerlessMKI Entity Privilege Monitoring Engine Integrations Sync integrations sync should update and create users within lastProcessedMarker range during update detection
  • [job] [logs] FTR Configs #52 / Entity Analytics - Privilege Monitoring @ess @serverless @skipInServerlessMKI Entity Privilege Monitoring Engine Integrations Sync integrations sync should update and create users within lastProcessedMarker range during update detection
  • [job] [logs] FTR Configs #112 / Search Profiler Profile should return profile results for a valid index
  • [job] [logs] FTR Configs #82 / Serverless Observability feature flag testing - Deployment-agnostic Observability Agent API integration tests Observability Agent tool: observability.get_anomaly_detection_jobs "after all" hook for "returns job without anomalies when time range excludes them"
  • [job] [logs] FTR Configs #82 / Serverless Observability feature flag testing - Deployment-agnostic Observability Agent API integration tests Observability Agent tool: observability.get_anomaly_detection_jobs "after all" hook for "returns job without anomalies when time range excludes them"
  • [job] [logs] FTR Configs #82 / Serverless Observability feature flag testing - Deployment-agnostic Observability Agent API integration tests Observability Agent tool: observability.get_anomaly_detection_jobs "before all" hook for "filters by specific job ID"
  • [job] [logs] FTR Configs #82 / Serverless Observability feature flag testing - Deployment-agnostic Observability Agent API integration tests Observability Agent tool: observability.get_anomaly_detection_jobs "before all" hook for "filters by specific job ID"

Metrics [docs]

✅ unchanged

History

cc @csr

@csr csr merged commit 71a871a into elastic:main Nov 20, 2025
12 checks passed
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Nov 20, 2025

Starting backport for target branches: 8.19, 9.1, 9.2

https://github.com/elastic/kibana/actions/runs/19540830324

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 20, 2025
@csr
[FTR] Add integration tests to verify FTR exit code preservation (ela…
91ac60f 
…stic#243630)

This PR adds two integration tests to verify that FTR preserves exit
codes correctly when cleanup handlers are registered via
`@kbn/cleanup-before-exit`.

This test would have caught the bug fixed in elastic#243499, where
`cleanupBeforeExit` was accidentally resetting the exit code to 0 when
tests failed.

(cherry picked from commit 71a871a)
@kibanamachine kibanamachine mentioned this pull request Nov 20, 2025
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 20, 2025
@csr
[FTR] Add integration tests to verify FTR exit code preservation (ela…
3c7ab92 
…stic#243630)

This PR adds two integration tests to verify that FTR preserves exit
codes correctly when cleanup handlers are registered via
`@kbn/cleanup-before-exit`.

This test would have caught the bug fixed in elastic#243499, where
`cleanupBeforeExit` was accidentally resetting the exit code to 0 when
tests failed.

(cherry picked from commit 71a871a)
@kibanamachine kibanamachine mentioned this pull request Nov 20, 2025
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Nov 20, 2025
@csr
[FTR] Add integration tests to verify FTR exit code preservation (ela…
dcfe3e2 
…stic#243630)

This PR adds two integration tests to verify that FTR preserves exit
codes correctly when cleanup handlers are registered via
`@kbn/cleanup-before-exit`.

This test would have caught the bug fixed in elastic#243499, where
`cleanupBeforeExit` was accidentally resetting the exit code to 0 when
tests failed.

(cherry picked from commit 71a871a)
@kibanamachine kibanamachine mentioned this pull request Nov 20, 2025
Merged
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Nov 20, 2025

💚 All backports created successfully

Status Branch Result
✅ 8.19
✅ 9.1
✅ 9.2

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

andrimal pushed a commit to andrimal/kibana that referenced this pull request Nov 20, 2025
@csr @andrimal
[FTR] Add integration tests to verify FTR exit code preservation (ela…
59387e4 
…stic#243630)

This PR adds two integration tests to verify that FTR preserves exit
codes correctly when cleanup handlers are registered via
`@kbn/cleanup-before-exit`.

This test would have caught the bug fixed in elastic#243499, where
`cleanupBeforeExit` was accidentally resetting the exit code to 0 when
tests failed.
kibanamachine added a commit that referenced this pull request Nov 20, 2025
@kibanamachine @csr
[9.1] [FTR] Add integration tests to verify FTR exit code preservation (
71710e3 
#243630) (#243673)

# Backport

This will backport the following commits from `main` to `9.1`:
- [[FTR] Add integration tests to verify FTR exit code preservation
(#243630)](#243630)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Cesare de
Cal","email":"cesare.decal@elastic.co"},"sourceCommit":{"committedDate":"2025-11-20T14:46:51Z","message":"[FTR]
Add integration tests to verify FTR exit code preservation
(#243630)\n\nThis PR adds two integration tests to verify that FTR
preserves exit\ncodes correctly when cleanup handlers are registered
via\n`@kbn/cleanup-before-exit`.\n\nThis test would have caught the bug
fixed in #243499, where\n`cleanupBeforeExit` was accidentally resetting
the exit code to 0 when\ntests
failed.","sha":"71a871a3c402387be2b4040f2728f850eea17f7e","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:all-open","v9.3.0"],"title":"[FTR]
Add integration tests to verify FTR exit code
preservation","number":243630,"url":"https://github.com/elastic/kibana/pull/243630","mergeCommit":{"message":"[FTR]
Add integration tests to verify FTR exit code preservation
(#243630)\n\nThis PR adds two integration tests to verify that FTR
preserves exit\ncodes correctly when cleanup handlers are registered
via\n`@kbn/cleanup-before-exit`.\n\nThis test would have caught the bug
fixed in #243499, where\n`cleanupBeforeExit` was accidentally resetting
the exit code to 0 when\ntests
failed.","sha":"71a871a3c402387be2b4040f2728f850eea17f7e"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/243630","number":243630,"mergeCommit":{"message":"[FTR]
Add integration tests to verify FTR exit code preservation
(#243630)\n\nThis PR adds two integration tests to verify that FTR
preserves exit\ncodes correctly when cleanup handlers are registered
via\n`@kbn/cleanup-before-exit`.\n\nThis test would have caught the bug
fixed in #243499, where\n`cleanupBeforeExit` was accidentally resetting
the exit code to 0 when\ntests
failed.","sha":"71a871a3c402387be2b4040f2728f850eea17f7e"}}]}]
BACKPORT-->

Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
kibanamachine added a commit that referenced this pull request Nov 20, 2025
@kibanamachine @csr
[9.2] [FTR] Add integration tests to verify FTR exit code preservation (
af96f31 
#243630) (#243674)

# Backport

This will backport the following commits from `main` to `9.2`:
- [[FTR] Add integration tests to verify FTR exit code preservation
(#243630)](#243630)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Cesare de
Cal","email":"cesare.decal@elastic.co"},"sourceCommit":{"committedDate":"2025-11-20T14:46:51Z","message":"[FTR]
Add integration tests to verify FTR exit code preservation
(#243630)\n\nThis PR adds two integration tests to verify that FTR
preserves exit\ncodes correctly when cleanup handlers are registered
via\n`@kbn/cleanup-before-exit`.\n\nThis test would have caught the bug
fixed in #243499, where\n`cleanupBeforeExit` was accidentally resetting
the exit code to 0 when\ntests
failed.","sha":"71a871a3c402387be2b4040f2728f850eea17f7e","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:all-open","v9.3.0"],"title":"[FTR]
Add integration tests to verify FTR exit code
preservation","number":243630,"url":"https://github.com/elastic/kibana/pull/243630","mergeCommit":{"message":"[FTR]
Add integration tests to verify FTR exit code preservation
(#243630)\n\nThis PR adds two integration tests to verify that FTR
preserves exit\ncodes correctly when cleanup handlers are registered
via\n`@kbn/cleanup-before-exit`.\n\nThis test would have caught the bug
fixed in #243499, where\n`cleanupBeforeExit` was accidentally resetting
the exit code to 0 when\ntests
failed.","sha":"71a871a3c402387be2b4040f2728f850eea17f7e"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/243630","number":243630,"mergeCommit":{"message":"[FTR]
Add integration tests to verify FTR exit code preservation
(#243630)\n\nThis PR adds two integration tests to verify that FTR
preserves exit\ncodes correctly when cleanup handlers are registered
via\n`@kbn/cleanup-before-exit`.\n\nThis test would have caught the bug
fixed in #243499, where\n`cleanupBeforeExit` was accidentally resetting
the exit code to 0 when\ntests
failed.","sha":"71a871a3c402387be2b4040f2728f850eea17f7e"}}]}]
BACKPORT-->

Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
kibanamachine added a commit that referenced this pull request Nov 20, 2025
@kibanamachine @csr
[8.19] [FTR] Add integration tests to verify FTR exit code preservati…
3fbd530 
…on (#243630) (#243672)

# Backport

This will backport the following commits from `main` to `8.19`:
- [[FTR] Add integration tests to verify FTR exit code preservation
(#243630)](#243630)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Cesare de
Cal","email":"cesare.decal@elastic.co"},"sourceCommit":{"committedDate":"2025-11-20T14:46:51Z","message":"[FTR]
Add integration tests to verify FTR exit code preservation
(#243630)\n\nThis PR adds two integration tests to verify that FTR
preserves exit\ncodes correctly when cleanup handlers are registered
via\n`@kbn/cleanup-before-exit`.\n\nThis test would have caught the bug
fixed in #243499, where\n`cleanupBeforeExit` was accidentally resetting
the exit code to 0 when\ntests
failed.","sha":"71a871a3c402387be2b4040f2728f850eea17f7e","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:all-open","v9.3.0"],"title":"[FTR]
Add integration tests to verify FTR exit code
preservation","number":243630,"url":"https://github.com/elastic/kibana/pull/243630","mergeCommit":{"message":"[FTR]
Add integration tests to verify FTR exit code preservation
(#243630)\n\nThis PR adds two integration tests to verify that FTR
preserves exit\ncodes correctly when cleanup handlers are registered
via\n`@kbn/cleanup-before-exit`.\n\nThis test would have caught the bug
fixed in #243499, where\n`cleanupBeforeExit` was accidentally resetting
the exit code to 0 when\ntests
failed.","sha":"71a871a3c402387be2b4040f2728f850eea17f7e"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/243630","number":243630,"mergeCommit":{"message":"[FTR]
Add integration tests to verify FTR exit code preservation
(#243630)\n\nThis PR adds two integration tests to verify that FTR
preserves exit\ncodes correctly when cleanup handlers are registered
via\n`@kbn/cleanup-before-exit`.\n\nThis test would have caught the bug
fixed in #243499, where\n`cleanupBeforeExit` was accidentally resetting
the exit code to 0 when\ntests
failed.","sha":"71a871a3c402387be2b4040f2728f850eea17f7e"}}]}]
BACKPORT-->

Co-authored-by: Cesare de Cal <cesare.decal@elastic.co>
@csr csr deleted the add-ftr-exist-code-integration-test branch November 27, 2025 09:48
eokoneyo pushed a commit to eokoneyo/kibana that referenced this pull request Dec 2, 2025
@csr @eokoneyo
[FTR] Add integration tests to verify FTR exit code preservation (ela…
0da9e94 
…stic#243630)

This PR adds two integration tests to verify that FTR preserves exit
codes correctly when cleanup handlers are registered via
`@kbn/cleanup-before-exit`.

This test would have caught the bug fixed in elastic#243499, where
`cleanupBeforeExit` was accidentally resetting the exit code to 0 when
tests failed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fake-haris fake-haris fake-haris approved these changes

+1 more reviewer

@dgieselaar dgieselaar dgieselaar left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

@csr csr

Labels

backport:all-open Backport to all branches that could still receive a release release_note:skip Skip the PR/issue when compiling release notes v8.19.8 v9.1.8 v9.2.2 v9.3.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@csr @elasticmachine @kibanamachine @dgieselaar @fake-haris