[Detection Engine] Investigation: Testing whether es_archiver can cause duplicate documents#223043
Closed
rylnd wants to merge 16 commits intoelastic:mainfrom
Closed
[Detection Engine] Investigation: Testing whether es_archiver can cause duplicate documents#223043rylnd wants to merge 16 commits intoelastic:mainfrom
rylnd wants to merge 16 commits intoelastic:mainfrom
Conversation
This integration test is executed 100 times, with the single-document archive being loaded and unloaded before/after each test. If this test fails, it will eliminate the detection engine as a culprit.
Contributor
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
Contributor
Author
|
200 runs (with 100 executions each, for a total of 20_000 executions): https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8349 |
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8349[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 160/200 tests passed. |
Conflicts: x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/index.ts
I don't know if APM is disabled by default in my scenario, but this will ensure it is.
Contributor
Author
|
Attempting to trigger another build with APM manually enabled: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8561 |
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8561[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 71/100 tests passed. |
I think the values in this config will be respected, as per this thread: https://elastic.slack.com/archives/C05UT5PP1EF/p1741788732650719, but we'll find out.
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8577[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 0/10 tests passed. |
At this point I'm just seeing what sticks.
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8597[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 0/10 tests passed. |
After looking more closely at how these env variables interact, I think that the context propagation var needs to explicitly be false in order to allow reporting; I had assumed that this wasn't needed originally. This also removes the duplicated definitions in a substep of the pipeline.
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8600[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 1/10 tests passed. |
Contributor
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8601[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 10/10 tests passed. |
Contributor
Author
|
/ci |
Now that we're running an actual build, we can't get to the tests until it lints 🙄
Contributor
Author
|
/ci |
Contributor
💔 Build Failed
Failed CI StepsTest Failures
Metrics [docs]
History
|
This is copied liberally from the APM performance FTRs.
So that this all happens at the base config, instead.
Contributor
Author
|
New FTR build cribbing APM's FTR config: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8661#01980fd1-6b52-41f2-bdd6-e264f39cae28 |
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8661[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 9/10 tests passed. |
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8662[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 7/10 tests passed. |
* Adds a delay at the end of the run to allow APM to report * Adds server args for telemetryOptIn and labels I assumed telemetry was opt out, but perhaps this was the missing piece.
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8665[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 5/10 tests passed. |
Contributor
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8690[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 10/10 tests passed. |
We set a default key from vault, but my guess is that this is conflicting with the server/token being used to send to the performance APM server.
Contributor
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8691[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 10/10 tests passed. |
I suspect that setting the pipeline env var to `undefined` might not override an existing value, there, so I'm trying to remove any setting of that API key at all (in addition to unsetting the server URL, which _seems_ to be getting overridden elsewhere, to the perf server, but at this point I have no confidence in anything.
Contributor
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8692[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 10/10 tests passed. |
Contributor
Author
|
Okay, with this change I was able to unset the default APM API key, which was causing our reporting to the perf server to be rejected. I'm now able to see my data there, but:
I'm addressing the second point with a new 50x build, and will attempt to clarify the first point with the others involved. |
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8695[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 25/50 tests passed. |
Contributor
Author
|
|
This adds APM around our functional test runner, which should allow us to collect data on what's happening with elasticsearchduring our (failing) test runs.
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8738[❌] x-pack/test/security_solution_api_integration/test_suites/detections_response/detection_engine/rule_execution_logic/general_logic/basic_license_essentials_tier/configs/ess.config.ts: 0/20 tests passed. |
Contributor
Author
|
Closing this for now, in favor of the more productive (and more straightforward) #228850. |
dmlemeshko
added a commit
that referenced
this pull request
Aug 6, 2025
## Summary This PR fixes duplicate document creation in esArchiver by generating an `_id` for index (non-data-stream, non-time-series) documents that don't have an id already. ### Details - Under some circumstances, the `es-helper-bulk` that is used by esArchiver can ingest a duplicate document (just with different id), see investigations [here](#228556) and [here](#223043), also bug report [here](elastic/elasticsearch-js#2924). - With explicitly setting the id, the flakiness didn't show up anymore, which matches the expected behavior as of the [bulk docs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) `A create action fails if a document with the same ID already exists in the target An index action adds or replaces a document as necessary.` - In order to unblock testing, this PR is actually working around the underlying problem, which should still be investigated separately --------- Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Aug 6, 2025
## Summary This PR fixes duplicate document creation in esArchiver by generating an `_id` for index (non-data-stream, non-time-series) documents that don't have an id already. ### Details - Under some circumstances, the `es-helper-bulk` that is used by esArchiver can ingest a duplicate document (just with different id), see investigations [here](elastic#228556) and [here](elastic#223043), also bug report [here](elastic/elasticsearch-js#2924). - With explicitly setting the id, the flakiness didn't show up anymore, which matches the expected behavior as of the [bulk docs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) `A create action fails if a document with the same ID already exists in the target An index action adds or replaces a document as necessary.` - In order to unblock testing, this PR is actually working around the underlying problem, which should still be investigated separately --------- Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co> (cherry picked from commit 42377e4)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Aug 6, 2025
## Summary This PR fixes duplicate document creation in esArchiver by generating an `_id` for index (non-data-stream, non-time-series) documents that don't have an id already. ### Details - Under some circumstances, the `es-helper-bulk` that is used by esArchiver can ingest a duplicate document (just with different id), see investigations [here](elastic#228556) and [here](elastic#223043), also bug report [here](elastic/elasticsearch-js#2924). - With explicitly setting the id, the flakiness didn't show up anymore, which matches the expected behavior as of the [bulk docs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) `A create action fails if a document with the same ID already exists in the target An index action adds or replaces a document as necessary.` - In order to unblock testing, this PR is actually working around the underlying problem, which should still be investigated separately --------- Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co> (cherry picked from commit 42377e4)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Aug 6, 2025
## Summary This PR fixes duplicate document creation in esArchiver by generating an `_id` for index (non-data-stream, non-time-series) documents that don't have an id already. ### Details - Under some circumstances, the `es-helper-bulk` that is used by esArchiver can ingest a duplicate document (just with different id), see investigations [here](elastic#228556) and [here](elastic#223043), also bug report [here](elastic/elasticsearch-js#2924). - With explicitly setting the id, the flakiness didn't show up anymore, which matches the expected behavior as of the [bulk docs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) `A create action fails if a document with the same ID already exists in the target An index action adds or replaces a document as necessary.` - In order to unblock testing, this PR is actually working around the underlying problem, which should still be investigated separately --------- Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co> (cherry picked from commit 42377e4)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Aug 6, 2025
## Summary This PR fixes duplicate document creation in esArchiver by generating an `_id` for index (non-data-stream, non-time-series) documents that don't have an id already. ### Details - Under some circumstances, the `es-helper-bulk` that is used by esArchiver can ingest a duplicate document (just with different id), see investigations [here](elastic#228556) and [here](elastic#223043), also bug report [here](elastic/elasticsearch-js#2924). - With explicitly setting the id, the flakiness didn't show up anymore, which matches the expected behavior as of the [bulk docs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) `A create action fails if a document with the same ID already exists in the target An index action adds or replaces a document as necessary.` - In order to unblock testing, this PR is actually working around the underlying problem, which should still be investigated separately --------- Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co> (cherry picked from commit 42377e4)
kibanamachine
added a commit
that referenced
this pull request
Aug 6, 2025
# Backport This will backport the following commits from `main` to `9.1`: - [FTR - fix esArchiver duplicate doc ingestion (#229457)](#229457) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Robert Oskamp","email":"robert.oskamp@elastic.co"},"sourceCommit":{"committedDate":"2025-08-06T15:28:34Z","message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:version","v9.2.0","v9.0.5","v9.1.1","v8.18.5","v8.19.1"],"title":"FTR - fix esArchiver duplicate doc ingestion","number":229457,"url":"https://github.com/elastic/kibana/pull/229457","mergeCommit":{"message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","9.1","8.18","8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/229457","number":229457,"mergeCommit":{"message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0"}},{"branch":"9.0","label":"v9.0.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Robert Oskamp <robert.oskamp@elastic.co> Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
kibanamachine
added a commit
that referenced
this pull request
Aug 6, 2025
# Backport This will backport the following commits from `main` to `8.19`: - [FTR - fix esArchiver duplicate doc ingestion (#229457)](#229457) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Robert Oskamp","email":"robert.oskamp@elastic.co"},"sourceCommit":{"committedDate":"2025-08-06T15:28:34Z","message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:version","v9.2.0","v9.0.5","v9.1.1","v8.18.5","v8.19.1"],"title":"FTR - fix esArchiver duplicate doc ingestion","number":229457,"url":"https://github.com/elastic/kibana/pull/229457","mergeCommit":{"message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","9.1","8.18","8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/229457","number":229457,"mergeCommit":{"message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0"}},{"branch":"9.0","label":"v9.0.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Robert Oskamp <robert.oskamp@elastic.co> Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
kibanamachine
added a commit
that referenced
this pull request
Aug 6, 2025
# Backport This will backport the following commits from `main` to `9.0`: - [FTR - fix esArchiver duplicate doc ingestion (#229457)](#229457) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Robert Oskamp","email":"robert.oskamp@elastic.co"},"sourceCommit":{"committedDate":"2025-08-06T15:28:34Z","message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:version","v9.2.0","v9.0.5","v9.1.1","v8.18.5","v8.19.1"],"title":"FTR - fix esArchiver duplicate doc ingestion","number":229457,"url":"https://github.com/elastic/kibana/pull/229457","mergeCommit":{"message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","9.1","8.18","8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/229457","number":229457,"mergeCommit":{"message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0"}},{"branch":"9.0","label":"v9.0.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Robert Oskamp <robert.oskamp@elastic.co> Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
kibanamachine
added a commit
that referenced
this pull request
Aug 6, 2025
# Backport This will backport the following commits from `main` to `8.18`: - [FTR - fix esArchiver duplicate doc ingestion (#229457)](#229457) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Robert Oskamp","email":"robert.oskamp@elastic.co"},"sourceCommit":{"committedDate":"2025-08-06T15:28:34Z","message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:version","v9.2.0","v9.0.5","v9.1.1","v8.18.5","v8.19.1"],"title":"FTR - fix esArchiver duplicate doc ingestion","number":229457,"url":"https://github.com/elastic/kibana/pull/229457","mergeCommit":{"message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","9.1","8.18","8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/229457","number":229457,"mergeCommit":{"message":"FTR - fix esArchiver duplicate doc ingestion (#229457)\n\n## Summary\n\nThis PR fixes duplicate document creation in esArchiver by generating an\n`_id` for index (non-data-stream, non-time-series) documents that don't\nhave an id already.\n\n### Details\n\n- Under some circumstances, the `es-helper-bulk` that is used by\nesArchiver can ingest a duplicate document (just with different id), see\ninvestigations [here](#228556) and\n[here](#223043), also bug report\n[here](https://github.com/elastic/elasticsearch-js/issues/2924).\n- With explicitly setting the id, the flakiness didn't show up anymore,\nwhich matches the expected behavior as of the [bulk\ndocs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk)\n`A create action fails if a document with the same ID already exists in\nthe target An index action adds or replaces a document as necessary.`\n- In order to unblock testing, this PR is actually working around the\nunderlying problem, which should still be investigated separately\n\n---------\n\nCo-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>","sha":"42377e498dc7a563367cf1e259ea068e117c9ad0"}},{"branch":"9.0","label":"v9.0.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Robert Oskamp <robert.oskamp@elastic.co> Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
gergoabraham
pushed a commit
to gergoabraham/kibana
that referenced
this pull request
Aug 7, 2025
## Summary This PR fixes duplicate document creation in esArchiver by generating an `_id` for index (non-data-stream, non-time-series) documents that don't have an id already. ### Details - Under some circumstances, the `es-helper-bulk` that is used by esArchiver can ingest a duplicate document (just with different id), see investigations [here](elastic#228556) and [here](elastic#223043), also bug report [here](elastic/elasticsearch-js#2924). - With explicitly setting the id, the flakiness didn't show up anymore, which matches the expected behavior as of the [bulk docs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) `A create action fails if a document with the same ID already exists in the target An index action adds or replaces a document as necessary.` - In order to unblock testing, this PR is actually working around the underlying problem, which should still be investigated separately --------- Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
denar50
pushed a commit
to denar50/kibana
that referenced
this pull request
Aug 8, 2025
## Summary This PR fixes duplicate document creation in esArchiver by generating an `_id` for index (non-data-stream, non-time-series) documents that don't have an id already. ### Details - Under some circumstances, the `es-helper-bulk` that is used by esArchiver can ingest a duplicate document (just with different id), see investigations [here](elastic#228556) and [here](elastic#223043), also bug report [here](elastic/elasticsearch-js#2924). - With explicitly setting the id, the flakiness didn't show up anymore, which matches the expected behavior as of the [bulk docs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) `A create action fails if a document with the same ID already exists in the target An index action adds or replaces a document as necessary.` - In order to unblock testing, this PR is actually working around the underlying problem, which should still be investigated separately --------- Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
NicholasPeretti
pushed a commit
to NicholasPeretti/kibana
that referenced
this pull request
Aug 18, 2025
## Summary This PR fixes duplicate document creation in esArchiver by generating an `_id` for index (non-data-stream, non-time-series) documents that don't have an id already. ### Details - Under some circumstances, the `es-helper-bulk` that is used by esArchiver can ingest a duplicate document (just with different id), see investigations [here](elastic#228556) and [here](elastic#223043), also bug report [here](elastic/elasticsearch-js#2924). - With explicitly setting the id, the flakiness didn't show up anymore, which matches the expected behavior as of the [bulk docs](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) `A create action fails if a document with the same ID already exists in the target An index action adds or replaces a document as necessary.` - In order to unblock testing, this PR is actually working around the underlying problem, which should still be investigated separately --------- Co-authored-by: Dzmitry Lemechko <dzmitry.lemechko@elastic.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This integration test is executed 100 times, with the single-document archive being loaded and unloaded before/after each test. If this test fails, it will eliminate the detection engine as a culprit.