…1439)

Fixes: elastic#166376 (main ticket)
Fixes: elastic#186544 (handling of
specific fields)
Fixes: elastic#180195 (replace PATCH
with PUT logic on rule upgrade)

## Summary

- Enhances the `/upgrade/_perform` endpoint to upgrade rules in a way
that works with prebuilt rules customized by users and resolve conflicts
between user customizations and updates from Elastic.
- Handles special fields under the hood (see below)
- Replaces the update prebuilt rule logic to work with PUT instead of
PATCH.

### Rough implementation plan
- For each `upgradeableRule`, we attempt to build the payload necessary
to pass to `upgradePrebuiltRules()`, which is of type
`PrebuiltRuleAsset`. So we retrieve the field names from
`FIELDS_PAYLOAD_BY_RULE_TYPE` and loop through them.
- If any of those `field`s are non-upgreadable, (i.e. its value needs to
be handled under the hood) we do so in `determineFieldUpgradeStatus`.
- Otherwise, we continue to build a `FieldUpgradeSpecifier` for each
field, which will help us determine if that field needs to be set to the
base, current, target version, OR if it needs to be calculated as a
MERGED value, or it is passed in the request payload as a RESOLVED
value.
- Notice that we are iterating over "flat" (non-grouped) fields which
are part of the `PrebuiltRuleAsset` schema. This means that mapping is
necessary between these flat fields and the diffable (grouped) fields
that are used in the API contract, part of `DiffableRule`. For example,
if we try to determine the value for the `query` field, we will need to
look up for its value in the `eql_query` field if the target rule is
`eql` or in `esql_query` if the target rule is `esql`. All these
mappings can be found in `diffable_rule_fields_mappings.ts`.
- Once a `FieldUpgradeSpecifier` has been retrieved for each field of
the payload we are building, retrieve its actual value: either fetching
it from the base, current or target versions of the rule, from the three
way diff calculation, or retrieving it from the request payload if it
resolved.
- Do this for all upgreadable rules, and the pass the payload array into
`upgradePrebuiltRules()`.
- **IMPORTANT:** The upgrade prebuilt rules logic has been changed from
PATCH to PUT. That means that if the next version of a rule removes a
field, and the user updates to that target version, those fields will be
undefined in the resulting rule. **Additional example:** a installs a
rule, and creates a `timeline_id` for it rule by modifying it. If
neither the next version (target version) still does not have a
`timeline_id` field for it, and the user updates to that target version
fully (without resolving the conflict), that field will not exist
anymore in the resulting rule.

## Acceptance criteria

- [x] Extend the contract of the API endpoint according to the
[POC](elastic#144060):
- [x] Add the ability to pick the `MERGED` version for rule upgrades. If
the `MERGED` version is selected, the diffs are recalculated and the
rule fields are updated to the result of the diff calculation. This is
only possible if all field diffs return a `conflict` value of either
`NO`. If any fields returns a value of `NON_SOLVABLE` or `SOLVABLE`,
reject the request with an error specifying that there are conflicts,
and that they must be resolved on a per-field basis.
- [x] Calculate diffs inside this endpoint, when the value of
`pick_version` is `MERGED`.
- [x] Add the ability to specify rule field versions, to update specific
fields to different `pick_versions`: `BASE' | 'CURRENT' | 'TARGET' |
'MERGED' | 'RESOLVED'` (See `FieldUpgradeRequest` in
[PoC](elastic#144060) for details)

## Handling of special fields

Specific fields are handled under the hood based on
elastic#186544

See implementation in
`x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`,
which imports fields to handle under the hood:
- `DiffableFieldsToOmit`
- `FieldsToUpdateToCurrentVersion`

## Edge cases

- [x] If target version of rule has a **rule type change**, check that
all `pick_version`, at all levels, match `TARGET`. Otherwise, create new
error and add to ruleErrors array.
- [x] if a rule has a specific `targetVersion.type` (for example, EQL)
and the user includes in its `fields` object of the request payload any
fields which do not match that rule type (in this case, for example,
sending in `machine_learning_job_id` as part of `fields`), throw an
error for that rule.
- [x] Calculation of field diffs: what happens if some fields have a
conflict value of `NON_SOLVABLE`:
- [x] If the whole rule is being updated to `MERGED`, and **ANY** fields
return with a `NON_SOLVABLE` conflict, reject the whole update for that
rule: create new error and add to ruleErrors array.
- [x] **EXCEPTION** for case above: the whole rule is being updated to
`MERGED`, and one or more of the fields return with a `NON_SOLVABLE`
conflict, BUT those same fields have a specific `pick_version` for them
in the `fields` object which **ARE NOT** `MERGED`. No error should be
reported in this case.
- [x] The whole rule is being updated to any `pick_version` other than
MERGED, but any specific field in the `fields` object is set to upgrade
to `MERGED`, and the diff for that fields returns a `NON_SOLVABLE`
conflict. In that case, create new error and add to ruleErrors array.

### TODO

- [[Security Solution] Add InvestigationFields and AlertSuppression
fields to the upgrade workflow
[elastic#190597]](elastic#190597):
InvestigationFields is already working, but AlertSuppression is still
currently handled under the hood to update to current version.

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
(cherry picked from commit 7c38873)

) (#196471)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Extend upgrade perform endpoint logic
(#191439)](#191439)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Juan Pablo
Djeredjian","email":"jpdjeredjian@gmail.com"},"sourceCommit":{"committedDate":"2024-10-16T01:51:25Z","message":"[Security
Solution] Extend upgrade perform endpoint logic (#191439)\n\nFixes:
#166376 (main ticket)\r\nFixes:
#186544 (handling of\r\nspecific
fields)\r\nFixes: #180195
(replace PATCH\r\nwith PUT logic on rule upgrade)\r\n\r\n##
Summary\r\n\r\n- Enhances the `/upgrade/_perform` endpoint to upgrade
rules in a way\r\nthat works with prebuilt rules customized by users and
resolve conflicts\r\nbetween user customizations and updates from
Elastic.\r\n- Handles special fields under the hood (see below)\r\n-
Replaces the update prebuilt rule logic to work with PUT instead
of\r\nPATCH.\r\n\r\n### Rough implementation plan\r\n- For each
`upgradeableRule`, we attempt to build the payload necessary\r\nto pass
to `upgradePrebuiltRules()`, which is of type\r\n`PrebuiltRuleAsset`. So
we retrieve the field names from\r\n`FIELDS_PAYLOAD_BY_RULE_TYPE` and
loop through them.\r\n- If any of those `field`s are non-upgreadable,
(i.e. its value needs to\r\nbe handled under the hood) we do so in
`determineFieldUpgradeStatus`.\r\n- Otherwise, we continue to build a
`FieldUpgradeSpecifier` for each\r\nfield, which will help us determine
if that field needs to be set to the\r\nbase, current, target version,
OR if it needs to be calculated as a\r\nMERGED value, or it is passed in
the request payload as a RESOLVED\r\nvalue.\r\n- Notice that we are
iterating over \"flat\" (non-grouped) fields which\r\nare part of the
`PrebuiltRuleAsset` schema. This means that mapping is\r\nnecessary
between these flat fields and the diffable (grouped) fields\r\nthat are
used in the API contract, part of `DiffableRule`. For example,\r\nif we
try to determine the value for the `query` field, we will need
to\r\nlook up for its value in the `eql_query` field if the target rule
is\r\n`eql` or in `esql_query` if the target rule is `esql`. All
these\r\nmappings can be found in
`diffable_rule_fields_mappings.ts`.\r\n- Once a `FieldUpgradeSpecifier`
has been retrieved for each field of\r\nthe payload we are building,
retrieve its actual value: either fetching\r\nit from the base, current
or target versions of the rule, from the three\r\nway diff calculation,
or retrieving it from the request payload if it\r\nresolved.\r\n- Do
this for all upgreadable rules, and the pass the payload array
into\r\n`upgradePrebuiltRules()`.\r\n- **IMPORTANT:** The upgrade
prebuilt rules logic has been changed from\r\nPATCH to PUT. That means
that if the next version of a rule removes a\r\nfield, and the user
updates to that target version, those fields will be\r\nundefined in the
resulting rule. **Additional example:** a installs a\r\nrule, and
creates a `timeline_id` for it rule by modifying it. If\r\nneither the
next version (target version) still does not have a\r\n`timeline_id`
field for it, and the user updates to that target version\r\nfully
(without resolving the conflict), that field will not exist\r\nanymore
in the resulting rule.\r\n\r\n## Acceptance criteria\r\n\r\n- [x] Extend
the contract of the API endpoint according to
the\r\n[POC](https://github.com/elastic/kibana/pull/144060):\r\n- [x]
Add the ability to pick the `MERGED` version for rule upgrades.
If\r\nthe `MERGED` version is selected, the diffs are recalculated and
the\r\nrule fields are updated to the result of the diff calculation.
This is\r\nonly possible if all field diffs return a `conflict` value of
either\r\n`NO`. If any fields returns a value of `NON_SOLVABLE` or
`SOLVABLE`,\r\nreject the request with an error specifying that there
are conflicts,\r\nand that they must be resolved on a per-field
basis.\r\n- [x] Calculate diffs inside this endpoint, when the value
of\r\n`pick_version` is `MERGED`.\r\n- [x] Add the ability to specify
rule field versions, to update specific\r\nfields to different
`pick_versions`: `BASE' | 'CURRENT' | 'TARGET' |\r\n'MERGED' |
'RESOLVED'` (See `FieldUpgradeRequest`
in\r\n[PoC](#144060) for
details)\r\n\r\n## Handling of special fields\r\n\r\nSpecific fields are
handled under the hood based
on\r\nhttps://github.com//issues/186544\r\n\r\nSee
implementation
in\r\n`x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`,\r\nwhich
imports fields to handle under the hood:\r\n-
`DiffableFieldsToOmit`\r\n- `FieldsToUpdateToCurrentVersion`\r\n\r\n##
Edge cases\r\n\r\n- [x] If target version of rule has a **rule type
change**, check that\r\nall `pick_version`, at all levels, match
`TARGET`. Otherwise, create new\r\nerror and add to ruleErrors
array.\r\n- [x] if a rule has a specific `targetVersion.type` (for
example, EQL)\r\nand the user includes in its `fields` object of the
request payload any\r\nfields which do not match that rule type (in this
case, for example,\r\nsending in `machine_learning_job_id` as part of
`fields`), throw an\r\nerror for that rule.\r\n- [x] Calculation of
field diffs: what happens if some fields have a\r\nconflict value of
`NON_SOLVABLE`:\r\n- [x] If the whole rule is being updated to `MERGED`,
and **ANY** fields\r\nreturn with a `NON_SOLVABLE` conflict, reject the
whole update for that\r\nrule: create new error and add to ruleErrors
array.\r\n- [x] **EXCEPTION** for case above: the whole rule is being
updated to\r\n`MERGED`, and one or more of the fields return with a
`NON_SOLVABLE`\r\nconflict, BUT those same fields have a specific
`pick_version` for them\r\nin the `fields` object which **ARE NOT**
`MERGED`. No error should be\r\nreported in this case.\r\n- [x] The
whole rule is being updated to any `pick_version` other than\r\nMERGED,
but any specific field in the `fields` object is set to upgrade\r\nto
`MERGED`, and the diff for that fields returns a
`NON_SOLVABLE`\r\nconflict. In that case, create new error and add to
ruleErrors array.\r\n\r\n### TODO\r\n\r\n- [[Security Solution] Add
InvestigationFields and AlertSuppression\r\nfields to the upgrade
workflow\r\n[#190597]](https://github.com/elastic/kibana/issues/190597):\r\nInvestigationFields
is already working, but AlertSuppression is still\r\ncurrently handled
under the hood to update to current version.\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by:
Maxim Palenov
<maxim.palenov@elastic.co>","sha":"7c3887309cec54cc21e1abf8a2522afa49147712","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","Team:Fleet","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:prev-minor","v8.16.0"],"title":"[Security Solution]
Extend upgrade perform endpoint
logic","number":191439,"url":"https://github.com/elastic/kibana/pull/191439","mergeCommit":{"message":"[Security
Solution] Extend upgrade perform endpoint logic (#191439)\n\nFixes:
#166376 (main ticket)\r\nFixes:
#186544 (handling of\r\nspecific
fields)\r\nFixes: #180195
(replace PATCH\r\nwith PUT logic on rule upgrade)\r\n\r\n##
Summary\r\n\r\n- Enhances the `/upgrade/_perform` endpoint to upgrade
rules in a way\r\nthat works with prebuilt rules customized by users and
resolve conflicts\r\nbetween user customizations and updates from
Elastic.\r\n- Handles special fields under the hood (see below)\r\n-
Replaces the update prebuilt rule logic to work with PUT instead
of\r\nPATCH.\r\n\r\n### Rough implementation plan\r\n- For each
`upgradeableRule`, we attempt to build the payload necessary\r\nto pass
to `upgradePrebuiltRules()`, which is of type\r\n`PrebuiltRuleAsset`. So
we retrieve the field names from\r\n`FIELDS_PAYLOAD_BY_RULE_TYPE` and
loop through them.\r\n- If any of those `field`s are non-upgreadable,
(i.e. its value needs to\r\nbe handled under the hood) we do so in
`determineFieldUpgradeStatus`.\r\n- Otherwise, we continue to build a
`FieldUpgradeSpecifier` for each\r\nfield, which will help us determine
if that field needs to be set to the\r\nbase, current, target version,
OR if it needs to be calculated as a\r\nMERGED value, or it is passed in
the request payload as a RESOLVED\r\nvalue.\r\n- Notice that we are
iterating over \"flat\" (non-grouped) fields which\r\nare part of the
`PrebuiltRuleAsset` schema. This means that mapping is\r\nnecessary
between these flat fields and the diffable (grouped) fields\r\nthat are
used in the API contract, part of `DiffableRule`. For example,\r\nif we
try to determine the value for the `query` field, we will need
to\r\nlook up for its value in the `eql_query` field if the target rule
is\r\n`eql` or in `esql_query` if the target rule is `esql`. All
these\r\nmappings can be found in
`diffable_rule_fields_mappings.ts`.\r\n- Once a `FieldUpgradeSpecifier`
has been retrieved for each field of\r\nthe payload we are building,
retrieve its actual value: either fetching\r\nit from the base, current
or target versions of the rule, from the three\r\nway diff calculation,
or retrieving it from the request payload if it\r\nresolved.\r\n- Do
this for all upgreadable rules, and the pass the payload array
into\r\n`upgradePrebuiltRules()`.\r\n- **IMPORTANT:** The upgrade
prebuilt rules logic has been changed from\r\nPATCH to PUT. That means
that if the next version of a rule removes a\r\nfield, and the user
updates to that target version, those fields will be\r\nundefined in the
resulting rule. **Additional example:** a installs a\r\nrule, and
creates a `timeline_id` for it rule by modifying it. If\r\nneither the
next version (target version) still does not have a\r\n`timeline_id`
field for it, and the user updates to that target version\r\nfully
(without resolving the conflict), that field will not exist\r\nanymore
in the resulting rule.\r\n\r\n## Acceptance criteria\r\n\r\n- [x] Extend
the contract of the API endpoint according to
the\r\n[POC](https://github.com/elastic/kibana/pull/144060):\r\n- [x]
Add the ability to pick the `MERGED` version for rule upgrades.
If\r\nthe `MERGED` version is selected, the diffs are recalculated and
the\r\nrule fields are updated to the result of the diff calculation.
This is\r\nonly possible if all field diffs return a `conflict` value of
either\r\n`NO`. If any fields returns a value of `NON_SOLVABLE` or
`SOLVABLE`,\r\nreject the request with an error specifying that there
are conflicts,\r\nand that they must be resolved on a per-field
basis.\r\n- [x] Calculate diffs inside this endpoint, when the value
of\r\n`pick_version` is `MERGED`.\r\n- [x] Add the ability to specify
rule field versions, to update specific\r\nfields to different
`pick_versions`: `BASE' | 'CURRENT' | 'TARGET' |\r\n'MERGED' |
'RESOLVED'` (See `FieldUpgradeRequest`
in\r\n[PoC](#144060) for
details)\r\n\r\n## Handling of special fields\r\n\r\nSpecific fields are
handled under the hood based
on\r\nhttps://github.com//issues/186544\r\n\r\nSee
implementation
in\r\n`x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`,\r\nwhich
imports fields to handle under the hood:\r\n-
`DiffableFieldsToOmit`\r\n- `FieldsToUpdateToCurrentVersion`\r\n\r\n##
Edge cases\r\n\r\n- [x] If target version of rule has a **rule type
change**, check that\r\nall `pick_version`, at all levels, match
`TARGET`. Otherwise, create new\r\nerror and add to ruleErrors
array.\r\n- [x] if a rule has a specific `targetVersion.type` (for
example, EQL)\r\nand the user includes in its `fields` object of the
request payload any\r\nfields which do not match that rule type (in this
case, for example,\r\nsending in `machine_learning_job_id` as part of
`fields`), throw an\r\nerror for that rule.\r\n- [x] Calculation of
field diffs: what happens if some fields have a\r\nconflict value of
`NON_SOLVABLE`:\r\n- [x] If the whole rule is being updated to `MERGED`,
and **ANY** fields\r\nreturn with a `NON_SOLVABLE` conflict, reject the
whole update for that\r\nrule: create new error and add to ruleErrors
array.\r\n- [x] **EXCEPTION** for case above: the whole rule is being
updated to\r\n`MERGED`, and one or more of the fields return with a
`NON_SOLVABLE`\r\nconflict, BUT those same fields have a specific
`pick_version` for them\r\nin the `fields` object which **ARE NOT**
`MERGED`. No error should be\r\nreported in this case.\r\n- [x] The
whole rule is being updated to any `pick_version` other than\r\nMERGED,
but any specific field in the `fields` object is set to upgrade\r\nto
`MERGED`, and the diff for that fields returns a
`NON_SOLVABLE`\r\nconflict. In that case, create new error and add to
ruleErrors array.\r\n\r\n### TODO\r\n\r\n- [[Security Solution] Add
InvestigationFields and AlertSuppression\r\nfields to the upgrade
workflow\r\n[#190597]](https://github.com/elastic/kibana/issues/190597):\r\nInvestigationFields
is already working, but AlertSuppression is still\r\ncurrently handled
under the hood to update to current version.\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by:
Maxim Palenov
<maxim.palenov@elastic.co>","sha":"7c3887309cec54cc21e1abf8a2522afa49147712"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191439","number":191439,"mergeCommit":{"message":"[Security
Solution] Extend upgrade perform endpoint logic (#191439)\n\nFixes:
#166376 (main ticket)\r\nFixes:
#186544 (handling of\r\nspecific
fields)\r\nFixes: #180195
(replace PATCH\r\nwith PUT logic on rule upgrade)\r\n\r\n##
Summary\r\n\r\n- Enhances the `/upgrade/_perform` endpoint to upgrade
rules in a way\r\nthat works with prebuilt rules customized by users and
resolve conflicts\r\nbetween user customizations and updates from
Elastic.\r\n- Handles special fields under the hood (see below)\r\n-
Replaces the update prebuilt rule logic to work with PUT instead
of\r\nPATCH.\r\n\r\n### Rough implementation plan\r\n- For each
`upgradeableRule`, we attempt to build the payload necessary\r\nto pass
to `upgradePrebuiltRules()`, which is of type\r\n`PrebuiltRuleAsset`. So
we retrieve the field names from\r\n`FIELDS_PAYLOAD_BY_RULE_TYPE` and
loop through them.\r\n- If any of those `field`s are non-upgreadable,
(i.e. its value needs to\r\nbe handled under the hood) we do so in
`determineFieldUpgradeStatus`.\r\n- Otherwise, we continue to build a
`FieldUpgradeSpecifier` for each\r\nfield, which will help us determine
if that field needs to be set to the\r\nbase, current, target version,
OR if it needs to be calculated as a\r\nMERGED value, or it is passed in
the request payload as a RESOLVED\r\nvalue.\r\n- Notice that we are
iterating over \"flat\" (non-grouped) fields which\r\nare part of the
`PrebuiltRuleAsset` schema. This means that mapping is\r\nnecessary
between these flat fields and the diffable (grouped) fields\r\nthat are
used in the API contract, part of `DiffableRule`. For example,\r\nif we
try to determine the value for the `query` field, we will need
to\r\nlook up for its value in the `eql_query` field if the target rule
is\r\n`eql` or in `esql_query` if the target rule is `esql`. All
these\r\nmappings can be found in
`diffable_rule_fields_mappings.ts`.\r\n- Once a `FieldUpgradeSpecifier`
has been retrieved for each field of\r\nthe payload we are building,
retrieve its actual value: either fetching\r\nit from the base, current
or target versions of the rule, from the three\r\nway diff calculation,
or retrieving it from the request payload if it\r\nresolved.\r\n- Do
this for all upgreadable rules, and the pass the payload array
into\r\n`upgradePrebuiltRules()`.\r\n- **IMPORTANT:** The upgrade
prebuilt rules logic has been changed from\r\nPATCH to PUT. That means
that if the next version of a rule removes a\r\nfield, and the user
updates to that target version, those fields will be\r\nundefined in the
resulting rule. **Additional example:** a installs a\r\nrule, and
creates a `timeline_id` for it rule by modifying it. If\r\nneither the
next version (target version) still does not have a\r\n`timeline_id`
field for it, and the user updates to that target version\r\nfully
(without resolving the conflict), that field will not exist\r\nanymore
in the resulting rule.\r\n\r\n## Acceptance criteria\r\n\r\n- [x] Extend
the contract of the API endpoint according to
the\r\n[POC](https://github.com/elastic/kibana/pull/144060):\r\n- [x]
Add the ability to pick the `MERGED` version for rule upgrades.
If\r\nthe `MERGED` version is selected, the diffs are recalculated and
the\r\nrule fields are updated to the result of the diff calculation.
This is\r\nonly possible if all field diffs return a `conflict` value of
either\r\n`NO`. If any fields returns a value of `NON_SOLVABLE` or
`SOLVABLE`,\r\nreject the request with an error specifying that there
are conflicts,\r\nand that they must be resolved on a per-field
basis.\r\n- [x] Calculate diffs inside this endpoint, when the value
of\r\n`pick_version` is `MERGED`.\r\n- [x] Add the ability to specify
rule field versions, to update specific\r\nfields to different
`pick_versions`: `BASE' | 'CURRENT' | 'TARGET' |\r\n'MERGED' |
'RESOLVED'` (See `FieldUpgradeRequest`
in\r\n[PoC](#144060) for
details)\r\n\r\n## Handling of special fields\r\n\r\nSpecific fields are
handled under the hood based
on\r\nhttps://github.com//issues/186544\r\n\r\nSee
implementation
in\r\n`x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/api/perform_rule_upgrade/determine_field_upgrade_status.ts`,\r\nwhich
imports fields to handle under the hood:\r\n-
`DiffableFieldsToOmit`\r\n- `FieldsToUpdateToCurrentVersion`\r\n\r\n##
Edge cases\r\n\r\n- [x] If target version of rule has a **rule type
change**, check that\r\nall `pick_version`, at all levels, match
`TARGET`. Otherwise, create new\r\nerror and add to ruleErrors
array.\r\n- [x] if a rule has a specific `targetVersion.type` (for
example, EQL)\r\nand the user includes in its `fields` object of the
request payload any\r\nfields which do not match that rule type (in this
case, for example,\r\nsending in `machine_learning_job_id` as part of
`fields`), throw an\r\nerror for that rule.\r\n- [x] Calculation of
field diffs: what happens if some fields have a\r\nconflict value of
`NON_SOLVABLE`:\r\n- [x] If the whole rule is being updated to `MERGED`,
and **ANY** fields\r\nreturn with a `NON_SOLVABLE` conflict, reject the
whole update for that\r\nrule: create new error and add to ruleErrors
array.\r\n- [x] **EXCEPTION** for case above: the whole rule is being
updated to\r\n`MERGED`, and one or more of the fields return with a
`NON_SOLVABLE`\r\nconflict, BUT those same fields have a specific
`pick_version` for them\r\nin the `fields` object which **ARE NOT**
`MERGED`. No error should be\r\nreported in this case.\r\n- [x] The
whole rule is being updated to any `pick_version` other than\r\nMERGED,
but any specific field in the `fields` object is set to upgrade\r\nto
`MERGED`, and the diff for that fields returns a
`NON_SOLVABLE`\r\nconflict. In that case, create new error and add to
ruleErrors array.\r\n\r\n### TODO\r\n\r\n- [[Security Solution] Add
InvestigationFields and AlertSuppression\r\nfields to the upgrade
workflow\r\n[#190597]](https://github.com/elastic/kibana/issues/190597):\r\nInvestigationFields
is already working, but AlertSuppression is still\r\ncurrently handled
under the hood to update to current version.\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by:
Maxim Palenov
<maxim.palenov@elastic.co>","sha":"7c3887309cec54cc21e1abf8a2522afa49147712"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Juan Pablo Djeredjian <jpdjeredjian@gmail.com>

…rade with `MERGED` pick_version (#197262)

## Summary

The PR #191439 enhanced the
`/upgrade/_perform` API contract and functionality to allow the users of
the endpoint to upgrade rules to their `MERGED` version.

However, a bug slipped in, where the two different types of `DataSource`
(`type: index_patterns` or `type: data_view_id`) weren't properly
handled and would cause, in some cases, a rule payload to be created
having both an `index` and `data_view` field, causing upgrade to fail.

This PR fixes the issue by handling these two field in a specific way,
checking what the `DataSource` diffable field's type is, and setting the
other field to `undefined`.


### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed



### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)
- [ ] This will appear in the **Release Notes** and follow the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

…rade with `MERGED` pick_version (elastic#197262)

## Summary

The PR elastic#191439 enhanced the
`/upgrade/_perform` API contract and functionality to allow the users of
the endpoint to upgrade rules to their `MERGED` version.

However, a bug slipped in, where the two different types of `DataSource`
(`type: index_patterns` or `type: data_view_id`) weren't properly
handled and would cause, in some cases, a rule payload to be created
having both an `index` and `data_view` field, causing upgrade to fail.

This PR fixes the issue by handling these two field in a specific way,
checking what the `DataSource` diffable field's type is, and setting the
other field to `undefined`.

### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)
- [ ] This will appear in the **Release Notes** and follow the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

(cherry picked from commit 9656621)

…rade with `MERGED` pick_version (elastic#197262)

## Summary

The PR elastic#191439 enhanced the
`/upgrade/_perform` API contract and functionality to allow the users of
the endpoint to upgrade rules to their `MERGED` version.

However, a bug slipped in, where the two different types of `DataSource`
(`type: index_patterns` or `type: data_view_id`) weren't properly
handled and would cause, in some cases, a rule payload to be created
having both an `index` and `data_view` field, causing upgrade to fail.

This PR fixes the issue by handling these two field in a specific way,
checking what the `DataSource` diffable field's type is, and setting the
other field to `undefined`.

### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)
- [ ] This will appear in the **Release Notes** and follow the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

(cherry picked from commit 9656621)

… during rule upgrade with `MERGED` pick_version (#197262) (#197467)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Fix `DataSource` payload creation
during rule upgrade with `MERGED` pick_version
(#197262)](#197262)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Juan Pablo
Djeredjian","email":"jpdjeredjian@gmail.com"},"sourceCommit":{"committedDate":"2024-10-23T14:44:13Z","message":"[Security
Solution] Fix `DataSource` payload creation during rule upgrade with
`MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR
#191439 enhanced
the\r\n`/upgrade/_perform` API contract and functionality to allow the
users of\r\nthe endpoint to upgrade rules to their `MERGED`
version.\r\n\r\nHowever, a bug slipped in, where the two different types
of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`)
weren't properly\r\nhandled and would cause, in some cases, a rule
payload to be created\r\nhaving both an `index` and `data_view` field,
causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling
these two field in a specific way,\r\nchecking what the `DataSource`
diffable field's type is, and setting the\r\nother field to
`undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are
not applicable to this PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n-
[ ] This will appear in the **Release Notes** and follow
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","v8.16.0","backport:version","v8.17.0"],"title":"[Security
Solution] Fix `DataSource` payload creation during rule upgrade with
`MERGED`
pick_version","number":197262,"url":"https://github.com/elastic/kibana/pull/197262","mergeCommit":{"message":"[Security
Solution] Fix `DataSource` payload creation during rule upgrade with
`MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR
#191439 enhanced
the\r\n`/upgrade/_perform` API contract and functionality to allow the
users of\r\nthe endpoint to upgrade rules to their `MERGED`
version.\r\n\r\nHowever, a bug slipped in, where the two different types
of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`)
weren't properly\r\nhandled and would cause, in some cases, a rule
payload to be created\r\nhaving both an `index` and `data_view` field,
causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling
these two field in a specific way,\r\nchecking what the `DataSource`
diffable field's type is, and setting the\r\nother field to
`undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are
not applicable to this PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n-
[ ] This will appear in the **Release Notes** and follow
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10"}},"sourceBranch":"main","suggestedTargetBranches":["8.16","8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197262","number":197262,"mergeCommit":{"message":"[Security
Solution] Fix `DataSource` payload creation during rule upgrade with
`MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR
#191439 enhanced
the\r\n`/upgrade/_perform` API contract and functionality to allow the
users of\r\nthe endpoint to upgrade rules to their `MERGED`
version.\r\n\r\nHowever, a bug slipped in, where the two different types
of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`)
weren't properly\r\nhandled and would cause, in some cases, a rule
payload to be created\r\nhaving both an `index` and `data_view` field,
causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling
these two field in a specific way,\r\nchecking what the `DataSource`
diffable field's type is, and setting the\r\nother field to
`undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are
not applicable to this PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n-
[ ] This will appear in the **Release Notes** and follow
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Juan Pablo Djeredjian <jpdjeredjian@gmail.com>

…n during rule upgrade with `MERGED` pick_version (#197262) (#197466)

# Backport

This will backport the following commits from `main` to `8.16`:
- [[Security Solution] Fix `DataSource` payload creation
during rule upgrade with `MERGED` pick_version
(#197262)](#197262)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Juan Pablo
Djeredjian","email":"jpdjeredjian@gmail.com"},"sourceCommit":{"committedDate":"2024-10-23T14:44:13Z","message":"[Security
Solution] Fix `DataSource` payload creation during rule upgrade with
`MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR
#191439 enhanced
the\r\n`/upgrade/_perform` API contract and functionality to allow the
users of\r\nthe endpoint to upgrade rules to their `MERGED`
version.\r\n\r\nHowever, a bug slipped in, where the two different types
of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`)
weren't properly\r\nhandled and would cause, in some cases, a rule
payload to be created\r\nhaving both an `index` and `data_view` field,
causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling
these two field in a specific way,\r\nchecking what the `DataSource`
diffable field's type is, and setting the\r\nother field to
`undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are
not applicable to this PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n-
[ ] This will appear in the **Release Notes** and follow
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","v8.16.0","backport:version","v8.17.0"],"title":"[Security
Solution] Fix `DataSource` payload creation during rule upgrade with
`MERGED`
pick_version","number":197262,"url":"https://github.com/elastic/kibana/pull/197262","mergeCommit":{"message":"[Security
Solution] Fix `DataSource` payload creation during rule upgrade with
`MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR
#191439 enhanced
the\r\n`/upgrade/_perform` API contract and functionality to allow the
users of\r\nthe endpoint to upgrade rules to their `MERGED`
version.\r\n\r\nHowever, a bug slipped in, where the two different types
of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`)
weren't properly\r\nhandled and would cause, in some cases, a rule
payload to be created\r\nhaving both an `index` and `data_view` field,
causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling
these two field in a specific way,\r\nchecking what the `DataSource`
diffable field's type is, and setting the\r\nother field to
`undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are
not applicable to this PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n-
[ ] This will appear in the **Release Notes** and follow
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10"}},"sourceBranch":"main","suggestedTargetBranches":["8.16","8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197262","number":197262,"mergeCommit":{"message":"[Security
Solution] Fix `DataSource` payload creation during rule upgrade with
`MERGED` pick_version (#197262)\n\n## Summary\r\n\r\nThe PR
#191439 enhanced
the\r\n`/upgrade/_perform` API contract and functionality to allow the
users of\r\nthe endpoint to upgrade rules to their `MERGED`
version.\r\n\r\nHowever, a bug slipped in, where the two different types
of `DataSource`\r\n(`type: index_patterns` or `type: data_view_id`)
weren't properly\r\nhandled and would cause, in some cases, a rule
payload to be created\r\nhaving both an `index` and `data_view` field,
causing upgrade to fail.\r\n\r\nThis PR fixes the issue by handling
these two field in a specific way,\r\nchecking what the `DataSource`
diffable field's type is, and setting the\r\nother field to
`undefined`.\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are
not applicable to this PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [ ] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#_add_your_labels)\r\n-
[ ] This will appear in the **Release Notes** and follow
the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"9656621fcc8f6f9a615b0a27d45db9722e047a10"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Juan Pablo Djeredjian <jpdjeredjian@gmail.com>