Skip to content

[8.x] Auto increase fields limit of the alert indices (#216719)#218203

Merged
pmuellr merged 8 commits intoelastic:8.19from
ersin-erdal:backport/8.x/pr-216719
Apr 21, 2025
Merged

[8.x] Auto increase fields limit of the alert indices (#216719)#218203
pmuellr merged 8 commits intoelastic:8.19from
ersin-erdal:backport/8.x/pr-216719

Conversation

@ersin-erdal
Copy link
Copy Markdown
Contributor

Backport

This will backport the following commits from main to 8.x:

Questions ?

Please refer to the Backport tool documentation

This PR adds the auto-increase the fields limit on startup when an
alerts index reaches its limits because of the dynamic fields.

# To verify:
To be able to test this PR we need a rule type that adds dynamic fields.
I used the custom threshold rule for this:

Go to the custom threshold rule type definition and change its
alerts.mappings to:
```
  mappings: {
    // dynamic: true,
    fieldMap: {
      'kibana.alerting.grouping': {
        type: 'object',
        dynamic: true,
        array: false,
        required: false,
      },
      ...legacyExperimentalFieldMap,
      ...Array(412)
        .fill(0)
        .reduce((acc, val, i) => {
          acc[`${i + 1}`] = { type: 'keyword', array: false, required: false };
          return acc;
        }, {}),
    },
    dynamicTemplates: [
      {
        strings_as_keywords: {
          path_match: 'kibana.alert.grouping.*',
          match_mapping_type: 'string',
          mapping: {
            type: 'keyword',
            ignore_above: 1024,
          },
        },
      },
    ],
  },
  ```

  Above changes adds 412 dummy fields to the alerts index to make it close to reach its fields limit (default: 2500).
  And makes everything under `kibana.alert.grouping` path to be added to the index as dynamic fields.

  Then apply the below changes to the custom threshold rule executor:
  ```
  const grouping: Record<string, string> = {};
        groups?.forEach((groupObj) => (grouping[groupObj.field] = groupObj.value));

        const { uuid, start } = alertsClient.report({
          id: `${group}`,
          actionGroup: actionGroupId,
          payload: {
            [ALERT_REASON]: reason,
            [ALERT_EVALUATION_VALUES]: evaluationValues,
            [ALERT_EVALUATION_THRESHOLD]: threshold,
            [ALERT_GROUP]: groups,
            // @ts-ignore
            ['kibana.alerting.grouping']: grouping,
            ...flattenAdditionalContext(additionalContext),
            ...getEcsGroups(groups),
          },
        });
  ```

Above changes add the selected groups under `kibana.alerting.grouping` path.

Then:
- Run ES with ` path.data=../your-local-data-path` to keep the data for the next start.
- Run Kibana
- Create a custom threshold rule that generates an alert and has at least 2 groups.
- Let the rule run.
- Go to `Stack Management` > `Index Management` and search for observability threshold index.
- Check its mappings, it should show the dummy fields you have added to the rule type and the first grouping you have selected while you were creating the rule type.
- Go to the Dev Tools and find your alert in the `.internal.alerts-observability.threshold.alerts-default-000001` index.
The other groups you have selected should be saved  under `_ignored` field:
```
"_ignored": [
     "kibana.alerting.grouping.host.name"
],
```
- Stop Kibana
- increase the number of dummy fields you have added to the rule type definition:
```
  ...Array(412) <-- make this greater than 412
        .fill(0)
```
- Start kibana again.
- The new fields should be added to the mappings. Check them on `Stack Management` > `Index Management`
- Check also the index settings: `Stack Management` > `Index Management` > `.internal.alerts-observability.threshold.alerts-default-000001` > settings tab.
- `"mapping" > "total_fields" >  "limit" ` should be greater than 2500

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit f6c30d6)

# Conflicts:
#	x-pack/platform/plugins/shared/alerting/server/alerts_service/alerts_service.test.ts
#	x-pack/platform/plugins/shared/alerting/server/alerts_service/lib/create_concrete_write_index.ts
#	x-pack/platform/plugins/shared/alerting/server/alerts_service/lib/create_or_update_component_template.test.ts
#	x-pack/platform/plugins/shared/alerting/server/alerts_service/lib/create_or_update_component_template.ts
#	x-pack/platform/plugins/shared/alerting/server/alerts_service/lib/create_or_update_index_template.test.ts
#	x-pack/platform/plugins/shared/alerting/server/alerts_service/lib/create_or_update_index_template.ts
@ersin-erdal ersin-erdal added the backport This PR is a backport of another PR label Apr 15, 2025
@ersin-erdal ersin-erdal enabled auto-merge (squash) April 15, 2025 08:13
Copy link
Copy Markdown
Contributor

@pmuellr pmuellr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Copy Markdown
Contributor

@pmuellr pmuellr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jbudz jbudz changed the base branch from old-8.x to 8.19 April 18, 2025 18:38
@jbudz jbudz disabled auto-merge April 18, 2025 18:53
@jbudz
Copy link
Copy Markdown
Contributor

jbudz commented Apr 18, 2025

@ersin-erdal

I updated the target branch from old-8.x (previously 8.x) to 8.19 to reflect our global branch rename of 8.x to 8.19.

When you have a moment can you double check everything is in order and proceed as usual if so? Feel free to reach out if there's any questions.

@pmuellr
Copy link
Copy Markdown
Contributor

pmuellr commented Apr 18, 2025

Changing the merge branch looks ok to me.

I don't think it even matters, but going to run another ci check for this, just to make sure, since Ersin will be out next week.

@pmuellr
Copy link
Copy Markdown
Contributor

pmuellr commented Apr 18, 2025

/ci

@pmuellr
Copy link
Copy Markdown
Contributor

pmuellr commented Apr 21, 2025

@elasticmachine merge upstream

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
alerting 24.6KB 24.7KB +54.0B
Unknown metric groups

ESLint disabled line counts

id before after diff
@kbn/test-suites-xpack 762 763 +1

Total ESLint disabled count

id before after diff
@kbn/test-suites-xpack 788 789 +1

History

@pmuellr pmuellr merged commit 87d7a72 into elastic:8.19 Apr 21, 2025
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport This PR is a backport of another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants