Skip to content

Commit a65515b

Browse files
achyutjhunjhunwalaachyutjhunjhunwala
committed
[Dataset Quality] Add fix it flow for field limit (#195561)
## Summary Closes - #190330 This PR implements the logic to support - One click increasing of Field Limit for Field Limit Issues (applicable on for Integrations). For Non Integrations, only text is displayed as to how they can do it. - The One click increase updates the linked custom component template as well as the last backing Index - If Last Backing Index update fails due to any reason, it provides user an option to trigger a Rollover manually. ## Demo Not possible, to many things to display 😆 ## What's Pending ? Tests - [x] API tests - [x] Settings API - [x] Rollover API - [x] Apply New limit API - [x] FTR tests - [x] Displaying of various issues for integrations and non integrations - [x] Fix it Flow Good case, without Rollover - [x] Fix it Flow Good case, with Rollover - [x] Manual Mitigation - Click on Component Template shold navigate to proper logic based on Integration / Non - [x] Manual Mitigation - Ingest Pipeline - [x] Link for official Documentation ## How to setup a local environment We will be setting up 2 different data streams, one with integration and one without. Please follow the steps in the exact order 1. Start Local ES and Local Kibana 2. Install Nginx Integration 1st 3. Ingest data as per script here - https://gist.github.com/achyutjhunjhunwala/03ea29190c6594544f584d2f0efa71e5 4. Set the Limit for the 2 datasets ``` PUT logs-synth.3-default/_settings { "mapping.total_fields.limit": 36 } // Set the limit for Nginx PUT logs-nginx.access-default/_settings { "mapping.total_fields.limit": 52 } ``` 5. Now uncomment line number 59 from the synthtrace script to enable cloud.project.id field and run the scenario again 6. Do a Rollover ``` POST logs-synth.3-default/_rollover POST logs-nginx.access-default/_rollover ``` 7. Get last backing index for both dataset ``` GET _data_stream/logs-synth.3-default/ GET _data_stream/logs-nginx.access-default ``` 8. Increase the Limit by 1 but for last backing index ``` PUT .ds-logs-synth.3-default-2024.10.10-000002/_settings { "mapping.total_fields.limit": 37 } PUT .ds-logs-nginx.access-default-2024.10.10-000002/_settings { "mapping.total_fields.limit": 53 } ``` 9. Run the same Synthtrace scenario again. This setup will give you 3 fields for testings 1. cloud.availability_zone - Which will show the character limit isue 2. cloud.project - Which will show an obsolete error which happened in the past and now does not exists due to field limit 3. cloud.project.id - A current field limit issue --------- Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani01@gmail.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 3ece950) # Conflicts: # x-pack/test_serverless/functional/test_suites/observability/dataset_quality/degraded_field_flyout.ts
1 parent b7ad3e2 commit a65515b

50 files changed

Lines changed: 3707 additions & 719 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎x-pack/packages/index-management/index_management_shared_types/src/types.ts‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,21 @@ import type { LocatorPublic } from '@kbn/share-plugin/public';
1717
import { ExtensionsSetup } from './services/extensions_service';
1818
import { PublicApiServiceSetup } from './services/public_api_service';
1919

20-
export interface IndexManagementLocatorParams extends SerializableRecord {
21-
page: 'data_streams_details';
22-
dataStreamName?: string;
23-
}
20+
export type IndexManagementLocatorParams = SerializableRecord &
21+
(
22+
| {
23+
page: 'data_streams_details';
24+
dataStreamName?: string;
25+
}
26+
| {
27+
page: 'index_template';
28+
indexTemplate: string;
29+
}
30+
| {
31+
page: 'component_template';
32+
componentTemplate: string;
33+
}
34+
);
2435

2536
export type IndexManagementLocator = LocatorPublic<IndexManagementLocatorParams>;
2637

‎x-pack/plugins/index_management/public/application/services/routing.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import { Section } from '../../../common/constants';
1111
import type { IndexDetailsTabId } from '../../../common/constants';
1212
import { ExtensionsService } from '../../services/extensions_service';
1313
import { IndexDetailsSection } from '../../../common/constants';
14+
1415
export const getTemplateListLink = () => `/templates`;
1516

1617
export const getTemplateDetailsLink = (name: string, isLegacy?: boolean) => {
@@ -81,6 +82,11 @@ export const getComponentTemplatesLink = (usedByTemplateName?: string) => {
8182
}
8283
return url;
8384
};
85+
86+
export const getComponentTemplateDetailLink = (name: string) => {
87+
return `/component_templates/${encodeURIComponent(name)}`;
88+
};
89+
8490
export const navigateToIndexDetailsPage = (
8591
indexName: string,
8692
indicesListURLParams: string,

‎x-pack/plugins/index_management/public/locator.test.ts‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,4 +34,26 @@ describe('Index Management URL locator', () => {
3434
});
3535
expect(path).toBe('/data/index_management/data_streams/test');
3636
});
37+
38+
test('locator returns the correct url for index_template', async () => {
39+
const indexTemplateName = 'test@custom';
40+
const { path } = await locator.getLocation({
41+
page: 'index_template',
42+
indexTemplate: indexTemplateName,
43+
});
44+
expect(path).toBe(
45+
encodeURI(`/data/index_management/templates/${encodeURIComponent(indexTemplateName)}`)
46+
);
47+
});
48+
49+
test('locator returns the correct url for component_template', async () => {
50+
const componentTemplateName = 'log@custom';
51+
const { path } = await locator.getLocation({
52+
page: 'component_template',
53+
componentTemplate: componentTemplateName,
54+
});
55+
expect(path).toBe(
56+
`/data/index_management/component_templates/${encodeURIComponent(componentTemplateName)}`
57+
);
58+
});
3759
});

‎x-pack/plugins/index_management/public/locator.ts‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,11 @@
88
import { ManagementAppLocator } from '@kbn/management-plugin/common';
99
import { LocatorDefinition } from '@kbn/share-plugin/public';
1010
import { IndexManagementLocatorParams } from '@kbn/index-management-shared-types';
11-
import { getDataStreamDetailsLink } from './application/services/routing';
11+
import {
12+
getComponentTemplateDetailLink,
13+
getDataStreamDetailsLink,
14+
getTemplateDetailsLink,
15+
} from './application/services/routing';
1216
import { PLUGIN } from '../common/constants';
1317

1418
export const INDEX_MANAGEMENT_LOCATOR_ID = 'INDEX_MANAGEMENT_LOCATOR_ID';
@@ -37,6 +41,18 @@ export class IndexManagementLocatorDefinition
3741
path: location.path + getDataStreamDetailsLink(params.dataStreamName!),
3842
};
3943
}
44+
case 'index_template': {
45+
return {
46+
...location,
47+
path: location.path + getTemplateDetailsLink(params.indexTemplate),
48+
};
49+
}
50+
case 'component_template': {
51+
return {
52+
...location,
53+
path: location.path + getComponentTemplateDetailLink(params.componentTemplate),
54+
};
55+
}
4056
}
4157
};
4258
}

‎x-pack/plugins/observability_solution/dataset_quality/common/api_types.ts‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -134,22 +134,39 @@ export const degradedFieldAnalysisRt = rt.intersection([
134134
type: rt.string,
135135
ignore_above: rt.number,
136136
}),
137+
defaultPipeline: rt.string,
137138
}),
138139
]);
139140

140141
export type DegradedFieldAnalysis = rt.TypeOf<typeof degradedFieldAnalysisRt>;
141142

142-
export const dataStreamSettingsRt = rt.intersection([
143+
export const updateFieldLimitResponseRt = rt.intersection([
143144
rt.type({
144-
lastBackingIndexName: rt.string,
145+
isComponentTemplateUpdated: rt.union([rt.boolean, rt.undefined]),
146+
isLatestBackingIndexUpdated: rt.union([rt.boolean, rt.undefined]),
147+
customComponentTemplateName: rt.string,
145148
}),
146149
rt.partial({
147-
createdOn: rt.union([rt.null, rt.number]), // rt.null is needed because `createdOn` is not available on Serverless
148-
integration: rt.string,
149-
datasetUserPrivileges: datasetUserPrivilegesRt,
150+
error: rt.string,
150151
}),
151152
]);
152153

154+
export type UpdateFieldLimitResponse = rt.TypeOf<typeof updateFieldLimitResponseRt>;
155+
156+
export const dataStreamRolloverResponseRt = rt.type({
157+
acknowledged: rt.boolean,
158+
});
159+
160+
export type DataStreamRolloverResponse = rt.TypeOf<typeof dataStreamRolloverResponseRt>;
161+
162+
export const dataStreamSettingsRt = rt.partial({
163+
lastBackingIndexName: rt.string,
164+
indexTemplate: rt.string,
165+
createdOn: rt.union([rt.null, rt.number]), // rt.null is needed because `createdOn` is not available on Serverless
166+
integration: rt.string,
167+
datasetUserPrivileges: datasetUserPrivilegesRt,
168+
});
169+
153170
export type DataStreamSettings = rt.TypeOf<typeof dataStreamSettingsRt>;
154171

155172
export const dataStreamDetailsRt = rt.partial({

‎x-pack/plugins/observability_solution/dataset_quality/common/data_stream_details/types.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,3 +14,8 @@ export interface AnalyzeDegradedFieldsParams {
1414
lastBackingIndex: string;
1515
degradedField: string;
1616
}
17+
18+
export interface UpdateFieldLimitParams {
19+
dataStream: string;
20+
newFieldLimit: number;
21+
}

‎x-pack/plugins/observability_solution/dataset_quality/common/translations.ts‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -500,3 +500,182 @@ export const degradedFieldMessageIssueDoesNotExistInLatestIndex = i18n.translate
500500
'This issue was detected in an older version of the dataset, but not in the most recent version.',
501501
}
502502
);
503+
504+
export const possibleMitigationTitle = i18n.translate(
505+
'xpack.datasetQuality.details.degradedField.possibleMitigationTitle',
506+
{
507+
defaultMessage: 'Possible mitigation',
508+
}
509+
);
510+
511+
export const increaseFieldMappingLimitTitle = i18n.translate(
512+
'xpack.datasetQuality.details.degradedField.possibleMitigation.increaseFieldMappingLimitTitle',
513+
{
514+
defaultMessage: 'Increase field mapping limit',
515+
}
516+
);
517+
518+
export const fieldLimitMitigationDescriptionText = i18n.translate(
519+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationDescription',
520+
{
521+
defaultMessage:
522+
'The field mapping limit sets the maximum number of fields in an index. When exceeded, additional fields are ignored. To prevent this, increase your field mapping limit.',
523+
}
524+
);
525+
526+
export const fieldLimitMitigationConsiderationText = i18n.translate(
527+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationConsiderations',
528+
{
529+
defaultMessage: 'Before changing the field limit, consider the following:',
530+
}
531+
);
532+
533+
export const fieldLimitMitigationConsiderationText1 = i18n.translate(
534+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationConsiderations1',
535+
{
536+
defaultMessage: 'Increasing the field limit could slow cluster performance.',
537+
}
538+
);
539+
540+
export const fieldLimitMitigationConsiderationText2 = i18n.translate(
541+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationConsiderations2',
542+
{
543+
defaultMessage: 'Increasing the field limit also resolves field limit issues for other fields.',
544+
}
545+
);
546+
547+
export const fieldLimitMitigationConsiderationText3 = i18n.translate(
548+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationConsiderations3',
549+
{
550+
defaultMessage:
551+
'This change applies to the [name] component template and affects all namespaces in the template.',
552+
}
553+
);
554+
555+
export const fieldLimitMitigationConsiderationText4 = i18n.translate(
556+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationConsiderations4',
557+
{
558+
defaultMessage:
559+
'You need to roll over affected data streams to apply mapping changes to component templates.',
560+
}
561+
);
562+
563+
export const fieldLimitMitigationCurrentLimitLabelText = i18n.translate(
564+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationCurrentLimitLabelText',
565+
{
566+
defaultMessage: 'Current limit',
567+
}
568+
);
569+
570+
export const fieldLimitMitigationNewLimitButtonText = i18n.translate(
571+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationNewLimitButtonText',
572+
{
573+
defaultMessage: 'New limit',
574+
}
575+
);
576+
577+
export const fieldLimitMitigationNewLimitPlaceholderText = i18n.translate(
578+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationNewLimitPlaceholderText',
579+
{
580+
defaultMessage: 'New field limit',
581+
}
582+
);
583+
584+
export const fieldLimitMitigationApplyButtonText = i18n.translate(
585+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationApplyButtonText',
586+
{
587+
defaultMessage: 'Apply',
588+
}
589+
);
590+
591+
export const otherMitigationsLoadingAriaText = i18n.translate(
592+
'xpack.datasetQuality.details.degradedField.possibleMitigation.otherMitigationsLoadingText',
593+
{
594+
defaultMessage: 'Loading possible mitigations',
595+
}
596+
);
597+
598+
export const otherMitigationsCustomComponentTemplate = i18n.translate(
599+
'xpack.datasetQuality.details.degradedField.possibleMitigation.otherMitigationsCustomComponentTemplate',
600+
{
601+
defaultMessage: 'Add or edit custom component template',
602+
}
603+
);
604+
605+
export const otherMitigationsCustomIngestPipeline = i18n.translate(
606+
'xpack.datasetQuality.details.degradedField.possibleMitigation.otherMitigationsCustomIngestPipeline',
607+
{
608+
defaultMessage: 'Add or edit custom ingest pipeline',
609+
}
610+
);
611+
612+
export const fieldLimitMitigationOfficialDocumentation = i18n.translate(
613+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationOfficialDocumentation',
614+
{
615+
defaultMessage: 'Documentation',
616+
}
617+
);
618+
619+
export const fieldLimitMitigationSuccessMessage = i18n.translate(
620+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationSuccessMessage',
621+
{
622+
defaultMessage: 'New limit set!',
623+
}
624+
);
625+
626+
export const fieldLimitMitigationSuccessComponentTemplateLinkText = i18n.translate(
627+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationSuccessComponentTemplateLinkText',
628+
{
629+
defaultMessage: 'See component template',
630+
}
631+
);
632+
633+
export const fieldLimitMitigationPartiallyFailedMessage = i18n.translate(
634+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationPartiallyFailedMessage',
635+
{
636+
defaultMessage: 'Changes not applied to new data',
637+
}
638+
);
639+
640+
export const fieldLimitMitigationFailedMessage = i18n.translate(
641+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationFailedMessage',
642+
{
643+
defaultMessage: 'Changes not applied',
644+
}
645+
);
646+
647+
export const fieldLimitMitigationFailedMessageDescription = i18n.translate(
648+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationFailedMessageDescription',
649+
{
650+
defaultMessage: 'Failed to set new limit',
651+
}
652+
);
653+
654+
export const fieldLimitMitigationPartiallyFailedMessageDescription = i18n.translate(
655+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationPartiallyFailedMessageDescription',
656+
{
657+
defaultMessage:
658+
'The component template was successfully updated with the new field limit, but the changes were not applied to the most recent backing index. Perform a rollover to apply your changes to new data.',
659+
}
660+
);
661+
662+
export const fieldLimitMitigationRolloverButton = i18n.translate(
663+
'xpack.datasetQuality.details.degradedField.possibleMitigation.fieldLimitMitigationRolloverButton',
664+
{
665+
defaultMessage: 'Rollover',
666+
}
667+
);
668+
669+
export const manualMitigationCustomPipelineCopyPipelineNameAriaText = i18n.translate(
670+
'xpack.datasetQuality.details.degradedField.possibleMitigation.copyPipelineNameAriaText',
671+
{
672+
defaultMessage: 'Copy pipeline name',
673+
}
674+
);
675+
676+
export const manualMitigationCustomPipelineCreateEditPipelineLink = i18n.translate(
677+
'xpack.datasetQuality.details.degradedField.possibleMitigation.createEditPipelineLink',
678+
{
679+
defaultMessage: 'create or edit the pipeline',
680+
}
681+
);

‎x-pack/plugins/observability_solution/dataset_quality/common/utils/component_template_name.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
/*
9+
* There are index templates like this metrics-apm.service_transaction.10m@template which exists.
10+
* Hence this @ needs to be removed to derive the custom component template name.
11+
*/
12+
export function getComponentTemplatePrefixFromIndexTemplate(indexTemplate: string) {
13+
if (indexTemplate.includes('@')) {
14+
return indexTemplate.split('@')[0];
15+
}
16+
17+
return indexTemplate;
18+
}

0 commit comments

Comments
 (0)