[Security Solution] Fix cloud logs exclusion by stephmilovic · Pull Request #130069 · elastic/kibana

stephmilovic · 2022-04-12T23:24:20Z

Summary

The exclusion for elastic cloud logs needs to be at the end of the index pattern list. Moved the .sort() outside of the exclude. There are some other inconsistencies with exclude patterns that are being addressed by the ES team: Multi-target syntax bug with datastream backing indices and excluded targets elasticsearch#86019
~~Added additional log excludes: .ds-logs-elastic_agent*, .ds-logs-system-* as proposed in an SDH~~
Add support for exclude indices in the Advanced Settings index pattern as requested here: [Security Solution] [Sourcerer] Exclude patterns added to the securitySolution:defaultIndex should be applied to the default data view #130753
Move the Elastic cloud logs exclude from being appended on magically to the main Advanced Settings index pattern, giving control to the user

elasticmachine · 2022-04-12T23:24:22Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

elasticmachine · 2022-04-12T23:24:22Z

Pinging @elastic/security-solution (Team: SecuritySolution)

MadameSheema · 2022-04-21T12:45:43Z

@elasticmachine merge upstream

x-pack/plugins/security_solution/public/common/containers/sourcerer/index.tsx

x-pack/plugins/security_solution/public/common/containers/sourcerer/index.test.tsx

x-pack/plugins/security_solution/common/constants.ts

andrew-goldstein

Thanks @stephmilovic for fixing 130753, and for ensuring exclude patterns are always applied at the end as a workaround for 86019! 🙏

In addition to the fix and workaround above, making the -*elastic-cloud-logs-* exclude pattern an explicit part of the securitySolution:defaultIndex (and removing the code that appended it automatically in some cases) makes it easier to reason about exclusions, and makes the Security Data View return consistent results when the data view is used in other parts of Kibana (outside of the security solution).

Release notes

DOCS team, please include this fix in the release notes, and communicate the following:

This PR adds the -*elastic-cloud-logs-* exclude pattern to the default securitySolution:defaultIndex setting (in Kibana advanced settings)
Elastic Cloud users who modified the default securitySolution:defaultIndex setting may need to manually add -*elastic-cloud-logs-* to the securitySolution:defaultIndex setting to exclude logs from Elastic Cloud that were automatically excluded prior to this fix
This PR includes a fix for 130753, which enables arbitrary exclude patterns to be added to the securitySolution:defaultIndex setting
This PR includes a workaround for 86019 by ensuring exclude patterns always appear at the end of the list of indexes

✅ Desk tested locally

LGTM 🚀

andrew-goldstein · 2022-05-09T23:20:30Z

CC @paulewing

madirey

LGTM!

yctercero · 2022-05-10T22:51:10Z

x-pack/plugins/security_solution/public/common/containers/sourcerer/index.tsx

    strict: false,
  }) != null;
-
-export const isAlertsOrRulesDetailsPage = (pathname: string): boolean =>


Was this just dead code?

yes. sorry, should've left a comment

yctercero · 2022-05-10T22:53:14Z

x-pack/plugins/security_solution/public/common/store/sourcerer/helpers.ts

 import { SelectedDataViewPayload } from './actions';
 import { sourcererModel } from '../model';

+export const sortWithExcludesAtEnd = (indices: string[]) => {


Nit: Can we add a quick unit test for the new helper?

yctercero · 2022-05-10T22:56:23Z

x-pack/plugins/security_solution/common/constants.ts

 ];

+/** The comma-delimited list of Elasticsearch indices from which the SIEM app collects events, and the exclude index pattern */
+export const DEFAULT_INDEX_PATTERN = [...INCLUDE_INDEX_PATTERN, ...EXCLUDE_ELASTIC_CLOUD_INDICES];


Does the change mean the elastic cloud index will always be excluded (unless the user modifies the advanced settings) as opposed to before when it was just if the logs-* existed or something?

yctercero

LGTM - one question left where just trying to better understand the behavior related to the cloud logs and what functionality is missed if they're always excluded (if that is the case).

machadoum

security-threat-hunting-explore changes LGTM!

stephmilovic · 2022-05-11T14:37:50Z

x-pack/plugins/security_solution/public/common/containers/sourcerer/index.test.tsx

-    });
-
-    it('Should NOT exclude elastic cloud alias when selected patterns does NOT include "logs-*" as an alias', async () => {
+    it('Should put any excludes in the index pattern at the end of the pattern list, and sort both the includes and excludes', async () => {


Nit: Can we add a quick unit test for the new helper?

@yctercero @machadoum the sortWithExcludesAtEnd function is tested through this test. would you prefer i explicitly add a test in helpers.test.ts?

stephmilovic · 2022-05-11T16:01:50Z

@elasticmachine merge upstream

stephmilovic · 2022-05-11T18:34:31Z

@elasticmachine merge upstream

kibana-ci · 2022-05-11T19:46:36Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	4.8MB	4.8MB	-562.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	246.3KB	246.6KB	+268.0B

History

💔 Build #44114 failed b975476
💛 Build #43698 was flaky 7b8c557
💔 Build #43506 failed 723de21
💔 Build #43472 failed 9f60d27
💔 Build #43365 failed adc0ea9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2022-05-11T19:51:26Z

💔 All backports failed

Status	Branch	Result
❌	8.2	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 130069

Questions ?

Please refer to the Backport tool documentation

* fix conflicts * fix old import

fix cloud logs

8fa1669

stephmilovic requested a review from andrew-goldstein April 12, 2022 23:24

stephmilovic requested a review from a team as a code owner April 12, 2022 23:24

stephmilovic marked this pull request as draft April 12, 2022 23:41

stephmilovic added 3 commits April 13, 2022 13:44

update test

bbe077b

Merge branch 'main' into cloud_logs_fix

8190666

test fixing

5abe389

kibanamachine and others added 6 commits April 21, 2022 07:45

Merge branch 'main' into cloud_logs_fix

0df0204

Merge branch 'main' into cloud_logs_fix

c53c1aa

add more log excludes

8f15970

fix snapshots

09fbf55

support exclude patterns

26140ef

Merge branch 'main' into cloud_logs_fix

b6083bc

stephmilovic marked this pull request as ready for review April 28, 2022 21:36

stephmilovic requested review from a team as code owners April 28, 2022 21:36

andrew-goldstein reviewed Apr 29, 2022

View reviewed changes

x-pack/plugins/security_solution/public/common/containers/sourcerer/index.tsx Outdated Show resolved Hide resolved

andrew-goldstein reviewed Apr 29, 2022

View reviewed changes

x-pack/plugins/security_solution/public/common/containers/sourcerer/index.tsx Outdated Show resolved Hide resolved

andrew-goldstein reviewed Apr 29, 2022

View reviewed changes

x-pack/plugins/security_solution/public/common/containers/sourcerer/index.test.tsx Show resolved Hide resolved

include excludes in the default index pattern

9f60d27

kqualters-elastic approved these changes May 9, 2022

View reviewed changes

andrew-goldstein reviewed May 9, 2022

View reviewed changes

x-pack/plugins/security_solution/common/constants.ts Outdated Show resolved Hide resolved

update indices and tests

723de21

andrew-goldstein approved these changes May 9, 2022

View reviewed changes

andrew-goldstein added release_note:fix and removed release_note:skip Skip the PR/issue when compiling release notes labels May 9, 2022

stephmilovic added 2 commits May 10, 2022 07:33

fix i81n

1a8eef1

fix cypress

1c34bd3

stephmilovic requested review from a team as code owners May 10, 2022 13:45

fix cypress better

7b8c557

madirey approved these changes May 10, 2022

View reviewed changes

yctercero reviewed May 10, 2022

View reviewed changes

yctercero approved these changes May 10, 2022

View reviewed changes

machadoum approved these changes May 11, 2022

View reviewed changes

stephmilovic commented May 11, 2022

View reviewed changes

Merge branch 'main' into cloud_logs_fix

b975476

Merge branch 'main' into cloud_logs_fix

978db99

stephmilovic added the v8.2.1 label May 11, 2022

stephmilovic merged commit 6fc82ba into elastic:main May 11, 2022

stephmilovic mentioned this pull request May 11, 2022

[8.2] [Security Solution] Fix cloud logs exclusion (#130069) #132067

Merged

stephmilovic added a commit that referenced this pull request May 12, 2022

[8.2] [Security Solution] Fix cloud logs exclusion (#130069) (#132067)

8f4a578

* fix conflicts * fix old import

Conversation

stephmilovic commented Apr 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

elasticmachine commented Apr 12, 2022

Uh oh!

elasticmachine commented Apr 12, 2022

Uh oh!

MadameSheema commented Apr 21, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrew-goldstein left a comment

Choose a reason for hiding this comment

Release notes

Uh oh!

andrew-goldstein commented May 9, 2022

Uh oh!

madirey left a comment

Choose a reason for hiding this comment

Uh oh!

yctercero May 10, 2022

Choose a reason for hiding this comment

Uh oh!

stephmilovic May 11, 2022

Choose a reason for hiding this comment

Uh oh!

yctercero May 10, 2022

Choose a reason for hiding this comment

Uh oh!

machadoum May 11, 2022

Choose a reason for hiding this comment

Uh oh!

yctercero May 10, 2022

Choose a reason for hiding this comment

Uh oh!

stephmilovic May 11, 2022

Choose a reason for hiding this comment

Uh oh!

yctercero left a comment

Choose a reason for hiding this comment

Uh oh!

machadoum left a comment

Choose a reason for hiding this comment

Uh oh!

stephmilovic May 11, 2022

Choose a reason for hiding this comment

Uh oh!

stephmilovic commented May 11, 2022

Uh oh!

stephmilovic commented May 11, 2022

Uh oh!

kibana-ci commented May 11, 2022

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

History

Uh oh!

kibanamachine commented May 11, 2022

💔 All backports failed

Manual backport

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

stephmilovic commented Apr 12, 2022 •

edited

Loading