[SIEM][Detections Engine] Add note markdown field to backend by yctercero · Pull Request #59796 · elastic/kibana

yctercero · 2020-03-10T17:16:13Z

Summary

This is part of #59176 - breaking up into backend and frontend PRs.

Problem to solve/Customer Benefit: Analysts need as much context as possible when investigating signals. If a richer format of information can be provided to them via their UX, then they may become more effective at completing investigations and cases.

This PR adds new note field for markdown and updates unit tests. One of the bigger considerations in implementing this new field was how it should deal with backwards compatibility (migration, schema requirements, etc). Decided to make the new field optional (and no default if not present) on the way in and optional on the way out.

Manual tests done:

testing rule without note value
- ./post_rule.sh
- ./patch_rule.sh
- ./update_rule.sh
testing rule with note value
- ./post_rule.sh ./rules/queries/query_with_note.json
- ./patch_rule.sh ./rules/patches/update_note.json
- ./update_rule.sh ./rules/updates/update_note.json
- ./update_rule.sh ./rules/updates/update_note.json removing note value prior to running (resulting in updated note having no note value any longer)

Checklist

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2020-03-10T18:09:47Z

Pinging @elastic/siem (Team:SIEM)

FrankHassanabad · 2020-03-10T19:02:10Z

You have one error from the build to fix with e2e tests:

...plugins/siem/server/lib/detection_engine/routes/schemas/add_prepackaged_rules_schema.test.ts

...gacy/plugins/siem/server/lib/detection_engine/routes/schemas/add_prepackaged_rules_schema.ts

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/schemas/create_rules_schema.ts

...k/legacy/plugins/siem/server/lib/detection_engine/routes/schemas/update_rules_schema.test.ts

...gacy/plugins/siem/server/lib/detection_engine/routes/schemas/patch_rules_bulk_schema.test.ts

...ck/legacy/plugins/siem/server/lib/detection_engine/routes/schemas/patch_rules_schema.test.ts

…wn_de_be

FrankHassanabad · 2020-03-11T14:56:46Z

You will want to add a note to this file:
query_with_everything.json

Something like this within that file:

  "timeline_id": "timeline_id",
  "timeline_title": "timeline_title",
  "note": "Some note for you",
  "version": 1

Test it like so:

./post_rule.sh ./rules/queries/query_with_everything.json

That's the file we communicate with people such as documentation and front end tests and we use it to test things manually if the need comes up.

In the queries folder I would add something like this query file and contents:

query_with_note.json

{
  "name": "Query with a note",
  "description": "Query with a note",
  "rule_id": "query-with-note",
  "risk_score": 1,
  "severity": "high",
  "type": "query",
  "query": "user.name: root or user.name: admin",
  "note": "Hello I am a markdown note"
}

And then in the patches and update folder I would add something similar so it's easy for us to run manual tests when we need to and that would pretty much complete the feature.

You can run the update_rule.sh and the patch_rule.sh scripts to ensure it works as expected. Then you can update your comments section at the top of this to let everyone know what scripts they can run to test out the notes section.

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/index/signals_mapping.json

yctercero · 2020-03-11T17:19:31Z

@elasticmachine merge upstream

FrankHassanabad

Thanks for all the updates for this and the tests. I checked it out and tested it and everything looks really good and the code reads really clean. LGTM! 👍

kibanamachine · 2020-03-11T18:56:33Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: e551118

History

💔 Build #32462 failed c7876c2
💔 Build #32447 failed e429a5f
💚 Build #32391 succeeded 0025dca
💔 Build #32297 failed 1e28319
💔 Build #32168 failed f8513d3

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…#59796) * add new note markdown field to DE backend Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…#59947) * add new note markdown field to DE backend Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

wip - updated DE backend to take new "note" param

f8513d3

yctercero changed the title ~~[SIEM][Detections Engine] - Add note markdown field~~ [SIEM][Detections Engine] - Add note markdown field to backend Mar 10, 2020

yctercero self-assigned this Mar 10, 2020

yctercero added Team:SIEM v7.7.0 v8.0.0 labels Mar 10, 2020

yctercero added the release_note:enhancement label Mar 10, 2020

yctercero marked this pull request as ready for review March 10, 2020 18:17

yctercero linked an issue Mar 10, 2020 that may be closed by this pull request

[SIEM] Detections Markdown-Capable Field Design Mockups #59176

Closed

FrankHassanabad reviewed Mar 10, 2020

View reviewed changes

...plugins/siem/server/lib/detection_engine/routes/schemas/add_prepackaged_rules_schema.test.ts Show resolved Hide resolved

FrankHassanabad reviewed Mar 10, 2020

View reviewed changes

...gacy/plugins/siem/server/lib/detection_engine/routes/schemas/add_prepackaged_rules_schema.ts Outdated Show resolved Hide resolved

FrankHassanabad reviewed Mar 10, 2020

View reviewed changes

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/schemas/create_rules_schema.ts Outdated Show resolved Hide resolved

updated to remove default of note to empty string, updated tests

1e28319

yctercero commented Mar 11, 2020

View reviewed changes

...k/legacy/plugins/siem/server/lib/detection_engine/routes/schemas/update_rules_schema.test.ts Show resolved Hide resolved

adjusted integration tests, re-ran all integration tests, got all green

0025dca

yctercero changed the title ~~[SIEM][Detections Engine] - Add note markdown field to backend~~ [SIEM][Detections Engine] Add note markdown field to backend Mar 11, 2020

FrankHassanabad reviewed Mar 11, 2020

View reviewed changes

...gacy/plugins/siem/server/lib/detection_engine/routes/schemas/patch_rules_bulk_schema.test.ts Outdated Show resolved Hide resolved

FrankHassanabad reviewed Mar 11, 2020

View reviewed changes

...ck/legacy/plugins/siem/server/lib/detection_engine/routes/schemas/patch_rules_schema.test.ts Show resolved Hide resolved

yctercero added 2 commits March 11, 2020 10:52

adjusted unit test to avoid false positives in future

81ef4b8

Merge branch 'master' of github.com:yctercero/kibana into rule_markdo…

e429a5f

…wn_de_be

updated DE scripts for ease of use testing new note feature

c7876c2

FrankHassanabad reviewed Mar 11, 2020

View reviewed changes

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/index/signals_mapping.json Show resolved Hide resolved

Merge branch 'master' into rule_markdown_de_be

e551118

FrankHassanabad approved these changes Mar 11, 2020

View reviewed changes

yctercero merged commit f9273cc into elastic:master Mar 11, 2020

yctercero mentioned this pull request Mar 11, 2020

[7.x] [SIEM][Detections Engine] Add note markdown field to backend (#59796) #59947

Merged

yctercero added a commit that referenced this pull request Mar 11, 2020

[SIEM][Detections Engine] Add note markdown field to backend (#59796) (…

25df364

…#59947) * add new note markdown field to DE backend Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

yctercero mentioned this pull request Mar 13, 2020

[SIEM][Detections Engine] - Add rule markdown field to rule create, detail, and edit flows #60108

Merged

5 tasks

yctercero mentioned this pull request Mar 26, 2020

[SIEM][Detections Engine] - Add rule markdown to timeline global notes #61026

Merged

5 tasks

yctercero deleted the rule_markdown_de_be branch July 20, 2020 01:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM][Detections Engine] Add note markdown field to backend#59796

[SIEM][Detections Engine] Add note markdown field to backend#59796
yctercero merged 7 commits intoelastic:masterfrom
yctercero:rule_markdown_de_be

yctercero commented Mar 10, 2020 •

edited

Loading

Uh oh!

elasticmachine commented Mar 10, 2020

Uh oh!

FrankHassanabad commented Mar 10, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FrankHassanabad commented Mar 11, 2020 •

edited

Loading

Uh oh!

Uh oh!

yctercero commented Mar 11, 2020

Uh oh!

FrankHassanabad left a comment

Uh oh!

kibanamachine commented Mar 11, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

yctercero commented Mar 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

For maintainers

Uh oh!

elasticmachine commented Mar 10, 2020

Uh oh!

FrankHassanabad commented Mar 10, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FrankHassanabad commented Mar 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

yctercero commented Mar 11, 2020

Uh oh!

FrankHassanabad left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Mar 11, 2020

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yctercero commented Mar 10, 2020 •

edited

Loading

FrankHassanabad commented Mar 11, 2020 •

edited

Loading