New Log Type JSON format by petardz · Pull Request #465 · opensearch-project/security-analytics

petardz · 2023-06-20T00:02:41Z

Description

Removed converting "." to "_" when parsing Sigma rules
Introduced new Log Type JSON format to replace existing fieldmappings.yml/mappings.json files
Added loading of log types from new Log Type JSON files
Replaced usage of fieldmappings.yml with LogType loaded from new JSON format

Issues Resolved

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

codecov · 2023-06-21T12:18:35Z

Codecov Report

Merging #465 (2cfcb39) into main (d14e764) will increase coverage by 0.02%.
The diff coverage is 30.13%.

@@             Coverage Diff              @@
##               main     #465      +/-   ##
============================================
+ Coverage     28.21%   28.23%   +0.02%     
- Complexity      897      906       +9     
============================================
  Files           231      235       +4     
  Lines          9326     9443     +117     
  Branches       1074     1081       +7     
============================================
+ Hits           2631     2666      +35     
- Misses         6459     6545      +86     
+ Partials        236      232       -4

Impacted Files	Coverage Δ
...rch/securityanalytics/SecurityAnalyticsPlugin.java	`0.00% <0.00%> (ø)`
...ecurityanalytics/logtype/BuiltinLogTypeLoader.java	`0.00% <0.00%> (ø)`
...arch/securityanalytics/logtype/LogTypeService.java	`0.00% <0.00%> (ø)`
...lytics/transport/TransportIndexDetectorAction.java	`0.00% <0.00%> (ø)`
...yanalytics/transport/TransportIndexRuleAction.java	`0.00% <0.00%> (ø)`
...g/opensearch/securityanalytics/util/FileUtils.java	`0.00% <0.00%> (ø)`
...opensearch/securityanalytics/util/RuleIndices.java	`0.00% <0.00%> (ø)`
...rg/opensearch/securityanalytics/model/LogType.java	`71.42% <71.42%> (ø)`
...ecurityanalytics/rules/backend/OSQueryBackend.java	`67.24% <100.00%> (+0.86%)`	⬆️
.../securityanalytics/rules/backend/QueryBackend.java	`64.22% <100.00%> (+0.11%)`	⬆️

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

fix flaky tests

sbcd90 · 2023-06-22T16:49:49Z

src/main/java/org/opensearch/securityanalytics/model/LogType.java

+
+        for(Mapping m : mappings) {
+            out.writeString(m.getRawField());
+            out.writeString(m.getEcs());


can we use out.writeCollection method here?

sbcd90 · 2023-06-22T16:56:31Z

src/main/java/org/opensearch/securityanalytics/model/LogType.java

+
+public class LogType implements Writeable, ToXContentObject {
+
+    private static final String ID = "id";


can we add unit tests for this class similar to https://github.com/opensearch-project/common-utils/blob/main/src/test/kotlin/org/opensearch/commons/alerting/model/FindingTests.kt & https://github.com/opensearch-project/common-utils/blob/main/src/test/kotlin/org/opensearch/commons/alerting/model/WriteableTests.kt

sbcd90 · 2023-06-22T17:00:26Z

src/main/java/org/opensearch/securityanalytics/transport/TransportIndexDetectorAction.java

-            for(String category: ruleCategories){
-                queryBackendMap.put(category, new OSQueryBackend(category, true, true));
+            for(String category: ruleCategories) {
+                Map<String, String> fieldMappings = logTypeService.getRuleFieldMappings(category);


getRuleFieldMappings internally calls getAllLogTypes from BuiltinLogTypeLoader which internally calls loadBuiltinLogTypes.
Why do we need to load all log types & then filter for particular category in memory?

Actual loading of logTypes from disk will be done only 1 time. I replaced getAllLogTypes with BuiltinLogTypeLoader.getLogTypeByName(logType) existing method.

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

* New Log Type JSON format Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> (cherry picked from commit faccf6c)

initial commit

0922913

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

petardz requested review from AWSHurneyt, amsiglan, eirsep, getsaurabh02, lezzago, praveensameneni and sbcd90 as code owners June 20, 2023 00:02

petardz and others added 7 commits June 20, 2023 02:07

removed unused methods from LogTypeService

55200c1

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

import optimize

3634dfe

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

javadoc fix

488b2ad

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

test fix

8cffd44

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

minor fix due to alerting changes

fd7201b

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

Merge pull request #3 from sbcd90/petar

f7351d6

queryBackend tests fix

0ae5066

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

sbcd90 and others added 2 commits June 21, 2023 22:48

fix flaky tests

f1cccb9

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

Merge pull request #5 from sbcd90/petar1

df30ccf

fix flaky tests

sbcd90 requested changes Jun 22, 2023

View reviewed changes

petardz added 3 commits June 23, 2023 00:48

addressing pr comments

f0b4bf2

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

test fix

80b93b3

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

license fix

aa37d1f

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

sbcd90 previously approved these changes Jun 23, 2023

View reviewed changes

small refactor

b8e6334

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

petardz dismissed sbcd90’s stale review via b8e6334 June 23, 2023 18:40

tests for QueryBackend with no mappings

2cfcb39

Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com>

petardz mentioned this pull request Jun 28, 2023

[BUG] - Missing Sigma field CommandLine for Windows category #380

Closed

sbcd90 approved these changes Jul 5, 2023

View reviewed changes

praveensameneni approved these changes Jul 5, 2023

View reviewed changes

sbcd90 added the backport 2.x label Jul 5, 2023

sbcd90 merged commit faccf6c into opensearch-project:main Jul 5, 2023

opensearch-trigger-bot bot mentioned this pull request Jul 5, 2023

[Backport 2.x] New Log Type JSON format #474

Merged

sbcd90 added a commit that referenced this pull request Jul 11, 2023

New Log Type JSON format (#465)

b9f3a1e

* New Log Type JSON format Signed-off-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Subhobrata Dey <sbcd90@gmail.com> (cherry picked from commit faccf6c)

amsiglan mentioned this pull request Sep 28, 2023

[BUG] Missing Image mapping field #410

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Log Type JSON format#465

New Log Type JSON format#465
sbcd90 merged 15 commits intoopensearch-project:mainfrom
petardz:new-logtype-json-format

petardz commented Jun 20, 2023

Uh oh!

codecov bot commented Jun 21, 2023 •

edited

Loading

Uh oh!

sbcd90 Jun 22, 2023

Uh oh!

petardz Jun 22, 2023

Uh oh!

sbcd90 Jun 22, 2023

Uh oh!

petardz Jun 22, 2023

Uh oh!

sbcd90 Jun 22, 2023

Uh oh!

petardz Jun 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		public class LogType implements Writeable, ToXContentObject {

		private static final String ID = "id";

Conversation

petardz commented Jun 20, 2023

Description

Issues Resolved

Check List

Uh oh!

codecov bot commented Jun 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sbcd90 Jun 22, 2023

Choose a reason for hiding this comment

Uh oh!

petardz Jun 22, 2023

Choose a reason for hiding this comment

Uh oh!

sbcd90 Jun 22, 2023

Choose a reason for hiding this comment

Uh oh!

petardz Jun 22, 2023

Choose a reason for hiding this comment

Uh oh!

sbcd90 Jun 22, 2023

Choose a reason for hiding this comment

Uh oh!

petardz Jun 22, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Jun 21, 2023 •

edited

Loading