Skip to content

[system] Add routing pipeline to security data_stream#2523

Merged
taylor-swanson merged 1 commit intoelastic:masterfrom
taylor-swanson:win-eventid-reuse
Jan 13, 2022
Merged

[system] Add routing pipeline to security data_stream#2523
taylor-swanson merged 1 commit intoelastic:masterfrom
taylor-swanson:win-eventid-reuse

Conversation

@taylor-swanson
Copy link
Copy Markdown
Contributor

@taylor-swanson taylor-swanson commented Jan 12, 2022

What does this PR do?

Added a routing pipeline to the security data_stream and limited the providers to Microsoft-Windows-Eventlog and Microsoft-Windows-Security-Auditing. Events that do not have these providers will still be indexed, but won't receive additional enrichment (which would likely lead to invalid data).

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

cd packages/system && elastic-package test
  • While I'm not sure how to generate logs from non-supported providers, I did test to make sure that existing logs from the supported providers (namely Microsoft-Windows-Security-Auditing ) are still being collected properly.

Related issues

@taylor-swanson
[System] Add routing pipeline to security data_stream, limit to speci…
9c6a862 
…fic providers

- Added a routing pipeline to the security data_stream and limited the providers to
Microsoft-Windows-Eventlog and Microsoft-Windows-Security-Auditing. Events that do
not have these providers will still be indexed, but won't receive additional
enrichment (which would likely lead to invalid data).
@taylor-swanson taylor-swanson added bug Something isn't working, use only for issues Team:Security-External Integrations labels Jan 12, 2022
@taylor-swanson
Copy link
Copy Markdown
Contributor Author

taylor-swanson commented Jan 12, 2022

Github isn't displaying the diff very well on this one, so here's another way of looking at it:

 4 files changed, 64 insertions(+), 3420 deletions(-)
 rewrite packages/system/data_stream/security/elasticsearch/ingest_pipeline/default.yml (98%)
 copy packages/system/data_stream/security/elasticsearch/ingest_pipeline/{default.yml => standard.yml} (99%)

I copied the original pipeline to standard.yml, it contains all of the specific enrichment processors for the providers I mentioned. Anything generic was moved/kept in default.yml, along with the routing logic. I changed very few lines in the end, despite what the stats say.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Jan 12, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-01-12T14:50:09.677+0000

  • Duration: 15 min 0 sec

  • Commit: 9c6a862

Test stats 🧪

Test Results
Failed 0
Passed 269
Skipped 0
Total 269

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@taylor-swanson taylor-swanson marked this pull request as ready for review January 12, 2022 15:05
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Jan 12, 2022

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@taylor-swanson taylor-swanson mentioned this pull request Jan 12, 2022
Merged
2 tasks
@taylor-swanson taylor-swanson merged commit 175d4ff into elastic:master Jan 13, 2022
@taylor-swanson taylor-swanson deleted the win-eventid-reuse branch January 13, 2022 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@efd6 efd6 efd6 approved these changes

Assignees

No one assigned

Labels

bug Something isn't working, use only for issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@taylor-swanson @elasticmachine @efd6