Skip to content

[O365]Moving edge processing to ingest pipelines#983

Merged
marc-gr merged 12 commits intoelastic:masterfrom
P1llus:package_o365_remove_edge_processing
Jun 9, 2021
Merged

[O365]Moving edge processing to ingest pipelines#983
marc-gr merged 12 commits intoelastic:masterfrom
P1llus:package_o365_remove_edge_processing

Conversation

@P1llus
Copy link
Copy Markdown
Member

@P1llus P1llus commented May 11, 2021 •
edited
Loading

What does this PR do?

This PR removes all edge processing in favor of ingest pipelines

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.

Related issues

@P1llus P1llus self-assigned this May 11, 2021
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 11, 2021 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #983 updated

  • Start Time: 2021-06-09T07:55:07.088+0000

  • Duration: 17 min 23 sec

  • Commit: 663d9fb

Test stats 🧪

Test Results
Failed 0
Passed 25
Skipped 0
Total 25

Trends 🧪

Image of Build Times

Image of Tests

@P1llus P1llus requested a review from adriansr May 25, 2021 11:25
@P1llus P1llus marked this pull request as ready for review May 25, 2021 11:28
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 25, 2021

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

leehinman
leehinman reviewed May 25, 2021
View reviewed changes
packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
copy_from: o365audit.ObjectId
if: ctx.event?.code == "AzureActiveDirectory"
## AzureActiveDirectory Schema new user
- set:
Copy link
Copy Markdown
Contributor

@leehinman leehinman May 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a suggestion, I'm OK merging as is.

It looks likeevent.code and event.action are used several times to set ECS categorization. I'm wondering if we could generalize to a script that takes parameters. Get all the logic to make those decisions in one place.

Copy link
Copy Markdown
Member Author

@P1llus P1llus Jun 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback @leehinman. There is an ongoing discussion on exactly this, also for google_workspace, around things like performance and which way would be better.

If its okay with you I would like to do the same here, in which I leave it like this for now, as I feel it is better for performance, while keeping an eye on the overall discussion. If it turns out that there is no difference or the other way does not have any high impact on benchmarks I will go back and implement it on both.

@P1llus
Copy link
Copy Markdown
Member Author

P1llus commented Jun 1, 2021

run tests

@marc-gr
Copy link
Copy Markdown
Contributor

marc-gr commented Jun 8, 2021

/test

@marc-gr marc-gr mentioned this pull request Jun 9, 2021
Closed
43 tasks
@marc-gr marc-gr merged commit 8f67eb2 into elastic:master Jun 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leehinman leehinman leehinman left review comments

@marc-gr marc-gr marc-gr approved these changes

@adriansr adriansr Awaiting requested review from adriansr

Assignees

@P1llus P1llus

Labels

Integration:o365 Microsoft Office 365

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Convert o365's edge processing to Ingest Node pipeline

5 participants

@P1llus @elasticmachine @marc-gr @leehinman @jamiehynds