Skip to content

{microsoft_defender_endpoint, m365_defender}.vulnerability: New API implementation#15603

Merged
efd6 merged 15 commits intoelastic:mainfrom
kcreddy:mde-vuln-reqrite
Oct 12, 2025
Merged

{microsoft_defender_endpoint, m365_defender}.vulnerability: New API implementation#15603
efd6 merged 15 commits intoelastic:mainfrom
kcreddy:mde-vuln-reqrite

Conversation

@kcreddy
Copy link
Copy Markdown
Contributor

@kcreddy kcreddy commented Oct 8, 2025 •
edited
Loading

Proposed commit message

{microsoft_defender_endpoint, m365_defender}.vulnerability: New API implementation

Existing CEL program uses 3 API endpoints to fetch vulnerability
data. Although we fetch more fields using this approach, it doesn't 
scale well and hence is unusable even for few hundred machines.

This PR updates the vulnerability data stream with new 
SoftwareVulnerabilitiesExport API[1], which is recommended for 
larger workloads. While there are few data points missed in this 
new implementation[2], we maintain all the required fields for 
3rd party vulnerability workflow[3].

Other changes:
- Updates microsoft_defender_endpoint min stack version to "8.19.3"
  as the permissions for the transform were actually applied in 
  "8.19.3" version, and not in "8.19.2"[4].
- Add dataset filter to all visualisations of vulnerability dashboards.

[1]: https://learn.microsoft.com/en-us/defender-endpoint/api/get-assessment-software-vulnerabilities#2-export-software-vulnerabilities-assessment-via-files
[2]: https://github.com/elastic/integrations/issues/15521#issuecomment-3380969284
[3]: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide
[4]: https://github.com/elastic/elasticsearch/pull/132629

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • Verified there are no prebuilt rules effected by this change.
  • Update dashboards to remove references to old fields.
  • Add video for upgrade flow.

How to test this PR locally

Pipeline tests and System tests pass.

--- Test results for package: m365_defender - START ---
╭───────────────┬───────────────┬───────────┬───────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM   │ TEST TYPE │ TEST NAME                                         │ RESULT │ TIME ELAPSED │
├───────────────┼───────────────┼───────────┼───────────────────────────────────────────────────┼────────┼──────────────┤
│ m365_defender │ vulnerability │ pipeline  │ (ingest pipeline warnings test-vulnerability.log) │ PASS   │ 480.561041ms │
│ m365_defender │ vulnerability │ pipeline  │ test-vulnerability.log                            │ PASS   │  87.309125ms │
╰───────────────┴───────────────┴───────────┴───────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: m365_defender - END   ---
Done

Related issues

Screenshots

Updated dashboards:

m365-defender-vulnerability microsoft_defender_endpoint-vulnerability_overview

Upgrade Flow

mde-vulnerability-3.1-to-3.2-upgrade-flow.mp4

@kcreddy kcreddy mentioned this pull request Oct 8, 2025
Closed
@andrewkroh andrewkroh added Integration:microsoft_defender_endpoint Microsoft Defender for Endpoint dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. labels Oct 8, 2025
@kcreddy kcreddy marked this pull request as ready for review October 9, 2025 19:24
@kcreddy kcreddy requested a review from a team as a code owner October 9, 2025 19:24
@kcreddy kcreddy added the Integration:m365_defender Microsoft Defender XDR label Oct 9, 2025
@kcreddy kcreddy self-assigned this Oct 9, 2025
@andrewkroh andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Oct 9, 2025
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 9, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6
efd6 reviewed Oct 9, 2025
View reviewed changes
@kcreddy kcreddy requested a review from efd6 October 10, 2025 05:48
efd6
efd6 reviewed Oct 10, 2025
View reviewed changes
@kcreddy kcreddy requested a review from efd6 October 10, 2025 07:40
@kcreddy kcreddy changed the title {microsoft_defender_endpoint, m365_defender}.vulnerability: Update implementation {microsoft_defender_endpoint, m365_defender}.vulnerability: New API implementation Oct 10, 2025
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Oct 10, 2025 •
edited
Loading

🚀 Benchmarks report

Package microsoft_defender_endpoint 👍(2) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
machine 3086.42 2392.34 -694.08 (-22.49%) 💔

To see the full report comment with /test benchmark fullreport

@brijesh-elastic
Copy link
Copy Markdown
Collaborator

brijesh-elastic commented Oct 11, 2025

Since there is a change in the transform schema, the fleet_transform_version and destination indices need to be updated.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 12, 2025

💚 Build Succeeded

History

cc @kcreddy

@efd6 efd6 merged commit 7c2d2ef into elastic:main Oct 12, 2025
7 checks passed
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Oct 12, 2025

Package m365_defender - 5.0.0 containing this change is available at https://epr.elastic.co/package/m365_defender/5.0.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Oct 12, 2025

Package microsoft_defender_endpoint - 4.0.0 containing this change is available at https://epr.elastic.co/package/microsoft_defender_endpoint/4.0.0/

agithomas pushed a commit to agithomas/integrations that referenced this pull request Oct 30, 2025
@kcreddy @agithomas
{microsoft_defender_endpoint,m365_defender}.vulnerability: New API im…
6cfc264 
…plementation (elastic#15603)

Existing CEL program uses 3 API endpoints to fetch vulnerability
data. Although we fetch more fields using this approach, it doesn't 
scale well and hence is unusable even for few hundred machines.

This PR updates the vulnerability data stream with new 
SoftwareVulnerabilitiesExport API[1], which is recommended for 
larger workloads. While there are few data points missed in this 
new implementation[2], we maintain all the required fields for 
3rd party vulnerability workflow[3].

Other changes:
- Updates microsoft_defender_endpoint min stack version to "8.19.3"
  as the permissions for the transform were actually applied in 
  "8.19.3" version, and not in "8.19.2"[4].
- Add dataset filter to all visualisations of vulnerability dashboards.

[1]: https://learn.microsoft.com/en-us/defender-endpoint/api/get-assessment-software-vulnerabilities#2-export-software-vulnerabilities-assessment-via-files
[2]: elastic#15521 (comment)
[3]: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide
[4]: elastic/elasticsearch#132629
tehbooom pushed a commit to tehbooom/integrations that referenced this pull request Nov 19, 2025
@kcreddy
{microsoft_defender_endpoint,m365_defender}.vulnerability: New API im…
9e2f804 
…plementation (elastic#15603)

Existing CEL program uses 3 API endpoints to fetch vulnerability
data. Although we fetch more fields using this approach, it doesn't 
scale well and hence is unusable even for few hundred machines.

This PR updates the vulnerability data stream with new 
SoftwareVulnerabilitiesExport API[1], which is recommended for 
larger workloads. While there are few data points missed in this 
new implementation[2], we maintain all the required fields for 
3rd party vulnerability workflow[3].

Other changes:
- Updates microsoft_defender_endpoint min stack version to "8.19.3"
  as the permissions for the transform were actually applied in 
  "8.19.3" version, and not in "8.19.2"[4].
- Add dataset filter to all visualisations of vulnerability dashboards.

[1]: https://learn.microsoft.com/en-us/defender-endpoint/api/get-assessment-software-vulnerabilities#2-export-software-vulnerabilities-assessment-via-files
[2]: elastic#15521 (comment)
[3]: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide
[4]: elastic/elasticsearch#132629
@kcreddy kcreddy mentioned this pull request Nov 26, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@brijesh-elastic brijesh-elastic brijesh-elastic left review comments

@efd6 efd6 efd6 approved these changes

Assignees

@kcreddy kcreddy

Labels

breaking change dashboard Relates to a Kibana dashboard bug, enhancement, or modification. documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:m365_defender Microsoft Defender XDR Integration:microsoft_defender_endpoint Microsoft Defender for Endpoint Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

{microsoft_defender_endpoint, m365_defender}: vulnerability data stream scaling problem

5 participants

@kcreddy @elasticmachine @brijesh-elastic @efd6 @andrewkroh