[Cloud Security] add misconfiguration latest transform to Wiz integration#10965
[Cloud Security] add misconfiguration latest transform to Wiz integration#10965maxcold merged 6 commits intoelastic:mainfrom maxcold:csp-add-wiz-misconfigruation-findings-transform
Conversation
|
/ci |
🚀 Benchmarks reportTo see the full report comment with |
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
| move_on_creation: true | ||
| latest: | ||
| unique_key: | ||
| - rule.uuid |
There was a problem hiding this comment.
Why are we using rule.uuid instead of rule.id? The native Transform utilizes rule.id, and I noticed that it is also part of the ECS schema.
There was a problem hiding this comment.
@CohenIdo this is a valid question, the reasons I chose rule.uuid for the unique key are the following:
- our native integration doesn't provide
rule.uuidbutrule.idhas the formatfe083488-fa0f-5408-9624-ac27607ac2ffwhich is basically the uuid as per ECS schema - As per ECS
rule.idis a unique id in some context and this context is narrower than forrule.uuid. In the context of Cloud Security it might be that the id is unique within the benchmark, but not unique across multiple benchmarks, therefore I thought usinguuidallow for better uniqueness, taking into account that our native also uses uuid, while providing it in therule.idfield
I will reflect this in the guide, sharing that the id should be unique across benchmarks
There was a problem hiding this comment.
For user convenience, I believe it's worth using the same field names across integrations.
However, if you decide to go with rule.uuid, let's at least create a ticket to migrate from rule.id to rule.uuid in the native integration.
There was a problem hiding this comment.
For user convenience, I believe it's worth using the same field names across integrations.
Overall I agree that consistency is better, but do you have a specific influence on the UX if we go for a different field between Wiz and CSP? I could not think of any specific downside for the users
However, if you decide to go with rule.uuid, let's at least create a ticket to migrate from rule.id to rule.uuid in the native integration.
I summarised all current mismatches between Wiz and our native CSP integration in this document https://docs.google.com/document/d/1GG3st6wR-9lNh4HjHsNlam5lo2oWceRK67-IcvyIVjk/edit , the mismatch between rule.uuid and rule.id is one of the points there. I will create GitHub issues after I gather some feedback on these points. Can you also review the document?
There was a problem hiding this comment.
One scenario is where a user is investigating misconfiguration documents and needs to determine the rule ID field for each integration. Or, when a user wants to add a rule ID column to the data grid and has both integrations installed, they would have to add two columns that represent the same information.
There was a problem hiding this comment.
@CohenIdo both use cases related to the rule.id vs rule.uuid in general, and not to which field to use in the transform as far as I understand. Whatever field we choose for the transform, these problems will stay due to our native integration not following ECS and providing uiud in rule.id and not providing rule.uuid at all. I believe we should not make changes in Wiz to make it not compliant with ECS just for it to be consistent with our native integration. I think we rather need to fix our native integration to be compliant with ECS and then the problem will be resolved properly
There was a problem hiding this comment.
The transform's manifest.yml could be defined as per this spec: https://github.com/elastic/package-spec/blob/main/spec/integration/elasticsearch/transform/manifest.spec.yml
It contains properties such as start, but more importantly destination_index_template.settings.index.sort which defines the sort order of documents within the index. According to Elasticsearch doc, we don't seem to have default field for index.sort.
By default Lucene does not apply any sort. The index.sort.* settings define which fields should be used to sort the documents inside each Segment.
So I think we have to define this manually. Example: https://github.com/elastic/integrations/blob/main/packages/ti_threatconnect/elasticsearch/transform/latest/manifest.yml
| - name: event.outcome | ||
| external: ecs | ||
| - name: observer.vendor | ||
| external: ecs No newline at end of file |
There was a problem hiding this comment.
There are few more ECS fields being populated in the source datastream's ingest pipeline, like cloud.region, event.category, etc. Shouldn't we be defininig them as well?
|
@kcreddy thanks for the code review. Regarding the |
|
💚 Build Succeeded
History
|
Based on the docs, I can see that when There are also caveats mentioned regarding performance while indexing the documents with
The default source data-stream backed indices of I agree that we should not set this property especially since you haven't experienced any downside yet in native integration. |
|
Package wiz - 2.0.0 containing this change is available at https://epr.elastic.co/search?package=wiz |
…tion (elastic#10965) * add misconfiguration latest transform to Wiz integration * add PR link to changelog * add ecs mapping to wiz latest cdr misconfigurations transform * add missing ecs fields to wiz misconfiguration transform mapping
…tion (elastic#10965) * add misconfiguration latest transform to Wiz integration * add PR link to changelog * add ecs mapping to wiz latest cdr misconfigurations transform * add missing ecs fields to wiz misconfiguration transform mapping
Proposed commit message
Introducing the latest transform for Wiz Cloud Configuration Finding data to support CDR workflows as per [Guide] Adapting data stream to Cloud Security in Kibana
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
elastic-package buildelastic-package stack up -v -dRelated issues
Screenshots