[symantec_endpoint_security] Event mapping improvements by chrisberkhout · Pull Request #13671 · elastic/integrations

chrisberkhout · 2025-04-24T14:34:36Z

Proposed commit message

[symantec_endpoint_security] Event mapping improvements

Added parsing of embedded JSON data, additional ECS mappings and one ECS
mapping improvement.

Discussion

Can be reviewed commit-by-commit.

Suggested mappings done with adjustments:

Suggestion: ses.actor.file.folder -> file.path
Done as ses.actor.file.folder -> file.directory to match ECS.
Suggestion: ses.cybox.files.folder -> file.path
Done as ses.cybox.files.folder -> file.directory to match ECS.
Suggestion: ses.file.folder -> file.path
Done as ses.file.folder -> file.directory to match ECS.
Suggestion: ses.connection.url.host -> url.original
Done as ses.connection.url.host -> url.domain to match ECS.
Suggestion: ses.connection.url.domain_name -> url.domain
Done as ses.connection.url.host -> url.domain to use the Symantec-documented URL field.
Suggestion: ses.process.file.original_name -> process.name
Done as ses.process.file.original_name -> file.name (as was already done for ses.process.file.name).
Suggestion: ses.reg_value_result.data -> registry.data.strings
Done as ses.reg_value_result.data -> registry.data.bytes because the sample value is base64, so it belongs in .bytes not .strings.
Suggestion: ses.reg_value.name -> registry.value
Done. Also changed prevous mapping ses.reg_value.data -> registry.value, to instead go to registry.data.bytes.
Suggestion: ses.product_data.bpe_description (from JSON) -> threat.enrichments.indicator.description (or message)
Done. Now ses.product_data is parsed if JSON, and message if empty is set to ses.product_data.bpe_description if present.

Suggested mappings not done:

Suggestion: ses.cybox.files.product_name -> file.name
Not done because product_name doesn’t seem to be a file name with extension. The name field exists and is mapped to ECS’s file.name.
Suggestion: ses.connection.dst_name -> related.host
Not done. Should already be done by the processor with the tag append_related_host_from_connection_dst_name.
Suggestion: ses.edr_enriched_data.rule_description -> message
Not done. We don’t have edr_enriched_data as a defined field, and our sample data doesn’t have it. According to the Symantec documentation, all the given event types all have a recommended message field, which is mapped from ses.message to message. None of them have a documented field with enriched in the name. I’d like to understand why these events aren’t getting message (ses.message) populated. If ses.edr_enriched_data.rule_description is a good fallback source for a message value, I’d at least like to have some sample data that shows that.
Suggestion: ses.data -> message
Not done. I don’t think message is the right field for this, since message is expected to be human-readable rather than structured data. Everything will be decoded under winlog.event_data.*, so I’m not sure that duplicating it under message (as well as ses.data) has any utility.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Closes [symantec_endpoint_security] Comprehensive mapping updates for various SES events #13476

elasticmachine · 2025-04-24T14:34:39Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6 · 2025-04-29T06:25:58Z

packages/symantec_endpoint_security/data_stream/event/fields/winlog.yml

How were these field removals/final inclusions chosen?

I included the ones that were mentioned in the request, plus some others that looked like they might be good to use in a query: ClientMachine, Reason, StartTime, State, SubjectDomain, User, Version.

efd6 · 2025-04-29T06:29:18Z

...dpoint_security/data_stream/event/elasticsearch/ingest_pipeline/pipeline_category_system.yml

+      description: Parse Windows Event data and store it in the same fields as the winlog integration does
+      field: ses.data
+      target_field: winlog.event_data
+      if: ctx.ses?.type_id == '8015' && ctx.ses?.data instanceof String && ctx.ses.data.length() > 0


ctx.ses?.type_id == '8015' && ctx.ses.data instanceof String && ctx.ses.data.length() > 0

efd6 · 2025-04-29T06:30:48Z

...dpoint_security/data_stream/event/elasticsearch/ingest_pipeline/pipeline_category_system.yml

+  - set:
+      field: winlog.event_id
+      copy_from: ses.ref_event
+      if: ctx.ses?.type_id == '8015' && ctx.ses?.ref_event != null


ctx.ses?.type_id == '8015' && ctx.ses.ref_event != null

efd6 · 2025-04-29T06:33:06Z

...c_endpoint_security/data_stream/event/elasticsearch/ingest_pipeline/pipeline_object_file.yml

+      field: file.hash.sha256
+      copy_from: ses.file.sha2
+      tag: set_file_hash_sha256_from_file_sha2
+      if: ctx.ses?.file?.sha2 != null && ctx.ses.file.sha2 != ''


instanceof String?

efd6 · 2025-04-29T07:05:14Z

...ndpoint_security/data_stream/event/elasticsearch/ingest_pipeline/pipeline_object_process.yml

+      field: process.hash.sha256
+      copy_from: ses.process.file.sha2
+      tag: set_file_hash_sha256_from_process_file_sha2
+      if: ctx.ses?.process?.file?.sha2 != null && ctx.ses.process.file.sha2 != ''


instanceof String?

efd6 · 2025-04-29T07:08:08Z

...endpoint_security/data_stream/event/elasticsearch/ingest_pipeline/pipeline_object_attack.yml

+            - _ingest._value.sub_technique_name
+          tag: remove_custom_duplicate_fields_from_ses_attacks
+          ignore_missing: true
+          if: ctx.tags?.contains('preserve_duplicate_custom_fields') != true


Isn't it the case that this condition must be true here given that it must have been true to run the foreach?

Correct. At the time I was thinking it's better to repeat the condition in case parts are moved around, but that doesn't actually make much sense here. I've removed it.

efd6

LGTM after conflicts are resolved.

Agree with the adjustments and the wont-do list in the description. Is it worth filing a follow-up for ses.edr_enriched_data.rule_description -> message?

…at elastic-package now generates.

…d limit of 2048.

…t "symantec_endpoint_security".

…CS fields. Specifically: ses.actor.file.folder -> file.directory ses.actor.file.original_name -> file.name ses.actor.user.sid -> user.id ses.actor.user.sid -> related.user

This follows the existing mapping of ses.process.file.name to file.name.

- Map `ses.reg_value.path` to `registry.path`. - Change mapping of `ses.reg_value.data` from to `registry.value` to `registry.data.bytes`. - Map `ses.reg_value.name` to `registry.value` (which is defined in ECS as "Name of the value written"). - Map `ses.reg_value_result.data` to `registry.data.bytes`.

chrisberkhout · 2025-05-02T10:36:58Z

LGTM after conflicts are resolved.

Agree with the adjustments and the wont-do list in the description. Is it worth filing a follow-up for ses.edr_enriched_data.rule_description -> message?

Thanks. Seems like a good one to leave out since it's not in the documented fields, but I'll double check with the requestor and do a follow-up if it does make sense.

elastic-vault-github-plugin-prod · 2025-05-02T10:58:48Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elastic-sonarqube · 2025-05-02T10:59:29Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
85.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-05-02T10:59:43Z

💚 Build Succeeded

Buildkite Build
Commit: 2239113

History

💚 Build #25283 succeeded 6da5dff
💚 Build #25099 succeeded 0926136

cc @chrisberkhout

elastic-vault-github-plugin-prod · 2025-05-02T11:15:45Z

Package symantec_endpoint_security - 1.11.0 containing this change is available at https://epr.elastic.co/package/symantec_endpoint_security/1.11.0/

chrisberkhout added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Integration:symantec_endpoint_security Symantec Endpoint Security labels Apr 24, 2025

chrisberkhout self-assigned this Apr 24, 2025

chrisberkhout requested a review from a team as a code owner April 24, 2025 14:34

efd6 reviewed Apr 29, 2025

View reviewed changes

chrisberkhout requested a review from efd6 April 29, 2025 13:49

efd6 approved these changes Apr 30, 2025

View reviewed changes

chrisberkhout added 21 commits May 2, 2025 12:28

Add newlines at end of pipeline test expected output, since that's wh…

037bc15

…at elastic-package now generates.

Add winlog.event_data.* fields from the winlog integration.

30aae9d

Add extra Windows Event fields.

b96bc7a

Remove most explicit winlog.event_data.* fields to come in under fiel…

3ca8cd3

…d limit of 2048.

Update README.

4050995

Parse and map Windows Event data.

1fc7881

Add winlog.event_id (value from ses.ref_event).

15f9685

Set event.module to ses.feature_name instead of the redundant constan…

3845cbd

…t "symantec_endpoint_security".

Set file and process hash ECS fields.

3950290

Set risk.calcuated_score and risk.calculated_level fields.

e282731

Populate threat.tactic.id and threat.technique.* fields.

7c29209

Set group.name from ses.device_group.

b27d085

Populate host.ip and host.mac from ses.device_networks.*.

f6e0547

Map ses.actor.file.{folder,original_name} and ses.actor.user.sid to E…

c70078f

…CS fields. Specifically: ses.actor.file.folder -> file.directory ses.actor.file.original_name -> file.name ses.actor.user.sid -> user.id ses.actor.user.sid -> related.user

Map ses.cybox.files.folder to file.directory.

b5fde2c

Map ses.file.folder to file.directory.

e35f608

Map ses.process.file.original_name to file.name.

b8ed662

This follows the existing mapping of ses.process.file.name to file.name.

Map ses.lineage to process.previous.executable.

9395fd2

Use populate message with ses.type if ses.message is absent or empty.

9c6ceb4

Append ses.process.user.name to user.name.

d862e27

Map ses.connection.url.{scheme,host} to url.{scheme,domain}.

404a204

chrisberkhout added 8 commits May 2, 2025 12:29

Parse ses.product_data if it's a JSON string.

ae18bc8

Set message if empty to ses.product_data.bpe_description if present.

c19d399

Update sample_event.json for event data stream.

3d3f6a8

Version bump, changelog entry.

ab60298

Remove redunant null check.

622bee7

Set ECS hash fields only if the value is a string.

80edff8

Remove redundant condition.

2239113

chrisberkhout force-pushed the symantec_mapping_improvements branch from 6da5dff to 2239113 Compare May 2, 2025 10:32

chrisberkhout enabled auto-merge (squash) May 2, 2025 10:34

chrisberkhout merged commit ee25461 into elastic:main May 2, 2025
7 checks passed

Conversation

chrisberkhout commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Discussion

Checklist

Related issues

Uh oh!

elasticmachine commented Apr 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

chrisberkhout commented May 2, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented May 2, 2025

🚀 Benchmarks report

Uh oh!

elastic-sonarqube bot commented May 2, 2025

Quality Gate passed

Uh oh!

elasticmachine commented May 2, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented May 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chrisberkhout commented Apr 24, 2025 •

edited

Loading