Skip to content

ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error"#12046

Merged
efd6 merged 0 commit intoelastic:mainfrom
efd6:12045-ssi-all
Dec 11, 2024
Merged

ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error"#12046
efd6 merged 0 commit intoelastic:mainfrom
efd6:12045-ssi-all

Conversation

@efd6
Copy link
Copy Markdown
Contributor

@efd6 efd6 commented Dec 10, 2024 •
edited
Loading

Proposed commit message

ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error"

Omit problematic packages: google_workspace, jamf_protect and ti_mandiant_advantage.

[git-generate]
for f in $(
    (
        for p in $(
            yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
            | grep -v -- ---
        ); do
            rg -l -g 'default.yml' "value: pipeline_error" packages/$p
        done
    )|sort|uniq|egrep -v 'google_workspace|jamf_protect|ti_mandiant_advantage'
); do
    (grep 'value: preserve_original_event' $f >/dev/null 2>&1) && continue
    perl -i -pe 'BEGIN{undef $/;} s/([a-z:"])
( *)(- set:.*value: pipeline_error)/$1
$2$3
$2- append:
$2    field: tags
$2    value: preserve_original_event
$2    allow_duplicates: false/smg' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
    (
        cd $p
        elastic-package changelog add \
            --description 'Add "preserve_original_event" tag to documents with `event.kind` set to "pipeline_error".' \
            --type enhancement \
            --next minor \
            --link https://github.com/elastic/integrations/pull/12046
    )>/dev/null 2>&1
done

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@efd6 efd6 added enhancement New feature or request Integration:All Applies to all integrations [Integration not found in source] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Dec 10, 2024
@efd6 efd6 self-assigned this Dec 10, 2024
@andrewkroh
Copy link
Copy Markdown
Member

andrewkroh commented Dec 10, 2024

I think we will want allow_duplicates: false on that append processor.

@efd6
Copy link
Copy Markdown
Contributor Author

efd6 commented Dec 10, 2024 •
edited
Loading

Yeah, it's not just that. There is some weirdness with the perl mutation that is brittle. I'm trying to fix other parts to see if that will fix this (I'm not optimistic).

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 10, 2024 •
edited
Loading

🚀 Benchmarks report

Package tines 👍(0) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
time_saved 35714.29 25000 -10714.29 (-30%) 💔

To see the full report comment with /test benchmark fullreport

@efd6 efd6 marked this pull request as ready for review December 10, 2024 08:12
@efd6 efd6 requested review from a team as code owners December 10, 2024 08:12
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 10, 2024

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@andrewkroh andrewkroh added Integration:carbon_black_cloud VMware Carbon Black Cloud Integration:1password 1Password (Partner supported) Integration:cisco_duo Cisco Duo Integration:atlassian_confluence Atlassian Confluence (Community supported) Integration:cloudflare Cloudflare (Community supported) Integration:entityanalytics_entra_id Microsoft Entra ID Entity Analytics Integration:atlassian_bitbucket Atlassian Bitbucket (Community supported) Integration:akamai Akamai (Community supported) Integration:bitwarden Bitwarden labels Dec 10, 2024
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_crowdstrike - 2.1.0 containing this change is available at https://epr.elastic.co/package/ti_crowdstrike/2.1.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_custom - 0.4.0 containing this change is available at https://epr.elastic.co/package/ti_custom/0.4.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_cybersixgill - 1.31.0 containing this change is available at https://epr.elastic.co/package/ti_cybersixgill/1.31.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_eclecticiq - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_eclecticiq/1.3.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_eset - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_eset/1.3.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_maltiverse - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_maltiverse/1.3.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_misp - 1.36.0 containing this change is available at https://epr.elastic.co/package/ti_misp/1.36.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_opencti - 2.4.0 containing this change is available at https://epr.elastic.co/package/ti_opencti/2.4.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_otx - 1.26.0 containing this change is available at https://epr.elastic.co/package/ti_otx/1.26.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_rapid7_threat_command - 2.1.0 containing this change is available at https://epr.elastic.co/package/ti_rapid7_threat_command/2.1.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_recordedfuture - 1.27.0 containing this change is available at https://epr.elastic.co/package/ti_recordedfuture/1.27.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_threatconnect - 1.4.0 containing this change is available at https://epr.elastic.co/package/ti_threatconnect/1.4.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package ti_threatq - 1.29.0 containing this change is available at https://epr.elastic.co/package/ti_threatq/1.29.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package tines - 1.13.0 containing this change is available at https://epr.elastic.co/package/tines/1.13.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package trellix_edr_cloud - 1.3.0 containing this change is available at https://epr.elastic.co/package/trellix_edr_cloud/1.3.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package trellix_epo_cloud - 1.12.0 containing this change is available at https://epr.elastic.co/package/trellix_epo_cloud/1.12.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package trend_micro_vision_one - 1.22.0 containing this change is available at https://epr.elastic.co/package/trend_micro_vision_one/1.22.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package trendmicro - 2.4.0 containing this change is available at https://epr.elastic.co/package/trendmicro/2.4.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package tychon - 0.2.0 containing this change is available at https://epr.elastic.co/package/tychon/0.2.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package vectra_detect - 1.10.0 containing this change is available at https://epr.elastic.co/package/vectra_detect/1.10.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package wiz - 2.5.0 containing this change is available at https://epr.elastic.co/package/wiz/2.5.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package zerofox - 1.26.0 containing this change is available at https://epr.elastic.co/package/zerofox/1.26.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package zeronetworks - 1.16.0 containing this change is available at https://epr.elastic.co/package/zeronetworks/1.16.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package zoom - 1.21.0 containing this change is available at https://epr.elastic.co/package/zoom/1.21.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package zscaler_zia - 3.5.0 containing this change is available at https://epr.elastic.co/package/zscaler_zia/3.5.0/

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Dec 11, 2024

Package zscaler_zpa - 1.19.0 containing this change is available at https://epr.elastic.co/package/zscaler_zpa/1.19.0/

This was referenced Dec 11, 2024
Merged
Merged
Merged
@andrewkroh andrewkroh mentioned this pull request Sep 17, 2025
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kcreddy kcreddy kcreddy approved these changes

@shmsr shmsr shmsr approved these changes

@ishleenk17 ishleenk17 ishleenk17 approved these changes

Assignees

@efd6 efd6

Labels

enhancement New feature or request Integration:abnormal_security Abnormal AI Integration:akamai Akamai (Community supported) Integration:amazon_security_lake Amazon Security Lake Integration:atlassian_bitbucket Atlassian Bitbucket (Community supported) Integration:atlassian_confluence Atlassian Confluence (Community supported) Integration:atlassian_jira Atlassian Jira (Community supported) Integration:auth0 Auth0 Integration:authentik authentik Integration:aws_bedrock Amazon Bedrock Integration:azure_frontdoor Azure Frontdoor (Community supported) Integration:azure_network_watcher_nsg Azure Network Watcher NSG Integration:azure_network_watcher_vnet Azure Network Watcher VNet Integration:barracuda_cloudgen_firewall Barracuda CloudGen Firewall Logs Integration:barracuda Barracuda Web Application Firewall Integration:bbot BBOT (Bighuge BLS OSINT Tool) (Community supported) Integration:bitdefender BitDefender (Community supported) Integration:bitwarden Bitwarden Integration:blacklens blacklens.io (Community supported) Integration:box_events Box Events Integration:canva Canva Integration:carbon_black_cloud VMware Carbon Black Cloud Integration:carbonblack_edr VMware Carbon Black EDR Integration:checkpoint_email Check Point Harmony Email & Collaboration Integration:checkpoint_harmony_endpoint Check Point Harmony Endpoint Integration:cisa_kevs CISA Known Exploited Vulnerabilities (Community supported) Integration:cisco_duo Cisco Duo Integration:cisco_meraki Cisco Meraki Integration:cisco_secure_endpoint Cisco Secure Endpoint Integration:cisco_umbrella Cisco Umbrella Integration:claroty_ctd Claroty CTD Integration:cloudflare_logpush Cloudflare Logpush Integration:cloudflare Cloudflare (Community supported) Integration:crowdstrike CrowdStrike Integration:cyberark_pta Cyberark Privileged Threat Analytics Integration:cyberarkpas CyberArk Privileged Access Security Integration:cybereason Cybereason Integration:cylance CylanceProtect Logs (Deprecated) Integration:darktrace Darktrace Integration:digital_guardian Digital Guardian Integration:entityanalytics_ad Active Directory Entity Analytics Integration:entityanalytics_entra_id Microsoft Entra ID Entity Analytics Integration:entityanalytics_okta Okta Entity Analytics Integration:eset_protect ESET PROTECT Integration:f5_bigip F5 BIG-IP Integration:f5 F5 Logs (Deprecated) [Integration not found in source] Integration:1password 1Password (Partner supported) Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@efd6 @andrewkroh @elasticmachine @kcreddy @shmsr @ishleenk17