Skip to content

[AWS][CloudTrail]- Fix CloudTrail pipeline errors causing transient json and _conf fields to leak into indexed documents by cleaning them up in the on_failure handler, preventing dynamic mapping conflicts#17806

Merged
ShourieG merged 2 commits intoelastic:mainfrom
ShourieG:bugfix/aws_6984
Mar 16, 2026

Conversation

@ShourieG
Copy link
Copy Markdown
Contributor

@ShourieG ShourieG commented Mar 13, 2026

Type of change

  • Bug

Proposed commit message

aws/cloudtrail: clean up transient json and _conf fields in on_failure handler

When the ingest pipeline fails at any processor, the top-level on_failure
handler catches the error but does not remove the transient `json` and `_conf`
fields. These fields are normally removed at the end of a successful pipeline
run, but on failure they leak into the indexed document.

Because CloudTrail aggregates events from many AWS services with inconsistent
schemas (e.g. `json.requestParameters.filter` is a string from IAM Identity
Center SCIM but an object from AWS Health/EC2), the leaked `json.*` fields
create dynamic mappings on the first failure, then reject subsequent documents
with a different type for the same path, causing document_parsing_exception
errors and filling up the Logstash DLQ.

Adds a remove processor for `json` and `_conf` to the on_failure handler,
mirroring the cleanup already done in the normal pipeline flow.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

…s to leak into indexed documents by cleaning them up in the on_failure handler, preventing dynamic mapping conflicts
@ShourieG ShourieG self-assigned this Mar 13, 2026
@ShourieG ShourieG requested review from a team as code owners March 13, 2026 15:47
@ShourieG ShourieG added Integration:aws AWS bugfix Pull request that fixes a bug issue Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Mar 13, 2026
@elasticmachine
Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

🚀 Benchmarks report

Package aws 👍(9) 💚(3) 💔(10)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
firewall_logs 4444.44 3533.57 -910.87 (-20.49%) 💔
inspector 1141.55 910.75 -230.8 (-20.22%) 💔
s3access 5813.95 3787.88 -2026.07 (-34.85%) 💔
securityhub_insights 1358.7 1144.16 -214.54 (-15.79%) 💔
vpcflow 8547.01 5555.56 -2991.45 (-35%) 💔
waf 7246.38 5376.34 -1870.04 (-25.81%) 💔
cloudwatch_logs 1e+06 500000 -500000 (-50%) 💔
config 4901.96 3436.43 -1465.53 (-29.9%) 💔
ec2_logs 38461.54 32258.06 -6203.48 (-16.13%) 💔
ec2_metrics 34482.76 26315.79 -8166.97 (-23.68%) 💔

To see the full report comment with /test benchmark fullreport

@elasticmachine
Copy link
Copy Markdown

💚 Build Succeeded

cc @ShourieG

@ShourieG ShourieG merged commit 445584c into elastic:main Mar 16, 2026
9 checks passed
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

Package aws - 6.3.1 containing this change is available at https://epr.elastic.co/package/aws/6.3.1/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue Integration:aws AWS Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants