Skip to content

[AWS] Handle duplicate fields in Network Firewall Logs data stream#13676

Merged
ShourieG merged 3 commits intoelastic:mainfrom
moxarth-rathod:aws-handle-duplicate-fields
May 7, 2025
Merged

[AWS] Handle duplicate fields in Network Firewall Logs data stream#13676
ShourieG merged 3 commits intoelastic:mainfrom
moxarth-rathod:aws-handle-duplicate-fields

Conversation

@moxarth-rathod
Copy link
Copy Markdown
Contributor

@moxarth-rathod moxarth-rathod commented Apr 25, 2025

Proposed commit message

aws: network firewall logs ingest pipeline duplicate field error

For Network Firewall Logs, fixed the handling of duplicate fields error with the help of 
`allow_duplicate_keys` option in the JSON processor.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/aws directory.
  • Run the following command to run tests.

elastic-package test

Related issues

@moxarth-rathod moxarth-rathod added Integration:aws AWS bugfix Pull request that fixes a bug issue Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Apr 25, 2025
@moxarth-rathod moxarth-rathod self-assigned this Apr 25, 2025
@moxarth-rathod moxarth-rathod requested review from a team as code owners April 25, 2025 05:54
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Apr 25, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Apr 25, 2025

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@ShourieG
Copy link
Copy Markdown
Contributor

ShourieG commented May 7, 2025 •
edited
Loading

@moxarth-rathod, is there any scenario where this duplicate key could be present in the mapping logic ? If so, we should have test sample data that has such duplicates and see the result.

@moxarth-rathod
Copy link
Copy Markdown
Contributor Author

moxarth-rathod commented May 7, 2025

@moxarth-rathod, is there any scenario where this duplicate key could be present in the mapping logic ? If so, we should have test sample data that has such duplicates and see the result.

@ShourieG Yes, i've already added a test case to fulfil this scenario, in the new test case these fields - subject and issuerdn are duplicated.

@elastic-sonarqube
Copy link
Copy Markdown

elastic-sonarqube bot commented May 7, 2025

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 7, 2025

💚 Build Succeeded

History

cc @moxarth-rathod

ShourieG
ShourieG approved these changes May 7, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@ShourieG ShourieG left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShourieG ShourieG merged commit daf29cb into elastic:main May 7, 2025
7 checks passed
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented May 7, 2025

Package aws - 3.3.1 containing this change is available at https://epr.elastic.co/package/aws/3.3.1/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ShourieG ShourieG ShourieG approved these changes

Assignees

@moxarth-rathod moxarth-rathod

Labels

bugfix Pull request that fixes a bug issue Integration:aws AWS Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[AWS] Network Firewall logs ingest pipeline duplicate field error

3 participants

@moxarth-rathod @elasticmachine @ShourieG