Skip to content

[azure] Handle single quotes objects inside message field#11728

Closed
zmoog wants to merge 0 commit intoelastic:mainfrom
zmoog:zmoog/azure-functions-logs-with-platformlogs
Closed

[azure] Handle single quotes objects inside message field#11728
zmoog wants to merge 0 commit intoelastic:mainfrom
zmoog:zmoog/azure-functions-logs-with-platformlogs

Conversation

@zmoog
Copy link
Copy Markdown
Contributor

@zmoog zmoog commented Nov 13, 2024

Proposed commit message

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@zmoog zmoog self-assigned this Nov 13, 2024
@zmoog zmoog added Integration:azure Azure Logs bugfix Pull request that fixes a bug issue Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels Nov 13, 2024
@zmoog zmoog changed the title Handle single quotes objects inside message field [azure] Handle single quotes objects inside message field Nov 13, 2024
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod bot commented Nov 13, 2024

🚀 Benchmarks report

Package azure 👍(8) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
identity_protection 4464.29 3311.26 -1153.03 (-25.83%) 💔
provisioning 2898.55 2262.44 -636.11 (-21.95%) 💔

To see the full report comment with /test benchmark fullreport

@elastic-sonarqube
Copy link
Copy Markdown

@elasticmachine
Copy link
Copy Markdown

💚 Build Succeeded

History

  • 💚 Build #18282 succeeded fc5d3c48b5b5e6fddae75fde003c3be9301f245f

cc @zmoog

zmoog added a commit that referenced this pull request Nov 14, 2024
… a string value (#11732)

Mitigates #11729

If the `azure.platformlogs.properties` field contains a string instead of the expected object, the pipeline renames it as `azure.platformlogs.properties.raw` field. 

This allows:

- Elasticsearch to index the log events instead of dropping them (status 400)
- Users to customize parsing of the `.raw` field in the `logs-azure.platformlogs@custom pipeline`

This avoids dropping log events while we build a complete and more robust [invalid JSON](#11729) handling solution with #11728
@botelastic
Copy link
Copy Markdown

botelastic bot commented Dec 13, 2024

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Dec 13, 2024
@zmoog
Copy link
Copy Markdown
Contributor Author

zmoog commented Jan 8, 2025

We do not want to decide how to manage the invalid JSON that comes from a few Azure services.

The invalid JSON containment approach #11732 is probably good enough. With this approach, users have all the options to decide how they want to process data.

@zmoog zmoog closed this Jan 8, 2025
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
… a string value (elastic#11732)

Mitigates elastic#11729

If the `azure.platformlogs.properties` field contains a string instead of the expected object, the pipeline renames it as `azure.platformlogs.properties.raw` field. 

This allows:

- Elasticsearch to index the log events instead of dropping them (status 400)
- Users to customize parsing of the `.raw` field in the `logs-azure.platformlogs@custom pipeline`

This avoids dropping log events while we build a complete and more robust [invalid JSON](elastic#11729) handling solution with elastic#11728
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
… a string value (elastic#11732)

Mitigates elastic#11729

If the `azure.platformlogs.properties` field contains a string instead of the expected object, the pipeline renames it as `azure.platformlogs.properties.raw` field. 

This allows:

- Elasticsearch to index the log events instead of dropping them (status 400)
- Users to customize parsing of the `.raw` field in the `logs-azure.platformlogs@custom pipeline`

This avoids dropping log events while we build a complete and more robust [invalid JSON](elastic#11729) handling solution with elastic#11728
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Pull request that fixes a bug issue Integration:azure Azure Logs Stalled Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants