[Azure Application Gateway] Fix parsing error client port is blank and adjust for timeStamp by nicpenning · Pull Request #5073 · elastic/integrations

nicpenning · 2023-01-20T21:17:18Z

Pipeline fails to parse Azure Application Gateway logs when the client port field exists with no data.

error.message : For input string: \"\" convert

What does this PR do?

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Pipeline fails to parse Azure Application Gateway logs when the client port field exists with no data. ``` error.message : For input string: \"\" convert ```

elasticmachine · 2023-01-20T21:20:33Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-02-03T15:55:17.758+0000
Duration: 15 min 46 sec

Test stats 🧪

Test	Results
Failed	0
Passed	124
Skipped	0
Total	124

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

zmoog · 2023-01-24T11:25:39Z

Hey @nicpenning, thank you for contributing to the Azure Application Gateway integration with this fix!

packages/azure/data_stream/application_gateway/elasticsearch/ingest_pipeline/default.yml

The fields are not in the same order but is very similar to other examples except with the blank clientPort field.

For some reason, the _ingest.timestamp is outside of the _source field. I have ommited it from here. Furthermore, it seems timestamp changed to timeStamp, so the pipeline will need to be updated.

nicpenning · 2023-01-25T02:11:24Z

When I was going through and trying to provide some example data I stumbled upon another issue which was the timestamp in the document I had was using timeStamp so I added that to the pipeline.

packages/azure/data_stream/application_gateway/elasticsearch/ingest_pipeline/default.yml

nicpenning · 2023-01-27T13:15:44Z

Not sure where my comment went, but yes, this data is from the Azure Gov cloud. 😀

packages/azure/data_stream/application_gateway/elasticsearch/ingest_pipeline/default.yml

nicpenning · 2023-01-27T15:53:49Z

Okay @zmoog, how is everything looking now?

zmoog · 2023-01-31T21:41:25Z

/test

elasticmachine · 2023-01-31T21:55:48Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`10/10`)	💚
Files	86.364% (`19/22`)	👎 -13.636
Classes	86.364% (`19/22`)	👎 -13.636
Methods	83.422% (`156/187`)	👎 -16.578
Lines	85.07% (`2792/3282`)	👍 14.272
Conditionals	100.0% (`0/0`)	💚

nicpenning · 2023-02-02T13:05:57Z

Any further action needed for this? I don't want this to get too stale.

zmoog · 2023-02-03T15:19:08Z

Any further action needed for this? I don't want this to get too stale.

Yep, someone must check and fix the errors in the CI pipeline. I can't merge the PR if they aren't all green.

Let me take a look at the failing one.

zmoog · 2023-02-03T15:19:25Z

/test

The pipeline test was failing due to differences between the expected and actual document. I fixed the pipeline tests error by regenerating the expected document using the following command: elastic-package test pipeline --generate The `--generate` option runs the pipeline and save the result in a test result file.

zmoog · 2023-02-03T16:19:33Z

@nicpenning, the bot will comment on this PR when the new version is available.

Thank you for your contribution and your patience in these back and forth!

elasticmachine · 2023-02-03T16:37:25Z

Package azure - 1.5.6 containing this change is available at https://epr.elastic.co/search?package=azure

Fix parsing error when no client port exists

b8afa0d

Pipeline fails to parse Azure Application Gateway logs when the client port field exists with no data. ``` error.message : For input string: \"\" convert ```

nicpenning requested a review from a team as a code owner January 20, 2023 21:17

nicpenning added 2 commits January 20, 2023 15:21

Update changelog.yml

6ea8f62

Update manifest.yml

b15a39b

nicpenning changed the title ~~Fix parsing error when no client port exists~~ [azure-application-gateway] Fix parsing error when no client port exists Jan 21, 2023

nicpenning changed the title ~~[azure-application-gateway] Fix parsing error when no client port exists~~ [Azure Application Gateway] Fix parsing error when no client port exists Jan 21, 2023

zmoog self-requested a review January 24, 2023 11:25

zmoog added bug Something isn't working, use only for issues Team:Cloud-Monitoring Label for the Cloud Monitoring team labels Jan 24, 2023

zmoog self-assigned this Jan 24, 2023

zmoog reviewed Jan 24, 2023

View reviewed changes

packages/azure/data_stream/application_gateway/elasticsearch/ingest_pipeline/default.yml Show resolved Hide resolved

nicpenning added 3 commits January 24, 2023 07:33

Update test-application-gateway-raw.log

7ba73d1

The fields are not in the same order but is very similar to other examples except with the blank clientPort field.

Update test-application-gateway-raw.log-expected.json

39dd991

For some reason, the _ingest.timestamp is outside of the _source field. I have ommited it from here. Furthermore, it seems timestamp changed to timeStamp, so the pipeline will need to be updated.

Add new timeStamp field

f65fc46

nicpenning changed the title ~~[Azure Application Gateway] Fix parsing error when no client port exists~~ [Azure Application Gateway] Fix parsing error client port is blank and adjust for timeStamp Jan 24, 2023

Update changelog.yml

802a273

zmoog requested changes Jan 26, 2023

View reviewed changes

Change to lowercase timeStamp instead

1090c78

zmoog reviewed Jan 27, 2023

View reviewed changes

packages/azure/data_stream/application_gateway/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

packages/azure/data_stream/application_gateway/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

nicpenning requested a review from zmoog January 27, 2023 13:10

zmoog requested changes Jan 27, 2023

View reviewed changes

packages/azure/data_stream/application_gateway/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

Change the lowercase to rename

8fa9555

Correct field rename

7f9fd54

zmoog approved these changes Jan 31, 2023

View reviewed changes

zmoog merged commit 30fb89f into elastic:main Feb 3, 2023

nicpenning deleted the patch-2 branch February 3, 2023 16:32

Conversation

nicpenning commented Jan 20, 2023

What does this PR do?

Checklist

Uh oh!

elasticmachine commented Jan 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

zmoog commented Jan 24, 2023

Uh oh!

Uh oh!

nicpenning commented Jan 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nicpenning commented Jan 27, 2023

Uh oh!

Uh oh!

nicpenning commented Jan 27, 2023

Uh oh!

zmoog commented Jan 31, 2023

Uh oh!

elasticmachine commented Jan 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

nicpenning commented Feb 2, 2023

Uh oh!

zmoog commented Feb 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zmoog commented Feb 3, 2023

Uh oh!

zmoog commented Feb 3, 2023

Uh oh!

elasticmachine commented Feb 3, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

elasticmachine commented Jan 20, 2023 •

edited

Loading

nicpenning commented Jan 25, 2023 •

edited

Loading

elasticmachine commented Jan 31, 2023 •

edited

Loading

zmoog commented Feb 3, 2023 •

edited

Loading