[FIlebeat] add strict_date_optional_time_nanos date format to PanOS module by legoguy1000 · Pull Request #26158 · elastic/beats

legoguy1000 · 2021-06-07T03:38:45Z

What does this PR do?

Adds strict_date_optional_time_nanos date format to the date processors

Why is it important?

See #26033

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

cd beats/x-pack/filebeat
TESTING_FILEBEAT_MODULES=panw mage -v pythonIntegTest

Related issues

Closes Palo Alto | include "strict_date_optional_time_nanos" format for date fields #26033

Use cases

Screenshots

Logs

elasticmachine · 2021-06-07T03:43:11Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #26158 updated
Start Time: 2021-06-24T16:58:26.096+0000
Duration: 103 min 17 sec
Commit: 45b9cb2

Test stats 🧪

Test	Results
Failed	0
Passed	7396
Skipped	1201
Total	8597

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	7396
Skipped	1201
Total	8597

legoguy1000 · 2021-06-07T12:54:32Z

@jamiehynds Do you have sample data I can add to this to validate the changes??

mergify · 2021-06-08T01:34:07Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26033-panos-date upstream/26033-panos-date
git merge upstream/master
git push upstream 26033-panos-date

mergify · 2021-06-08T11:33:16Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26033-panos-date upstream/26033-panos-date
git merge upstream/master
git push upstream 26033-panos-date

jamiehynds · 2021-06-14T11:59:06Z

@jamiehynds Do you have sample data I can add to this to validate the changes??

@legoguy1000 working on getting sample data for you. Thanks as always for the contribution!

elasticmachine · 2021-06-14T11:59:30Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

jamiehynds · 2021-06-15T09:02:54Z

@legoguy1000 sample log below. Adjusted the IP's to ensure they are anonymised.

1,2021-05-26T16:27:07.000000Z,no-serial,TRAFFIC,end,9.1,2021-05-26T16:26:47.000000Z,127.0.0.0,127.0.0.1,127.0.0.2,127.0.0.3,intrazone-default,,,web-browsing,vsys1,untrust,untrust,ethernet1/1,ethernet1/1,Cortex Data Lake,,688290,1,35834,443,35834,20077,0x1400070,tcp,allow,7291,1696,5595,21,2021-05-26T16:26:30.000000Z,1,medium-risk,,620386,0x8800000000000000,US,SG,,14,7,tcp-fin,22,18,0,0,,GP cloud service,from-policy,,,0,,0,1970-01-01T00:00:00.000000Z,N/A,0,0,0,0,6a2f6161-88f2-4afc-8dd5-256bc4505a64,0,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

mergify · 2021-06-16T15:59:50Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26033-panos-date upstream/26033-panos-date
git merge upstream/master
git push upstream 26033-panos-date

P1llus · 2021-06-24T13:07:24Z

/test

mergify · 2021-06-24T13:22:31Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26033-panos-date upstream/26033-panos-date
git merge upstream/master
git push upstream 26033-panos-date

…odule (#26158) * #26033: add strict_date_optional_time_nanos format * update changelog * added new sample log, need to check CSV parsing * update sample data Co-authored-by: Marius Iversen <marius.iversen@elastic.co> (cherry picked from commit 7742fda) # Conflicts: # x-pack/filebeat/module/panw/panos/ingest/pipeline.yml # x-pack/filebeat/module/panw/panos/test/global_protect.log-expected.json # x-pack/filebeat/module/panw/panos/test/hipmatch.log-expected.json

* master: (32 commits) [Metricbeat] Change Account ID to Project ID in `gcp.billing` module (elastic#26412) update libbeat fields.ecs.yml file and ecsVersion to 1.10.0 (elastic#26121) [Filebeat] Update AWS ELB ingest pipeline (elastic#26441) [FIlebeat] add strict_date_optional_time_nanos date format to PanOS module (elastic#26158) Fix the irregular and typo on prometheus module. (elastic#25726) [Filebeat] Parse additonal debug data fields for Okta module (elastic#25818) fix: update MSSQL Server linux image's Docker registry (elastic#26440) Update indexing.go godocs (elastic#26408) Do not close filestream harvester if an unexpected error is returned when close.on_state_change.* is enabled (elastic#26411) Add support for copytruncate method when rotating input logs with an external tool in `filestream` input (elastic#23457) Allow fields with ip_range datatype (elastic#26444) Add Anomali ThreatStream support to threatintel module (elastic#26350) fix: use the right param type (elastic#26469) [Automation] Update elastic stack version to 8.0.0-7640093f for testing (elastic#26460) Set SM Filebeat modules as GA (elastic#26226) Fix rfc5464 date parsing in the syslog input (elastic#26419) Add linked account information into billing metricset (elastic#26285) [Filebeat] Update HA Proxy log grok patterns (elastic#25835) disable metricbeat logstash test_node_stats (elastic#26436) chore: pass BEAT_VERSION when running E2E tests (elastic#26291) ...

legoguy1000 · 2021-06-28T13:04:11Z

@Mergifyio refresh

mergify · 2021-06-28T13:04:48Z

Command refresh: success

Pull request refreshed

legoguy1000 · 2021-06-28T13:05:40Z

@Mergifyio backport 7.x

mergify · 2021-06-28T13:05:44Z

Command backport 7.x: success

Backports have been created

#26491 [7.x](backport #26158) [FIlebeat] add strict_date_optional_time_nanos date format to PanOS module has been created for branch 7.x

…odule (elastic#26158) * elastic#26033: add strict_date_optional_time_nanos format * update changelog * added new sample log, need to check CSV parsing * update sample data Co-authored-by: Marius Iversen <marius.iversen@elastic.co>

…odule (#26158) (#26525) * #26033: add strict_date_optional_time_nanos format * update changelog * added new sample log, need to check CSV parsing * update sample data Co-authored-by: Marius Iversen <marius.iversen@elastic.co> Co-authored-by: Alex Resnick <adr8292@gmail.com> Co-authored-by: Marius Iversen <marius.iversen@elastic.co>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 7, 2021

legoguy1000 force-pushed the 26033-panos-date branch from 45834ce to e124d09 Compare June 7, 2021 03:45

legoguy1000 marked this pull request as ready for review June 8, 2021 01:33

legoguy1000 force-pushed the 26033-panos-date branch from e124d09 to e5a9725 Compare June 8, 2021 02:22

jamiehynds added the Team:Security-External Integrations label Jun 14, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 14, 2021

jamiehynds added enhancement Integration:Palo Alto labels Jun 14, 2021

legoguy1000 force-pushed the 26033-panos-date branch from 00411ae to 7d3dac3 Compare June 15, 2021 14:22

legoguy1000 marked this pull request as draft June 24, 2021 12:06

legoguy1000 marked this pull request as ready for review June 24, 2021 12:57

legoguy1000 added 3 commits June 24, 2021 12:58

elastic#26033: add strict_date_optional_time_nanos format

6d2303f

update changelog

c55d3e1

added new sample log, need to check CSV parsing

6f91cc8

legoguy1000 force-pushed the 26033-panos-date branch from c706f92 to 6bf7f12 Compare June 24, 2021 12:58

update sample data

a5858b3

legoguy1000 force-pushed the 26033-panos-date branch from 6bf7f12 to a5858b3 Compare June 24, 2021 13:00

Merge branch 'master' into 26033-panos-date

45b9cb2

P1llus approved these changes Jun 24, 2021

View reviewed changes

P1llus merged commit 7742fda into elastic:master Jun 24, 2021

P1llus added the backport-v7.14.0 Automated backport with mergify label Jun 24, 2021

mergify bot mentioned this pull request Jun 24, 2021

[7.x](backport #26158) [FIlebeat] add strict_date_optional_time_nanos date format to PanOS module #26491

Closed

P1llus added backport-v7.14.0 Automated backport with mergify and removed backport-v7.14.0 Automated backport with mergify labels Jun 28, 2021

marc-gr added the needs_integration_sync Changes in this PR need synced to elastic/integrations. label Jun 28, 2021

legoguy1000 deleted the 26033-panos-date branch June 28, 2021 13:03

marc-gr mentioned this pull request Jun 28, 2021

Cherry-pick #26158 to 7.x: [FIlebeat] add strict_date_optional_time_nanos date format to PanOS module #26525

Merged

6 tasks

marc-gr added the v7.14.0 label Jun 28, 2021

Conversation

legoguy1000 commented Jun 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

elasticmachine commented Jun 7, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

legoguy1000 commented Jun 7, 2021

Uh oh!

mergify bot commented Jun 8, 2021

Uh oh!

mergify bot commented Jun 8, 2021

Uh oh!

jamiehynds commented Jun 14, 2021

Uh oh!

elasticmachine commented Jun 14, 2021

Uh oh!

jamiehynds commented Jun 15, 2021

Uh oh!

mergify bot commented Jun 16, 2021

Uh oh!

P1llus commented Jun 24, 2021

Uh oh!

mergify bot commented Jun 24, 2021

Uh oh!

legoguy1000 commented Jun 28, 2021

Uh oh!

mergify bot commented Jun 28, 2021

Uh oh!

legoguy1000 commented Jun 28, 2021

Uh oh!

mergify bot commented Jun 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

legoguy1000 commented Jun 7, 2021 •

edited

Loading

elasticmachine commented Jun 7, 2021 •

edited by jenkins-beats-ci bot

Loading