Add system test for Netflow Log (udp) by andrewkroh · Pull Request #590 · elastic/integrations

andrewkroh · 2021-01-31T14:39:03Z

What does this PR do?

Add a system test for the netflow input in the Netflow Log data stream.

Initially the errors were:

netflow/log netflow:
[0] field "netflow.art_client_network_time_sum" is undefined
[1] field "netflow.art_count_late_responses" is undefined
[2] field "netflow.art_count_responses" is undefined
[3] field "netflow.art_count_retransmissions" is undefined
[4] field "netflow.art_count_transactions" is undefined
[5] field "netflow.art_network_time_sum" is undefined
[6] field "netflow.art_response_time_sum" is undefined
[7] field "netflow.art_server_network_time_sum" is undefined
[8] field "netflow.art_server_response_time_maximum" is undefined
[9] field "netflow.art_server_response_time_sum" is undefined
[10] field "netflow.art_total_response_time_sum" is undefined
[11] field "netflow.art_total_transaction_time_sum" is undefined
[12] field "netflow.waasoptimization_segment" is undefined
[13] parsing field value failed: field "network.iana_number"'s Go type, float64, does not match the expected field type: keyword (field value: 6)

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.

Related issues

Screenshots

Run system tests for the package
--- Test results for package: netflow - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼──────────────┤
│ netflow │ log         │ system    │ netflow   │ PASS   │ 49.80578431s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴──────────────╯
--- Test results for package: netflow - END   ---
Done
Run asset tests for the package
--- Test results for package: netflow - START ---
╭─────────┬─────────────┬───────────┬──────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME                                                            │ RESULT │ TIME ELAPSED │
├─────────┼─────────────┼───────────┼──────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ netflow │             │ asset     │ dashboard netflow-14387a13-53bc-43a4-b9cd-63977aa8d87c is loaded     │ PASS   │ 8.567861395s │
│ netflow │             │ asset     │ dashboard netflow-34e26884-161a-4448-9556-43b5bf2f62a2 is loaded     │ PASS   │       2.45µs │
│ netflow │             │ asset     │ dashboard netflow-38012abe-c611-4124-8497-381fcd85acc8 is loaded     │ PASS   │        421ns │
│ netflow │             │ asset     │ dashboard netflow-77326664-23be-4bf1-a126-6d7e60cfc024 is loaded     │ PASS   │        412ns │
│ netflow │             │ asset     │ dashboard netflow-94972700-de4a-4272-9143-2fa8d4981365 is loaded     │ PASS   │        392ns │
│ netflow │             │ asset     │ dashboard netflow-acd7a630-0c71-4840-bc9e-4a3801374a32 is loaded     │ PASS   │        417ns │
│ netflow │             │ asset     │ dashboard netflow-c64665f9-d222-421e-90b0-c7310d944b8a is loaded     │ PASS   │        421ns │
│ netflow │             │ asset     │ dashboard netflow-feebb4e6-b13e-4e4e-b9fc-d3a178276425 is loaded     │ PASS   │        374ns │
│ netflow │             │ asset     │ visualization netflow-0177bf1a-cba8-4ba6-a1d7-73caed86ffc2 is loaded │ PASS   │        560ns │
│ netflow │             │ asset     │ visualization netflow-036aef95-ec90-468d-ad7c-3cc4405e9e81 is loaded │ PASS   │        385ns │
│ netflow │             │ asset     │ visualization netflow-0528bc66-6981-400a-a02d-c1d221b38890 is loaded │ PASS   │        409ns │
│ netflow │             │ asset     │ visualization netflow-0b2818fd-aecc-4bef-b566-9466eb702ae4 is loaded │ PASS   │        534ns │
│ netflow │             │ asset     │ visualization netflow-12aad647-c45d-4667-a029-152c1a97cbbc is loaded │ PASS   │        394ns │
│ netflow │             │ asset     │ visualization netflow-14c7136d-b4aa-4367-9461-52bf8b5c4796 is loaded │ PASS   │        394ns │
│ netflow │             │ asset     │ visualization netflow-15295ea6-ba84-47db-8ced-9312abbf495c is loaded │ PASS   │        412ns │
│ netflow │             │ asset     │ visualization netflow-1558508d-591c-49be-bef4-85fdac18a960 is loaded │ PASS   │        375ns │
│ netflow │             │ asset     │ visualization netflow-15e2a267-2495-4df2-a121-abe410d2f18c is loaded │ PASS   │        370ns │
│ netflow │             │ asset     │ visualization netflow-16262df9-a979-4136-935e-d883c7d373d7 is loaded │ PASS   │        370ns │
│ netflow │             │ asset     │ visualization netflow-1cd36f5d-d9c7-4098-acdb-14d312ecfb72 is loaded │ PASS   │        373ns │
│ netflow │             │ asset     │ visualization netflow-1cf30eac-aae8-47fa-a156-37f6346d2d5a is loaded │ PASS   │      2.484µs │
│ netflow │             │ asset     │ visualization netflow-1e74d5cb-556d-42ee-8042-88f6c1af47f0 is loaded │ PASS   │        464ns │
│ netflow │             │ asset     │ visualization netflow-201d7dd1-a880-4a64-b631-db5629340db9 is loaded │ PASS   │        398ns │
│ netflow │             │ asset     │ visualization netflow-2316bb53-d98a-4f0f-8cd8-51e9fb317823 is loaded │ PASS   │        425ns │
│ netflow │             │ asset     │ visualization netflow-248e00b4-8fc2-406f-8907-729d5380aaa7 is loaded │ PASS   │        364ns │
│ netflow │             │ asset     │ visualization netflow-2b3d4e86-2254-4033-8fe3-ce4753fafd03 is loaded │ PASS   │        365ns │
│ netflow │             │ asset     │ visualization netflow-2dca3025-692c-4876-8bcc-e0b248dc9819 is loaded │ PASS   │        391ns │
│ netflow │             │ asset     │ visualization netflow-30cd1009-2925-4c9b-820d-d689f5d1efda is loaded │ PASS   │        377ns │
│ netflow │             │ asset     │ visualization netflow-31708a70-4957-4a8a-8065-5c88a344ad02 is loaded │ PASS   │        399ns │
│ netflow │             │ asset     │ visualization netflow-31b5f6fd-eb9d-4e97-90fd-367062ef217f is loaded │ PASS   │        368ns │
│ netflow │             │ asset     │ visualization netflow-32e712ed-fa15-4db7-8575-8476e8d65b03 is loaded │ PASS   │        391ns │
│ netflow │             │ asset     │ visualization netflow-3a4209e2-281c-467e-b5cb-315bf4a2661f is loaded │ PASS   │        388ns │
│ netflow │             │ asset     │ visualization netflow-3dec20c0-0d4f-43ef-8864-3779e1a1b33f is loaded │ PASS   │        498ns │
│ netflow │             │ asset     │ visualization netflow-3e27fb83-b3e3-4c15-b999-ed6da49b7a86 is loaded │ PASS   │        394ns │
│ netflow │             │ asset     │ visualization netflow-441c6c50-fa1a-489c-96c6-76f7925dea24 is loaded │ PASS   │        373ns │
│ netflow │             │ asset     │ visualization netflow-4ac97841-c89f-4d50-b3c6-6253f7e1dd1a is loaded │ PASS   │        435ns │
│ netflow │             │ asset     │ visualization netflow-4bb0255e-18ed-45e4-bfb9-de8e35b12094 is loaded │ PASS   │        367ns │
│ netflow │             │ asset     │ visualization netflow-5292a65b-c532-422a-9008-1251a8073a3a is loaded │ PASS   │        363ns │
│ netflow │             │ asset     │ visualization netflow-5303e99b-389c-47b7-ae7a-945c5a92ba49 is loaded │ PASS   │        361ns │
│ netflow │             │ asset     │ visualization netflow-57e13a20-e94f-4465-a942-42148634a1d2 is loaded │ PASS   │        365ns │
│ netflow │             │ asset     │ visualization netflow-5ccac452-e90a-4dde-ae9b-1be36ce3f761 is loaded │ PASS   │        378ns │
│ netflow │             │ asset     │ visualization netflow-5cfb2c9a-4815-4a25-9d7e-ab0ef55ffe63 is loaded │ PASS   │        380ns │
│ netflow │             │ asset     │ visualization netflow-5d868836-c7b2-4812-bf47-4838aac281d9 is loaded │ PASS   │        369ns │
│ netflow │             │ asset     │ visualization netflow-63ef5338-fdf2-488e-b78a-f0e98daccc95 is loaded │ PASS   │        363ns │
│ netflow │             │ asset     │ visualization netflow-67fdca65-a9df-47f0-a8a4-1e8b056325de is loaded │ PASS   │        466ns │
│ netflow │             │ asset     │ visualization netflow-681f0ce4-d828-4a99-b643-0c0715530050 is loaded │ PASS   │        372ns │
│ netflow │             │ asset     │ visualization netflow-6bbd6712-494a-4fd9-b3d3-757304681f0f is loaded │ PASS   │        424ns │
│ netflow │             │ asset     │ visualization netflow-717cd7c7-bfca-435d-8ee7-38259927aade is loaded │ PASS   │        362ns │
│ netflow │             │ asset     │ visualization netflow-751ecb6f-11c3-458d-b039-f6d57a6379fa is loaded │ PASS   │        365ns │
│ netflow │             │ asset     │ visualization netflow-7d447b22-89dc-4f32-b549-4b8620af4d76 is loaded │ PASS   │        369ns │
│ netflow │             │ asset     │ visualization netflow-7fa6cb0a-518d-46e9-a228-15cd4253a957 is loaded │ PASS   │        385ns │
│ netflow │             │ asset     │ visualization netflow-85ebf558-402b-45d2-a186-e15f8673ec07 is loaded │ PASS   │        370ns │
│ netflow │             │ asset     │ visualization netflow-8f83cf97-4a48-421f-8db5-690297d1f4fb is loaded │ PASS   │        362ns │
│ netflow │             │ asset     │ visualization netflow-a14c3248-952d-42aa-bd7d-9b39157a776f is loaded │ PASS   │        397ns │
│ netflow │             │ asset     │ visualization netflow-a1704d46-15fc-41c2-851d-796ceb49877f is loaded │ PASS   │        444ns │
│ netflow │             │ asset     │ visualization netflow-a5efa3dd-f53a-4d14-9d3f-ee73345fd93d is loaded │ PASS   │        389ns │
│ netflow │             │ asset     │ visualization netflow-a685420e-c45f-4b62-932b-5b76ac8b8ca2 is loaded │ PASS   │        362ns │
│ netflow │             │ asset     │ visualization netflow-abfa0b19-60cd-4984-9c3d-02ebf0aa1dfb is loaded │ PASS   │        384ns │
│ netflow │             │ asset     │ visualization netflow-ae334aec-31fa-4df7-a064-40b18831d819 is loaded │ PASS   │        369ns │
│ netflow │             │ asset     │ visualization netflow-aed09724-0a69-4331-84f5-3d2067c43930 is loaded │ PASS   │        366ns │
│ netflow │             │ asset     │ visualization netflow-af707b01-29f1-462b-b279-6d2e803f3645 is loaded │ PASS   │        360ns │
│ netflow │             │ asset     │ visualization netflow-b02c2713-17f0-41dd-88a3-ce33b446f19d is loaded │ PASS   │        444ns │
│ netflow │             │ asset     │ visualization netflow-b677cd82-b33e-49b3-8b6e-0e110177b163 is loaded │ PASS   │        389ns │
│ netflow │             │ asset     │ visualization netflow-c27c6a3b-93ee-44d5-8d0c-9b097e575f52 is loaded │ PASS   │        380ns │
│ netflow │             │ asset     │ visualization netflow-c54f5529-e6d7-4c26-8e8e-3b35de132035 is loaded │ PASS   │        361ns │
│ netflow │             │ asset     │ visualization netflow-cccff92f-cb71-49a9-9caf-84867751d31e is loaded │ PASS   │        395ns │
│ netflow │             │ asset     │ visualization netflow-cf399a85-e348-4ac1-a399-e8f5a44114c4 is loaded │ PASS   │        366ns │
│ netflow │             │ asset     │ visualization netflow-d27b5d74-b3b4-4311-a0e6-08ff8f4345df is loaded │ PASS   │        376ns │
│ netflow │             │ asset     │ visualization netflow-d3df8d28-65f8-4ea1-8b33-f479380a0600 is loaded │ PASS   │        361ns │
│ netflow │             │ asset     │ visualization netflow-d41a9663-e5ad-47a7-955e-3803ae4e23c0 is loaded │ PASS   │        388ns │
│ netflow │             │ asset     │ visualization netflow-d4e6520a-9ced-47c9-a8f2-7246e8cbd2d3 is loaded │ PASS   │        369ns │
│ netflow │             │ asset     │ visualization netflow-d5568704-e30b-4108-bb49-06a9b8dce6a6 is loaded │ PASS   │      7.403µs │
│ netflow │             │ asset     │ visualization netflow-d59a031c-70d6-47d7-966d-7fcb805be9be is loaded │ PASS   │        492ns │
│ netflow │             │ asset     │ visualization netflow-ddd27657-c3c8-4f82-8059-6d7763dd599b is loaded │ PASS   │        376ns │
│ netflow │             │ asset     │ visualization netflow-e822f94c-5f65-4963-a540-74ca9c25bd2d is loaded │ PASS   │        378ns │
│ netflow │             │ asset     │ visualization netflow-e99dc327-03de-4561-9e0c-f550710125c2 is loaded │ PASS   │        372ns │
│ netflow │             │ asset     │ visualization netflow-e9ad835b-b2f2-42d3-a3e7-555a593deacf is loaded │ PASS   │        419ns │
│ netflow │             │ asset     │ visualization netflow-ebea013f-9b5b-4f61-a9c8-c62bebf62ae9 is loaded │ PASS   │        389ns │
│ netflow │             │ asset     │ visualization netflow-f27c1479-0625-4cdc-92de-672e47db0f87 is loaded │ PASS   │        369ns │
│ netflow │             │ asset     │ visualization netflow-f4c8cb5a-7336-449e-ab99-6e867b435b85 is loaded │ PASS   │        370ns │
│ netflow │             │ asset     │ visualization netflow-f531f957-e8c0-497a-ad41-ef39c2d29671 is loaded │ PASS   │        370ns │
│ netflow │             │ asset     │ visualization netflow-f668ecdb-eec7-44c6-9060-26aaf9fc8404 is loaded │ PASS   │        365ns │
│ netflow │             │ asset     │ visualization netflow-f75063c7-48b7-4de4-b8cb-d07eb2cea0e9 is loaded │ PASS   │        384ns │
│ netflow │             │ asset     │ visualization netflow-f772028b-d5a6-4d55-b441-493871981a60 is loaded │ PASS   │        390ns │
│ netflow │             │ asset     │ visualization netflow-f7808e70-df2a-4532-a350-966704567c24 is loaded │ PASS   │        389ns │
│ netflow │             │ asset     │ visualization netflow-f86a7769-8ef6-408d-bbe3-985d0ea0a3f7 is loaded │ PASS   │        365ns │
│ netflow │             │ asset     │ visualization netflow-fd6c1144-5026-4795-b7af-a9aa3fc28c56 is loaded │ PASS   │        362ns │
│ netflow │             │ asset     │ search netflow-a34c6611-79d8-4b50-ae3f-8b328d28e24a is loaded        │ PASS   │        513ns │
│ netflow │ log         │ asset     │ index_template logs-netflow.log is loaded                            │ PASS   │        496ns │
│ netflow │ log         │ asset     │ ingest_pipeline logs-netflow.log-0.3.6 is loaded                     │ PASS   │        419ns │
╰─────────┴─────────────┴───────────┴──────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: netflow - END   ---
Done
Run pipeline tests for the package
--- Test results for package: netflow - START ---
No test results
--- Test results for package: netflow - END   ---

Add a system test for the netflow input in the Netflow Log data stream. Initially the errors were: netflow/log netflow: [0] field "netflow.art_client_network_time_sum" is undefined [1] field "netflow.art_count_late_responses" is undefined [2] field "netflow.art_count_responses" is undefined [3] field "netflow.art_count_retransmissions" is undefined [4] field "netflow.art_count_transactions" is undefined [5] field "netflow.art_network_time_sum" is undefined [6] field "netflow.art_response_time_sum" is undefined [7] field "netflow.art_server_network_time_sum" is undefined [8] field "netflow.art_server_response_time_maximum" is undefined [9] field "netflow.art_server_response_time_sum" is undefined [10] field "netflow.art_total_response_time_sum" is undefined [11] field "netflow.art_total_transaction_time_sum" is undefined [12] field "netflow.waasoptimization_segment" is undefined [13] parsing field value failed: field "network.iana_number"'s Go type, float64, does not match the expected field type: keyword (field value: 6)

elasticmachine · 2021-01-31T14:49:20Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #590 updated
- Start Time: 2021-01-31T16:07:05.510+0000
Duration: 8 min 47 sec
Commit: 5f68f46

Test stats 🧪

Test	Results
Failed	0
Passed	90
Skipped	0
Total	90

elasticmachine · 2021-01-31T16:06:27Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh added the Team:Security-External Integrations label Jan 31, 2021

andrewkroh mentioned this pull request Jan 31, 2021

[Filebeat] Add field definitions for known Netflow/IPFIX vendor fields elastic/beats#23773

Merged

6 tasks

andrewkroh added 2 commits January 31, 2021 10:40

Add field definitions for netflow vendor fields

ec7b538

Add numeric_keyword_fields to test config for network.iana_number

5f68f46

andrewkroh marked this pull request as ready for review January 31, 2021 16:06

andrewkroh mentioned this pull request Jan 31, 2021

Add automated tests for security packages #377

Closed

adriansr approved these changes Feb 1, 2021

View reviewed changes

andrewkroh merged commit 4e4c7be into elastic:master Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add system test for Netflow Log (udp)#590

Add system test for Netflow Log (udp)#590
andrewkroh merged 3 commits intoelastic:masterfrom
andrewkroh:feature/netflow/system-tests

andrewkroh commented Jan 31, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Jan 31, 2021 •

edited

Loading

Build stats

Test stats 🧪

Uh oh!

elasticmachine commented Jan 31, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

andrewkroh commented Jan 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Related issues

Screenshots

Uh oh!

elasticmachine commented Jan 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Uh oh!

elasticmachine commented Jan 31, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andrewkroh commented Jan 31, 2021 •

edited

Loading

elasticmachine commented Jan 31, 2021 •

edited

Loading