Fix bug with annotations dedot config on k8s not used#25111

Merged

ChrsMark merged 6 commits intoelastic:masterfrom

ChrsMark:fix_k8s_annotations

Apr 19, 2021

Member

ChrsMark commented Apr 15, 2021

What does this PR do?

This PR fixes a bug with k8s annotation's dedot option cannot be disabled cause we ignore its value and we use label's dedot option.

Why is it important?

Annotation's dedot cannot be disabled.

Related issues

Closes [Filebeat] Annotations dedot is ignored #25078


          Fix bug with annotations dedot config on k8s not used

f51f4fa

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark added bug Team:Integrations v7.11.0 v7.13.0 v7.12.1 labels

ChrsMark requested a review from jsoriano

April 15, 2021 11:24

ChrsMark self-assigned this

Contributor

elasticmachine commented Apr 15, 2021

Pinging @elastic/integrations (Team:Integrations)

botelastic bot added needs_team and removed needs_team labels

ChrsMark added 2 commits

April 15, 2021 14:25


          Add changelog

e4f11a1

Signed-off-by: chrismark <chrismarkou92@gmail.com>


          Add test

7f87f1d

Signed-off-by: chrismark <chrismarkou92@gmail.com>

Contributor

elasticmachine commented Apr 15, 2021 •

edited by jenkins-beats-ci bot

Loading

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: ChrsMark commented: /test
Start Time: 2021-04-19T10:26:21.031+0000
Duration: 49 min 56 sec
Commit: 91e8f9b

Test stats 🧪

Test	Results
Failed	174
Passed	3628
Skipped	309
Total	4111

Trends 🧪

Test errors

Expand to view the tests failures

> Show only the first 10 test failures

`Build&Test / filebeat-build / test_fileset_file_000_traefik – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'traefik.access.user_identifier': '-', 'traefik.access.frontend_name': 'Host-host-1', 'traefik.access.backend_url': 'http://172.19.0.3:5601', 'traefik.access.request_count': 262, 'source.address': '192.168.33.1', 'source.ip': '192.168.33.1', 'fileset.name': 'access', 'url.original': '/ui/favicons/favicon-16x16.png', 'input.type': 'log', '@timestamp': '2017-10-02T20:22:07.000Z', 'related.ip': ['192.168.33.1'], 'service.type': 'traefik', 'http.request.referrer': 'http://example.com/login', 'http.request.method': 'GET', 'http.response.status_code': 304, 'http.response.body.bytes': 0, 'http.version': '1.1', 'event.duration': 2000000, 'event.kind': 'event', 'event.module': 'traefik', 'event.category': ['web'], 'event.type': ['access'], 'event.dataset': 'traefik.access', 'event.outcome': 'success', 'user.name': '-', 'user_agent.original': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36', 'user_agent.os.name': 'Linux', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '61.0.3163.100'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_000_traefik>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_000_traefik>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/traefik/access/test/test.log'
objects = [{'@timestamp': '2017-10-02T20:22:07.000Z', 'agent': {'ephemeral_id': '6da227f8-ed86-4467-9b36-ffcf5bd8ad3a', 'id': 'd...ry': ['web'], 'created': '2021-04-19T10:51:10.043Z', 'dataset': 'traefik.access', 'duration': 8000000, ...}, ...}, ...]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'traefik.access.user_identifier': '-', 'traefik.access.frontend_name': 'Host-host-1', 'traefik.access.backend_url': 'http://172.19.0.3:5601', 'traefik.access.request_count': 262, 'source.address': '192.168.33.1', 'source.ip': '192.168.33.1', 'fileset.name': 'access', 'url.original': '/ui/favicons/favicon-16x16.png', 'input.type': 'log', '@timestamp': '2017-10-02T20:22:07.000Z', 'related.ip': ['192.168.33.1'], 'service.type': 'traefik', 'http.request.referrer': 'http://example.com/login', 'http.request.method': 'GET', 'http.response.status_code': 304, 'http.response.body.bytes': 0, 'http.version': '1.1', 'event.duration': 2000000, 'event.kind': 'event', 'event.module': 'traefik', 'event.category': ['web'], 'event.type': ['access'], 'event.dataset': 'traefik.access', 'event.outcome': 'success', 'user.name': '-', 'user_agent.original': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36', 'user_agent.os.name': 'Linux', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '61.0.3163.100'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_002_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 244, 'destination.address': '10.100.220.70', 'destination.port': 80, 'destination.ip': '10.100.220.70', 'source.address': '10.100.118.31', 'source.ip': '10.100.118.31', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.sub_status': 4, 'iis.access.win32_status': 2, '@timestamp': '2018-08-28T18:24:25.000Z', 'related.ip': ['10.100.118.31', '10.100.220.70'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 792000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR[ 2.0.50727](tel: 2050727); .NET CLR 3.0.30729)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '8.1', 'user_agent.os.full': 'Windows 8.1', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '11.0'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_002_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_002_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-iis-7.5.log'
objects = [{'@timestamp': '2018-08-28T18:24:25.000Z', 'agent': {'ephemeral_id': '472a8659-3eff-47c7-baef-afd42ffb184f', 'id': 'd...:0000:0000:3257:9652', 'ip': '2001:cdba:0000:0000:0000:0000:3257:9652', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 244, 'destination.address': '10.100.220.70', 'destination.port': 80, 'destination.ip': '10.100.220.70', 'source.address': '10.100.118.31', 'source.ip': '10.100.118.31', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.sub_status': 4, 'iis.access.win32_status': 2, '@timestamp': '2018-08-28T18:24:25.000Z', 'related.ip': ['10.100.118.31', '10.100.220.70'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 792000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.3; WOW64; Trident/7.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR[ 2.0.50727](tel: 2050727); .NET CLR 3.0.30729)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '8.1', 'user_agent.os.full': 'Windows 8.1', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '11.0'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_003_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'destination.address': '10.24.129.162', 'destination.port': 80, 'destination.domain': 'images.hogeschoolrotterdam.nl', 'destination.ip': '10.24.129.162', 'source.address': '10.24.136.240', 'source.ip': '10.24.136.240', 'fileset.name': 'access', 'url.path': '/favicon.ico', 'network.forwarded_ip': '116.189.86.89', 'input.type': 'log', 'iis.access.site_name': 'W3SVC2', 'iis.access.server_name': 'freca1', 'iis.access.sub_status': 0, 'iis.access.win32_status': 2, '@timestamp': '2020-10-04T22:00:34.000Z', 'related.ip': ['10.24.136.240', '10.24.129.162'], 'service.type': 'iis', 'http.request.referrer': 'https://images.hogeschoolrotterdam.nl/Blob/adeec119008c48758c1a6be53aeeb2ac/34ff475072d54117bcb46ea7f023bd87.jpg?width=1200&height=630&mode=crop', 'http.request.method': 'GET', 'http.request.body.bytes': 534, 'http.response.status_code': 404, 'http.response.body.bytes': 1437, 'http.version': '1.1', 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '81.0.'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_003_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_003_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-x-forward-for-extended.log'
objects = [{'@timestamp': '2020-10-04T22:00:34.000Z', 'agent': {'ephemeral_id': '021e0a4e-a81d-493a-9cd5-478d7df9099a', 'id': 'd...162', 'domain': 'images.hogeschoolrotterdam.nl', 'ip': '10.24.129.162', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'destination.address': '10.24.129.162', 'destination.port': 80, 'destination.domain': 'images.hogeschoolrotterdam.nl', 'destination.ip': '10.24.129.162', 'source.address': '10.24.136.240', 'source.ip': '10.24.136.240', 'fileset.name': 'access', 'url.path': '/favicon.ico', 'network.forwarded_ip': '116.189.86.89', 'input.type': 'log', 'iis.access.site_name': 'W3SVC2', 'iis.access.server_name': 'freca1', 'iis.access.sub_status': 0, 'iis.access.win32_status': 2, '@timestamp': '2020-10-04T22:00:34.000Z', 'related.ip': ['10.24.136.240', '10.24.129.162'], 'service.type': 'iis', 'http.request.referrer': 'https://images.hogeschoolrotterdam.nl/Blob/adeec119008c48758c1a6be53aeeb2ac/34ff475072d54117bcb46ea7f023bd87.jpg?width=1200&height=630&mode=crop', 'http.request.method': 'GET', 'http.request.body.bytes': 534, 'http.response.status_code': 404, 'http.response.body.bytes': 1437, 'http.version': '1.1', 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '81.0.'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_004_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 331, 'destination.address': '::1%0', 'destination.port': 80, 'destination.domain': 'example.com', 'destination.ip': '::1', 'source.address': '::1%0', 'source.ip': '::1', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.site_name': 'W3SVC1', 'iis.access.server_name': 'MACHINE-NAME', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T10:11:12.000Z', 'related.ip': ['::1', '::1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.request.body.bytes': 456, 'http.response.status_code': 200, 'http.response.body.bytes': 123, 'http.version': '1.1', 'event.duration': 789000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36', 'user_agent.os.name': 'Mac OS X', 'user_agent.os.version': '10.14.0', 'user_agent.os.full': 'Mac OS X 10.14.0', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Mac', 'user_agent.version': '70.0.3538.102'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_004_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_004_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-ipv6zone.log'
objects = [{'@timestamp': '2018-01-01T10:11:12.000Z', 'agent': {'ephemeral_id': '91b9e04d-c05a-4f73-8083-71933011e005', 'id': 'd...estination': {'address': '::1%0', 'domain': 'example.com', 'ip': '::1', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 331, 'destination.address': '::1%0', 'destination.port': 80, 'destination.domain': 'example.com', 'destination.ip': '::1', 'source.address': '::1%0', 'source.ip': '::1', 'fileset.name': 'access', 'url.path': '/', 'input.type': 'log', 'iis.access.site_name': 'W3SVC1', 'iis.access.server_name': 'MACHINE-NAME', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T10:11:12.000Z', 'related.ip': ['::1', '::1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.request.body.bytes': 456, 'http.response.status_code': 200, 'http.response.body.bytes': 123, 'http.version': '1.1', 'event.duration': 789000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36', 'user_agent.os.name': 'Mac OS X', 'user_agent.os.version': '10.14.0', 'user_agent.os.full': 'Mac OS X 10.14.0', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Mac', 'user_agent.version': '70.0.3538.102'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_005_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'destination.address': '192.168.16.11', 'destination.port': 443, 'destination.ip': '192.168.16.11', 'source.address': '192.168.7.63', 'source.ip': '192.168.7.63', 'fileset.name': 'access', 'url.path': '/Production-UI/api/finance/legacy/GeneralLedger/LoadBatchTotals', 'network.forwarded_ip': '192.168.198.23', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2020-10-07T23:00:17.000Z', 'related.ip': ['192.168.7.63', '192.168.16.11'], 'service.type': 'iis', 'http.request.referrer': 'https://onesolfarm.ggcity.org/Production-UI/ui/uiscreens/generalledger/GLJEUB', 'http.request.method': 'POST', 'http.response.status_code': 200, 'event.duration': 26000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '85.0.4183.121'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_005_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_005_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-x-forward-for.log'
objects = [{'@timestamp': '2020-10-07T23:00:17.000Z', 'agent': {'ephemeral_id': 'ed356231-e4a4-46eb-ae5a-97c48ebcf66b', 'id': 'd...destination': {'address': '192.168.16.11', 'ip': '192.168.16.11', 'port': 443}, 'ecs': {'version': '1.9.0'}, ...}, ...]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'destination.address': '192.168.16.11', 'destination.port': 443, 'destination.ip': '192.168.16.11', 'source.address': '192.168.7.63', 'source.ip': '192.168.7.63', 'fileset.name': 'access', 'url.path': '/Production-UI/api/finance/legacy/GeneralLedger/LoadBatchTotals', 'network.forwarded_ip': '192.168.198.23', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2020-10-07T23:00:17.000Z', 'related.ip': ['192.168.7.63', '192.168.16.11'], 'service.type': 'iis', 'http.request.referrer': 'https://onesolfarm.ggcity.org/Production-UI/ui/uiscreens/generalledger/GLJEUB', 'http.request.method': 'POST', 'http.response.status_code': 200, 'event.duration': 26000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '10', 'user_agent.os.full': 'Windows 10', 'user_agent.name': 'Chrome', 'user_agent.device.name': 'Other', 'user_agent.version': '85.0.4183.121'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_006_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['user_agent.device.type']]}, full object:    {'log.offset': 0, 'destination.address': '10.44.0.136', 'destination.port': 8080, 'destination.ip': '10.44.0.136', 'source.address': '10.50.6.188', 'source.ip': '10.50.6.188', 'fileset.name': 'access', 'url.path': '/pbserver/..À¯..À¯..À¯..À¯..À¯../winnt/system32/cmd.exe', 'url.query': '/c+dir+c:\\+/OG', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 64, '@timestamp': '2018-12-31T12:02:53.000Z', 'related.ip': ['10.50.6.188', '10.44.0.136'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': 'XP', 'user_agent.os.full': 'Windows XP', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '8.0'} assert 1 == 0  +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_006_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_006_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test-iis-7.2.log'
objects = [{'@timestamp': '2018-12-31T12:02:53.000Z', 'agent': {'ephemeral_id': '37e1b540-641b-43c0-b325-3e2f12447de5', 'id': 'd... ...}, 'destination': {'address': '10.44.0.136', 'ip': '10.44.0.136', 'port': 8080}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['user_agent.device.type']]}, full object: 
E             {'log.offset': 0, 'destination.address': '10.44.0.136', 'destination.port': 8080, 'destination.ip': '10.44.0.136', 'source.address': '10.50.6.188', 'source.ip': '10.50.6.188', 'fileset.name': 'access', 'url.path': '/pbserver/..À¯..À¯..À¯..À¯..À¯../winnt/system32/cmd.exe', 'url.query': '/c+dir+c:\\+/OG', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 64, '@timestamp': '2018-12-31T12:02:53.000Z', 'related.ip': ['10.50.6.188', '10.44.0.136'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 404, 'event.duration': 0, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'failure', 'user_agent.original': 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)', 'user_agent.os.name': 'Windows', 'user_agent.os.version': 'XP', 'user_agent.os.full': 'Windows XP', 'user_agent.name': 'IE', 'user_agent.device.name': 'Other', 'user_agent.version': '8.0'}
E           assert 1 == 0
E            +  where 1 = len({'dictionary_item_removed': [root['user_agent.device.type']]})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_007_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name'], root['user_agent.device.type']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object:    {'log.offset': 257, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.ip': '85.181.35.98', 'fileset.name': 'access', 'url.path': '/', 'url.query': 'q=100', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T08:09:10.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 200, 'event.duration': 123000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '7', 'user_agent.os.full': 'Windows 7', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '57.0.'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_007_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_007_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/access/test/test.log'
objects = [{'@timestamp': '2018-01-01T08:09:10.000Z', 'agent': {'ephemeral_id': '7e34cf99-0ea3-4a70-a4d3-31c170cac03f', 'id': 'd..., ...}, 'destination': {'address': '10.44.0.136', 'ip': '10.44.0.136', 'port': 443}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name'], root['user_agent.device.type']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object: 
E             {'log.offset': 257, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.ip': '85.181.35.98', 'fileset.name': 'access', 'url.path': '/', 'url.query': 'q=100', 'input.type': 'log', 'iis.access.sub_status': 0, 'iis.access.win32_status': 0, '@timestamp': '2018-01-01T08:09:10.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 200, 'event.duration': 123000000, 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.access', 'event.outcome': 'success', 'user_agent.original': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0', 'user_agent.os.name': 'Windows', 'user_agent.os.version': '7', 'user_agent.os.full': 'Windows 7', 'user_agent.name': 'Firefox', 'user_agent.device.name': 'Other', 'user_agent.version': '57.0.'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_010_iis – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object:    {'log.offset': 286, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.port': 2780, 'source.ip': '85.181.35.98', 'fileset.name': 'error', 'url.original': '/ThisIsMyUrl.htm', 'input.type': 'log', 'iis.error.reason_phrase': 'Hostname', '@timestamp': '2018-01-01T09:10:11.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 400, 'http.version': '1.1', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.error', 'event.outcome': 'failure'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_010_iis>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_010_iis>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/iis/error/test/test.log'
objects = [{'@timestamp': '2018-01-01T08:09:10.000Z', 'agent': {'ephemeral_id': 'ff929a9d-b1f3-4da7-ae3f-5c9356f8c67d', 'id': 'd...beat', ...}, 'destination': {'address': '127.0.0.1', 'ip': '127.0.0.1', 'port': 80}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.location.lat']": {'new_value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}}, full object: 
E             {'log.offset': 286, 'destination.address': '127.0.0.1', 'destination.port': 80, 'destination.ip': '127.0.0.1', 'source.geo.continent_name': 'Europe', 'source.geo.country_iso_code': 'DE', 'source.geo.country_name': 'Germany', 'source.geo.location.lon': 9.491, 'source.geo.location.lat': 51.2993, 'source.as.number': 6805, 'source.as.organization.name': 'Telefonica Germany', 'source.address': '85.181.35.98', 'source.port': 2780, 'source.ip': '85.181.35.98', 'fileset.name': 'error', 'url.original': '/ThisIsMyUrl.htm', 'input.type': 'log', 'iis.error.reason_phrase': 'Hostname', '@timestamp': '2018-01-01T09:10:11.000Z', 'related.ip': ['85.181.35.98', '127.0.0.1'], 'service.type': 'iis', 'http.request.method': 'GET', 'http.response.status_code': 400, 'http.version': '1.1', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.error', 'event.outcome': 'failure'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...value': 51.2993, 'old_value': 52.4473}, "root['source.geo.location.lon']": {'new_value': 9.491, 'old_value': 13.4531}}})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_013_haproxy – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}}}, full object:    {'process.name': 'haproxy', 'process.pid': 32450, 'log.offset': 0, 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 38862, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4'], 'service.type': 'haproxy', 'haproxy.server_name': 'docs', 'haproxy.backend_queue': 0, 'haproxy.total_waiting_time_ms': 0, 'haproxy.termination_state': '----', 'haproxy.connection_wait_time_ms': 1, 'haproxy.backend_name': 'docs_microservice', 'haproxy.http.request.raw_request_line': 'GET /component---src-pages-index-js-4b15624544f97cf0bb8f.js HTTP/1.1', 'haproxy.http.request.captured_cookie': '-', 'haproxy.http.request.captured_headers': ['docs.example.internal'], 'haproxy.http.request.time_wait_ms': 0, 'haproxy.http.request.time_wait_without_data_ms': 0, 'haproxy.http.response.captured_cookie': '-', 'haproxy.http.response.captured_headers': [], 'haproxy.frontend_name': 'incoming~', 'haproxy.server_queue': 0, 'haproxy.bytes_read': 168, 'haproxy.connections.server': 0, 'haproxy.connections.retries': 0, 'haproxy.connections.active': 6, 'haproxy.connections.backend': 0, 'haproxy.connections.frontend': 6, 'http.response.status_code': 304, 'http.response.bytes': 168, 'event.duration': 2000000, 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web'], 'event.dataset': 'haproxy.log', 'event.outcome': 'success'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...: 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}}})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_013_haproxy>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_013_haproxy>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/haproxy/log/test/haproxy.log'
objects = [{'@timestamp': '2018-07-30T09:03:52.726-02:00', 'agent': {'ephemeral_id': 'ce2d9a3b-6b83-48ad-9b9b-19b42e565e6d', 'id...ory': ['web'], 'dataset': 'haproxy.log', 'duration': 2000000, 'ingested': '2021-04-19T10:51:34.978789831Z', ...}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}}}, full object: 
E             {'process.name': 'haproxy', 'process.pid': 32450, 'log.offset': 0, 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 38862, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4'], 'service.type': 'haproxy', 'haproxy.server_name': 'docs', 'haproxy.backend_queue': 0, 'haproxy.total_waiting_time_ms': 0, 'haproxy.termination_state': '----', 'haproxy.connection_wait_time_ms': 1, 'haproxy.backend_name': 'docs_microservice', 'haproxy.http.request.raw_request_line': 'GET /component---src-pages-index-js-4b15624544f97cf0bb8f.js HTTP/1.1', 'haproxy.http.request.captured_cookie': '-', 'haproxy.http.request.captured_headers': ['docs.example.internal'], 'haproxy.http.request.time_wait_ms': 0, 'haproxy.http.request.time_wait_without_data_ms': 0, 'haproxy.http.response.captured_cookie': '-', 'haproxy.http.response.captured_headers': [], 'haproxy.frontend_name': 'incoming~', 'haproxy.server_queue': 0, 'haproxy.bytes_read': 168, 'haproxy.connections.server': 0, 'haproxy.connections.retries': 0, 'haproxy.connections.active': 6, 'haproxy.connections.backend': 0, 'haproxy.connections.frontend': 6, 'http.response.status_code': 304, 'http.response.bytes': 168, 'event.duration': 2000000, 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web'], 'event.dataset': 'haproxy.log', 'event.outcome': 'success'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...: 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}}})

tests/system/test_modules.py:217: AssertionError

`Build&Test / filebeat-build / test_fileset_file_014_haproxy – filebeat.tests.system.test_modules.Test`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}}}, full object:    {'process.name': 'haproxy', 'process.pid': 24551, 'log.offset': 0, 'destination.port': 5000, 'destination.ip': '1.2.3.4', 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 40780, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4', '1.2.3.4'], 'service.type': 'haproxy', 'haproxy.mode': 'HTTP', 'haproxy.frontend_name': 'main', 'haproxy.source': '1.2.3.4', 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'haproxy.log'} assert 2 == 0  +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...: 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}}})

Expand to view the stacktrace

 a = (<test_modules.Test testMethod=test_fileset_file_014_haproxy>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_014_haproxy>
test_file = '/go/src/github.com/elastic/beats/filebeat/module/haproxy/log/test/default.log'
objects = [{'@timestamp': '2021-09-20T15:42:59.000-02:00', 'agent': {'ephemeral_id': '88591338-7fd0-4d84-9297-6ad9339e633b', 'id...90fd33f7', 'type': 'filebeat', ...}, 'destination': {'ip': '1.2.3.4', 'port': 5000}, 'ecs': {'version': '1.9.0'}, ...}]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region_name']], 'values_changed': {"root['source.geo.country_name']": {'new_value': 'Australia', 'old_value': 'Russia'}, "root['source.geo.location.lat']": {'new_value': -33.494, 'old_value': 55.7527}, "root['source.geo.country_iso_code']": {'new_value': 'AU', 'old_value': 'RU'}, "root['source.geo.continent_name']": {'new_value': 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}}}, full object: 
E             {'process.name': 'haproxy', 'process.pid': 24551, 'log.offset': 0, 'destination.port': 5000, 'destination.ip': '1.2.3.4', 'source.geo.continent_name': 'Oceania', 'source.geo.country_iso_code': 'AU', 'source.geo.country_name': 'Australia', 'source.geo.location.lon': 143.2104, 'source.geo.location.lat': -33.494, 'source.address': '1.2.3.4', 'source.port': 40780, 'source.ip': '1.2.3.4', 'fileset.name': 'log', 'input.type': 'log', 'related.ip': ['1.2.3.4', '1.2.3.4'], 'service.type': 'haproxy', 'haproxy.mode': 'HTTP', 'haproxy.frontend_name': 'main', 'haproxy.source': '1.2.3.4', 'event.timezone': '-02:00', 'event.kind': 'event', 'event.module': 'haproxy', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'haproxy.log'}
E           assert 2 == 0
E            +  where 2 = len({'dictionary_item_removed': [root['source.geo.city_name'], root['source.geo.region_iso_code'], root['source.geo.region...: 'Oceania', 'old_value': 'Europe'}, "root['source.geo.location.lon']": {'new_value': 143.2104, 'old_value': 37.6172}}})

tests/system/test_modules.py:217: AssertionError

Steps errors

Expand to view the steps failures

`filebeat-build - mage build test`

Took 20 min 35 sec . View more details on here
Description: mage build test

`x-pack/filebeat-build - mage build test`

Took 32 min 23 sec . View more details on here
Description: mage build test

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-04-19T11:12:57.374Z]   Git commit:       46229ca
[2021-04-19T11:12:57.374Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-19T11:12:57.374Z]   OS/Arch:          linux/amd64
[2021-04-19T11:12:57.374Z]   Experimental:     false
[2021-04-19T11:12:57.374Z]  containerd:
[2021-04-19T11:12:57.374Z]   Version:          1.4.4
[2021-04-19T11:12:57.374Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-19T11:12:57.374Z]  runc:
[2021-04-19T11:12:57.374Z]   Version:          1.0.0-rc93
[2021-04-19T11:12:57.374Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-19T11:12:57.374Z]  docker-init:
[2021-04-19T11:12:57.374Z]   Version:          0.19.0
[2021-04-19T11:12:57.374Z]   GitCommit:        de40ad0
[2021-04-19T11:12:57.374Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-19T11:12:57.374Z] Unable to find image 'alpine:3.4' locally
[2021-04-19T11:12:57.945Z] 3.4: Pulling from library/alpine
[2021-04-19T11:12:58.207Z] c1e54eec4b57: Pulling fs layer
[2021-04-19T11:12:58.469Z] c1e54eec4b57: Download complete
[2021-04-19T11:12:58.733Z] c1e54eec4b57: Pull complete
[2021-04-19T11:12:58.733Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-04-19T11:12:58.734Z] Status: Downloaded newer image for alpine:3.4
[2021-04-19T11:13:00.649Z] Change permissions with write access of all files inside the specific folder
[2021-04-19T11:13:02.057Z] Running in /var/lib/jenkins/workspace/PR-25111-6-881e2353-b18f-4246-a5d8-45fa32f62173/src/github.com/elastic/beats/build
[2021-04-19T11:13:02.390Z] + rm -rf ve
[2021-04-19T11:13:02.390Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-04-19T11:13:02.725Z] + python .ci/scripts/pre_archive_test.py
[2021-04-19T11:13:05.272Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-04-19T11:13:05.286Z] Running in /var/lib/jenkins/workspace/PR-25111-6-881e2353-b18f-4246-a5d8-45fa32f62173/src/github.com/elastic/beats/build
[2021-04-19T11:13:05.303Z] Recording test results
[2021-04-19T11:13:08.261Z] [Checks API] No suitable checks publisher found.
[2021-04-19T11:13:08.579Z] + tar --version
[2021-04-19T11:13:08.908Z] + tar --exclude=test-build-artifacts-x-pack-filebeat-build-tgz -czf test-build-artifacts-x-pack-filebeat-build-tgz .
[2021-04-19T11:14:05.729Z] + gsutil --version
[2021-04-19T11:14:05.729Z] /var/lib/jenkins/workspace/PR-25111-6-881e2353-b18f-4246-a5d8-45fa32f62173/src/github.com/elastic/beats/build@tmp/durable-f2290544/script.sh: 1: /var/lib/jenkins/workspace/PR-25111-6-881e2353-b18f-4246-a5d8-45fa32f62173/src/github.com/elastic/beats/build@tmp/durable-f2290544/script.sh: gsutil: not found
[2021-04-19T11:14:05.799Z] Running in /var/lib/jenkins/workspace/PR-25111-6-881e2353-b18f-4246-a5d8-45fa32f62173/src/github.com/elastic/beats/build@tmp
[2021-04-19T11:14:06.116Z] + wget --version
[2021-04-19T11:14:06.441Z] + wget -q -O gsutil.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-319.0.0-linux-x86_64.tar.gz
[2021-04-19T11:14:07.033Z] + tar -xpf gsutil.tar.gz --strip-components=1
[2021-04-19T11:14:09.659Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-19T11:14:10.047Z] + gcloud auth activate-service-account --key-file ****
[2021-04-19T11:14:11.431Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-19T11:14:12.020Z] + gsutil -m -q cp -a public-read test-build-artifacts-x-pack-filebeat-build-tgz gs://beats-ci-temp/Beats/beats/PR-25111-6
[2021-04-19T11:14:17.689Z] + python .ci/scripts/search_system_tests.py
[2021-04-19T11:14:17.711Z] [INFO] system-tests='build/x-pack/filebeat/build/system-tests'. If no empty then let's create a tarball
[2021-04-19T11:14:18.069Z] + tar --version
[2021-04-19T11:14:18.392Z] + tar --exclude=x-pack-filebeat--system-tests-linux-tgz -czf x-pack-filebeat--system-tests-linux-tgz build/x-pack/filebeat/build/system-tests
[2021-04-19T11:14:50.868Z] + gsutil --version
[2021-04-19T11:14:50.868Z] /var/lib/jenkins/workspace/PR-25111-6-881e2353-b18f-4246-a5d8-45fa32f62173/src/github.com/elastic/beats@tmp/durable-7f68e656/script.sh: 1: /var/lib/jenkins/workspace/PR-25111-6-881e2353-b18f-4246-a5d8-45fa32f62173/src/github.com/elastic/beats@tmp/durable-7f68e656/script.sh: gsutil: not found
[2021-04-19T11:14:50.910Z] Running in /var/lib/jenkins/workspace/PR-25111-6-881e2353-b18f-4246-a5d8-45fa32f62173/src/github.com/elastic/beats@tmp
[2021-04-19T11:14:51.227Z] + wget --version
[2021-04-19T11:14:51.551Z] + wget -q -O gsutil.tar.gz https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-319.0.0-linux-x86_64.tar.gz
[2021-04-19T11:14:52.136Z] + tar -xpf gsutil.tar.gz --strip-components=1
[2021-04-19T11:14:54.755Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-19T11:14:55.076Z] + gcloud auth activate-service-account --key-file ****
[2021-04-19T11:14:56.463Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-19T11:14:56.788Z] + gsutil -m -q cp -a public-read x-pack-filebeat--system-tests-linux-tgz gs://beats-ci-temp/Beats/beats/PR-25111-6
[2021-04-19T11:15:00.480Z] + go clean -modcache
[2021-04-19T11:15:04.097Z] Cleaning up /var/lib/jenkins/workspace/PR-25111-6-881e2353-b18f-4246-a5d8-45fa32f62173
[2021-04-19T11:15:04.097Z] Client: Docker Engine - Community
[2021-04-19T11:15:04.097Z]  Version:           20.10.3
[2021-04-19T11:15:04.097Z]  API version:       1.41
[2021-04-19T11:15:04.097Z]  Go version:        go1.13.15
[2021-04-19T11:15:04.097Z]  Git commit:        48d30b5
[2021-04-19T11:15:04.097Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-04-19T11:15:04.097Z]  OS/Arch:           linux/amd64
[2021-04-19T11:15:04.097Z]  Context:           default
[2021-04-19T11:15:04.097Z]  Experimental:      true
[2021-04-19T11:15:04.097Z] 
[2021-04-19T11:15:04.097Z] Server: Docker Engine - Community
[2021-04-19T11:15:04.097Z]  Engine:
[2021-04-19T11:15:04.097Z]   Version:          20.10.3
[2021-04-19T11:15:04.097Z]   API version:      1.41 (minimum version 1.12)
[2021-04-19T11:15:04.097Z]   Go version:       go1.13.15
[2021-04-19T11:15:04.097Z]   Git commit:       46229ca
[2021-04-19T11:15:04.097Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-19T11:15:04.097Z]   OS/Arch:          linux/amd64
[2021-04-19T11:15:04.097Z]   Experimental:     false
[2021-04-19T11:15:04.097Z]  containerd:
[2021-04-19T11:15:04.097Z]   Version:          1.4.4
[2021-04-19T11:15:04.097Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-19T11:15:04.097Z]  runc:
[2021-04-19T11:15:04.097Z]   Version:          1.0.0-rc93
[2021-04-19T11:15:04.097Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-19T11:15:04.097Z]  docker-init:
[2021-04-19T11:15:04.098Z]   Version:          0.19.0
[2021-04-19T11:15:04.098Z]   GitCommit:        de40ad0
[2021-04-19T11:15:04.098Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-19T11:15:08.324Z] Change permissions with write access of all files inside the specific folder
[2021-04-19T11:15:08.645Z] Running in /var/lib/jenkins/workspace/PR-25111-6-881e2353-b18f-4246-a5d8-45fa32f62173
[2021-04-19T11:15:14.950Z] Failed in branch x-pack/filebeat-build
[2021-04-19T11:15:15.064Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-19T11:15:15.140Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25111/src/github.com/elastic/beats
[2021-04-19T11:15:15.882Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25111
[2021-04-19T11:15:15.939Z] [INFO] getVaultSecret: Getting secrets
[2021-04-19T11:15:16.067Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-19T11:15:16.858Z] + chmod 755 generate-build-data.sh
[2021-04-19T11:15:16.858Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25111/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25111/runs/6 FAILURE 2935562
[2021-04-19T11:15:17.109Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25111/runs/6/steps/?limit=10000 -o steps-info.json
[2021-04-19T11:15:18.458Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25111/runs/6/tests/?status=FAILED -o tests-errors.json
[2021-04-19T11:15:20.338Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25111/runs/6/log/ -o pipeline-log.txt

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	174
Passed	3628
Skipped	309
Total	4111

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / filebeat-build / test_fileset_file_000_traefik – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_002_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_003_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_004_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_005_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_006_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_007_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_010_iis – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_013_haproxy – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_014_haproxy – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_065_auditd – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_068_auditd – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_071_nginx – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_072_nginx – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_073_nginx – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_074_nginx – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_102_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_103_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_104_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_105_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_106_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_109_apache – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_133_system – filebeat.tests.system.test_modules.Test
Name: Build&Test / filebeat-build / test_fileset_file_136_system – filebeat.tests.system.test_modules.Test
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_001_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_002_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_003_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_004_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_005_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_006_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_007_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_008_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_009_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_010_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_011_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_012_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_013_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_014_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_015_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_016_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_017_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_018_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_019_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_020_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_021_gsuite – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_022_zscaler – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_023_zscaler – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_024_azure – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_027_azure – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_032_fortinet – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_036_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_037_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_038_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_039_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_040_juniper – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_046_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_053_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_059_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_060_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_061_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_064_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_065_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_068_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_069_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_077_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_079_zeek – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_082_checkpoint – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_083_sonicwall – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_093_gcp – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_095_gcp – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_101_f5 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_113_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_114_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_115_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_116_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_117_panw – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_118_squid – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_119_squid – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_122_tomcat – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_123_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_124_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_128_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_131_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_132_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_135_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_136_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_138_o365 – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_140_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_141_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_142_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_143_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_144_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_145_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_146_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_147_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_148_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_149_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_150_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_151_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest
Name: Build&Test / x-pack/filebeat-build / test_fileset_file_152_google_workspace – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest


          fix test

ab34710

Signed-off-by: chrismark <chrismarkou92@gmail.com>

jsoriano approved these changes

View reviewed changes

libbeat/common/kubernetes/metadata/pod_test.go

               	config, err := common.NewConfigFrom(map[string]interface{}{
-              		"include_annotations": []string{"app"},
+              		"include_annotations": []string{"app", "k8s.app"},
+              		"annotations.dedot":   false,

Member

jsoriano Apr 15, 2021

Do we have some other test case that would cover the default annotations.dedot: true?

Member Author

ChrsMark Apr 15, 2021 •

edited

Loading

Unfortunately we had no test cases for these options. I can add one before merging (CI is red anyways)

Contributor

mergify bot commented Apr 16, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix_k8s_annotations upstream/fix_k8s_annotations
git merge upstream/master
git push upstream fix_k8s_annotations

ChrsMark added 2 commits

April 19, 2021 10:27


          Merge remote-tracking branch 'upstream/master' into fix_k8s_annotations

c602943


          test a dedot case too

91e8f9b

Signed-off-by: chrismark <chrismarkou92@gmail.com>

Member Author

ChrsMark commented Apr 19, 2021

/test

ChrsMark merged commit 7561201 into elastic:master

ChrsMark mentioned this pull request

Cherry-pick #25111 to 7.x: Fix bug with annotations dedot config on k8s not used #25139

Merged

ChrsMark added a commit to ChrsMark/beats that referenced this pull request


          Fix bug with annotations dedot config on k8s not used (elastic#25111)

e860e87

(cherry picked from commit 7561201)

ChrsMark added a commit that referenced this pull request


          Cherry-pick #25111 to 7.x: Fix bug with annotations dedot config on k…

75e33d9

…8s not used (#25139)

ChrsMark mentioned this pull request

Cherry-pick #25111 to 7.12: Fix bug with annotations dedot config on k8s not used #25162

Merged

ChrsMark added a commit to ChrsMark/beats that referenced this pull request


          Fix bug with annotations dedot config on k8s not used (elastic#25111)

fdec0ac

(cherry picked from commit 7561201)

v1v added a commit to v1v/beats that referenced this pull request


          Merge remote-tracking branch 'upstream/master' into feature/customise…

2d9fb4d

…-github-pr-comment-template

* upstream/master:
  [Ingest Manager] Keep http and logging config during enroll (elastic#25132)
  Refactor kubernetes autodiscover to avoid skipping short-living pods (elastic#24742)
  [libbeat] New decode xml wineventlog processor (elastic#25115)
  Add svc to agent k8s clusterRole (elastic#25146)
  Add awsfargate module to collect container logs from Amazon ECS on Fargate (elastic#25041)
  [Filebeat][Cisco ASA] log enhancement and performance (elastic#24744)
  Watch kubernetes namespaces for autodiscover metadata for pods (elastic#25117)
  Cyberark Privileged Access Security module (elastic#24803)
  [Elastic Agent] Log the container command output with LOGS_PATH (elastic#25150)
  Fix for tests after `device...` field has been removed (elastic#25141)
  [Ingest Manager] Restart process on output change (elastic#24907)
  Set --insecure in container when FLEET_SERVER_ENABLE and FLEET_INSECURE set. (elastic#25137)
  [filebeat] Update documentation / changelog / beta warnings for the syslog input (elastic#25047)
  Add support for ignore_inactive in filestream input (elastic#25036)
  Fix bug with annotations dedot config on k8s not used (elastic#25111)

ChrsMark added a commit that referenced this pull request


          Cherry-pick #25111 to 7.12: Fix bug with annotations dedot config on …

8d3ffe9

…k8s not used (#25162)

leweafan pushed a commit to leweafan/beats that referenced this pull request


          Cherry-pick elastic#25111 to 7.12: Fix bug with annotations dedot con…

cc8459c

…fig on k8s not used (elastic#25162)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Team:Integrations v7.11.0 v7.12.1 v7.13.0