ci: enable windows-2022 for winlogbeat#30622

Closed

v1v wants to merge 4 commits intoelastic:mainfrom

v1v:feature/enable-windows-2022-winlogbeat

Member

v1v commented Mar 1, 2022

What does this PR do?

Add windows-2022 support in the extended meta-stage.

Extended meta-stage is the one that runs after the mandatory stage.

Why is it important?

Windows-2022 is supported in 8.x

Issues


          ci: enable windows-2022 for winlogbeat

169db96

botelastic bot added the needs_team label

Contributor

mergify bot commented Mar 1, 2022

This pull request does not have a backport label. Could you fix it @v1v? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

mergify bot assigned v1v

mergify bot added the backport-skip label

v1v added a commit to v1v/beats that referenced this pull request


          Update x-pack/winlogbeat/Jenkinsfile.yml

3b2d512

See elastic#30622

v1v mentioned this pull request

ci: add windows-2022 in the extended meta-stage #30528

Merged

Contributor

elasticmachine commented Mar 1, 2022 •

edited by jenkins-beats-ci bot

Loading

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-17T11:25:15.036+0000
Duration: 46 min 25 sec

Test stats 🧪

Test	Results
Failed	10
Passed	473
Skipped	9
Total	492

Test errors

Expand to view the tests failures

`Extended / winlogbeat-windows-2022-windows-2022 / test_fields_not_under_root – winlogbeat.tests.system.test_wineventlog.Test`

Expand to view the error details

 KeyError: 'message'

Expand to view the stacktrace

 self = <test_wineventlog.Test testMethod=test_fields_not_under_root>

    def test_fields_not_under_root(self):
        """
        wineventlog - Add custom fields (not under root)
        """
        msg = "Add fields (not under root)"
        self.write_event_log(msg)
        evts = self.read_events(config={
            "fields": {"global": "field", "env": "prod", "level": "overwrite"},
            "event_logs": [
                {
                    "name": self.providerName,
                    "api": self.api,
                    "fields": {"local": "field", "env": "dev", "num": 1}
                }
            ]
        })
        self.assertTrue(len(evts), 1)
>       self.assert_common_fields(evts[0], msg=msg, extra={
            "log.level": "information",
            "winlog.keywords": ["Classic"],
            "winlog.opcode": "Info",
            "fields.global": "field",
            "fields.env": "dev",
            "fields.level": "overwrite",
            "fields.local": "field",
            "fields.num": 1,
        })

tests\system\test_wineventlog.py:145: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <test_wineventlog.Test testMethod=test_fields_not_under_root>
evt = {'@metadata.beat': 'winlogbeat', '@metadata.type': '_doc', '@metadata.version': '8.2.0', '@timestamp': '2022-03-17T12:09:02.329Z', ...}
msg = 'Add fields (not under root)', eventID = '10', sid = None
level = 'information'
extra = {'fields.env': 'dev', 'fields.global': 'field', 'fields.level': 'overwrite', 'fields.local': 'field', ...}

    def assert_common_fields(self, evt, msg=None, eventID="10", sid=None,
                             level="information", extra=None):
    
        assert host_name(evt["winlog.computer_name"]).lower() == host_name(platform.node()).lower()
        assert "winlog.record_id" in evt
        expected = {
            "winlog.event_id": eventID,
            "event.code": eventID,
            "log.level": level.lower(),
            "winlog.channel": self.providerName,
            "winlog.provider_name": self.applicationName,
            "winlog.api": self.api,
        }
        assert expected.items() <= evt.items()
    
        if msg is None:
            assert "message" not in evt
        else:
>           self.assertEqual(evt["message"], msg)
E           KeyError: 'message'

tests\system\winlogbeat.py:156: KeyError

`Extended / winlogbeat-windows-2022-windows-2022 / test_fields_under_root – winlogbeat.tests.system.test_wineventlog.Test`

Expand to view the error details

 KeyError: 'message'

Expand to view the stacktrace

 self = <test_wineventlog.Test testMethod=test_fields_under_root>

    def test_fields_under_root(self):
        """
        wineventlog - Add tags and custom fields under root
        """
        msg = "Add tags and fields under root"
        self.write_event_log(msg)
        evts = self.read_events(config={
            "tags": ["global"],
            "fields": {"global": "field", "env": "prod", "log.level": "overwrite"},
            "fields_under_root": True,
            "event_logs": [
                {
                    "name": self.providerName,
                    "api": self.api,
                    "tags": ["local"],
                    "fields_under_root": True,
                    "fields": {"local": "field", "env": "dev"}
                }
            ]
        })
        self.assertTrue(len(evts), 1)
>       self.assert_common_fields(evts[0], msg=msg, level="overwrite", extra={
            "winlog.keywords": ["Classic"],
            "winlog.opcode": "Info",
            "global": "field",
            "env": "dev",
            "local": "field",
            "tags": ["global", "local"],
        })

tests\system\test_wineventlog.py:119: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <test_wineventlog.Test testMethod=test_fields_under_root>
evt = {'@metadata.beat': 'winlogbeat', '@metadata.type': '_doc', '@metadata.version': '8.2.0', '@timestamp': '2022-03-17T12:09:03.804Z', ...}
msg = 'Add tags and fields under root', eventID = '10', sid = None
level = 'overwrite'
extra = {'env': 'dev', 'global': 'field', 'local': 'field', 'tags': ['global', 'local'], ...}

    def assert_common_fields(self, evt, msg=None, eventID="10", sid=None,
                             level="information", extra=None):
    
        assert host_name(evt["winlog.computer_name"]).lower() == host_name(platform.node()).lower()
        assert "winlog.record_id" in evt
        expected = {
            "winlog.event_id": eventID,
            "event.code": eventID,
            "log.level": level.lower(),
            "winlog.channel": self.providerName,
            "winlog.provider_name": self.applicationName,
            "winlog.api": self.api,
        }
        assert expected.items() <= evt.items()
    
        if msg is None:
            assert "message" not in evt
        else:
>           self.assertEqual(evt["message"], msg)
E           KeyError: 'message'

tests\system\winlogbeat.py:156: KeyError

`Extended / winlogbeat-windows-2022-windows-2022 / test_include_xml – winlogbeat.tests.system.test_wineventlog.Test`

Expand to view the error details

 KeyError: 'message'

Expand to view the stacktrace

 self = <test_wineventlog.Test testMethod=test_include_xml>

    def test_include_xml(self):
        """
        wineventlog - Include raw XML event
        """
        msg = "Include raw XML event"
        self.write_event_log(msg)
        evts = self.read_events(config={
            "event_logs": [
                {
                    "name": self.providerName,
                    "api": self.api,
                    "include_xml": True,
                }
            ]
        })
        self.assertTrue(len(evts), 1)
>       self.assert_common_fields(evts[0], msg=msg)

tests\system\test_wineventlog.py:173: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <test_wineventlog.Test testMethod=test_include_xml>
evt = {'@metadata.beat': 'winlogbeat', '@metadata.type': '_doc', '@metadata.version': '8.2.0', '@timestamp': '2022-03-17T12:09:05.125Z', ...}
msg = 'Include raw XML event', eventID = '10', sid = None, level = 'information'
extra = None

    def assert_common_fields(self, evt, msg=None, eventID="10", sid=None,
                             level="information", extra=None):
    
        assert host_name(evt["winlog.computer_name"]).lower() == host_name(platform.node()).lower()
        assert "winlog.record_id" in evt
        expected = {
            "winlog.event_id": eventID,
            "event.code": eventID,
            "log.level": level.lower(),
            "winlog.channel": self.providerName,
            "winlog.provider_name": self.applicationName,
            "winlog.api": self.api,
        }
        assert expected.items() <= evt.items()
    
        if msg is None:
            assert "message" not in evt
        else:
>           self.assertEqual(evt["message"], msg)
E           KeyError: 'message'

tests\system\winlogbeat.py:156: KeyError

`Extended / winlogbeat-windows-2022-windows-2022 / test_multiline_events – winlogbeat.tests.system.test_wineventlog.Test`

Expand to view the error details

 AssertionError: 'message' not found in {'@timestamp': '2022-03-17T12:09:06.443Z', '@metadata.beat': 'winlogbeat', '@metadata.type': '_doc', '@metadata.version': '8.2.0', 'winlog.channel': 'WinlogbeatTestPython_93e80', 'winlog.computer_name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'winlog.event_id': '10', 'winlog.provider_name': 'SystemTest_93e80', 'winlog.record_id': 3, 'winlog.api': 'wineventlog', 'winlog.user.identifier': 'S-1-5-21-2815709104-2621471230-1930217857-1170', 'winlog.user.domain': 'BEATS-CI-IMMUTA', 'winlog.user.name': 'jenkins', 'winlog.user.type': 'User', 'winlog.event_data.param1': '\nA trusted logon process has been registered with the Local Security Authority.\nThis logon process will be trusted to submit logon requests.\n\nSubject:\n\nSecurity ID:  SYSTEM\nAccount Name:  MS4\\u001e$\nAccount Domain:  WORKGROUP\nLogon ID:  0x3e7\nLogon Process Name:  IKE', 'event.created': '2022-03-17T12:09:06.596Z', 'event.code': '10', 'event.kind': 'event', 'event.provider': 'SystemTest_93e80', 'log.level': 'information', 'host.name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'ecs.version': '8.0.0', 'agent.id': 'd0200409-835f-44f7-a7cc-8a7b7b1ffeb5', 'agent.name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'agent.type': 'winlogbeat', 'agent.version': '8.2.0', 'agent.ephemeral_id': '68712fb8-befd-42ac-8151-f225cbc6f071'} : {'@timestamp': '2022-03-17T12:09:06.443Z', '@metadata.beat': 'winlogbeat', '@metadata.type': '_doc', '@metadata.version': '8.2.0', 'winlog.channel': 'WinlogbeatTestPython_93e80', 'winlog.computer_name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'winlog.event_id': '10', 'winlog.provider_name': 'SystemTest_93e80', 'winlog.record_id': 3, 'winlog.api': 'wineventlog', 'winlog.user.identifier': 'S-1-5-21-2815709104-2621471230-1930217857-1170', 'winlog.user.domain': 'BEATS-CI-IMMUTA', 'winlog.user.name': 'jenkins', 'winlog.user.type': 'User', 'winlog.event_data.param1': '\nA trusted logon process has been registered with the Local Security Authority.\nThis logon process will be trusted to submit logon requests.\n\nSubject:\n\nSecurity ID:  SYSTEM\nAccount Name:  MS4\\u001e$\nAccount Domain:  WORKGROUP\nLogon ID:  0x3e7\nLogon Process Name:  IKE', 'event.created': '2022-03-17T12:09:06.596Z', 'event.code': '10', 'event.kind': 'event', 'event.provider': 'SystemTest_93e80', 'log.level': 'information', 'host.name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'ecs.version': '8.0.0', 'agent.id': 'd0200409-835f-44f7-a7cc-8a7b7b1ffeb5', 'agent.name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'agent.type': 'winlogbeat', 'agent.version': '8.2.0', 'agent.ephemeral_id': '68712fb8-befd-42ac-8151-f225cbc6f071'}

Expand to view the stacktrace

 self = <test_wineventlog.Test testMethod=test_multiline_events>

        def test_multiline_events(self):
            """
            wineventlog - Event with newlines and control characters
            """
            msg = """
    A trusted logon process has been registered with the Local Security Authority.
    This logon process will be trusted to submit logon requests.
    
    Subject:
    
    Security ID:  SYSTEM
    Account Name:  MS4\x1e$
    Account Domain:  WORKGROUP
    Logon ID:  0x3e7
    Logon Process Name:  IKE"""
            self.write_event_log(msg)
            evts = self.read_events()
            self.assertTrue(len(evts), 1)
            self.assertEqual(str(self.api), evts[0]["winlog.api"], msg=evts[0])
            self.assertNotIn("event.original", evts[0], msg=evts[0])
>           self.assertIn("message", evts[0], msg=evts[0])
E           AssertionError: 'message' not found in {'@timestamp': '2022-03-17T12:09:06.443Z', '@metadata.beat': 'winlogbeat', '@metadata.type': '_doc', '@metadata.version': '8.2.0', 'winlog.channel': 'WinlogbeatTestPython_93e80', 'winlog.computer_name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'winlog.event_id': '10', 'winlog.provider_name': 'SystemTest_93e80', 'winlog.record_id': 3, 'winlog.api': 'wineventlog', 'winlog.user.identifier': 'S-1-5-21-2815709104-2621471230-1930217857-1170', 'winlog.user.domain': 'BEATS-CI-IMMUTA', 'winlog.user.name': 'jenkins', 'winlog.user.type': 'User', 'winlog.event_data.param1': '\nA trusted logon process has been registered with the Local Security Authority.\nThis logon process will be trusted to submit logon requests.\n\nSubject:\n\nSecurity ID:  SYSTEM\nAccount Name:  MS4\\u001e$\nAccount Domain:  WORKGROUP\nLogon ID:  0x3e7\nLogon Process Name:  IKE', 'event.created': '2022-03-17T12:09:06.596Z', 'event.code': '10', 'event.kind': 'event', 'event.provider': 'SystemTest_93e80', 'log.level': 'information', 'host.name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'ecs.version': '8.0.0', 'agent.id': 'd0200409-835f-44f7-a7cc-8a7b7b1ffeb5', 'agent.name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'agent.type': 'winlogbeat', 'agent.version': '8.2.0', 'agent.ephemeral_id': '68712fb8-befd-42ac-8151-f225cbc6f071'} : {'@timestamp': '2022-03-17T12:09:06.443Z', '@metadata.beat': 'winlogbeat', '@metadata.type': '_doc', '@metadata.version': '8.2.0', 'winlog.channel': 'WinlogbeatTestPython_93e80', 'winlog.computer_name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'winlog.event_id': '10', 'winlog.provider_name': 'SystemTest_93e80', 'winlog.record_id': 3, 'winlog.api': 'wineventlog', 'winlog.user.identifier': 'S-1-5-21-2815709104-2621471230-1930217857-1170', 'winlog.user.domain': 'BEATS-CI-IMMUTA', 'winlog.user.name': 'jenkins', 'winlog.user.type': 'User', 'winlog.event_data.param1': '\nA trusted logon process has been registered with the Local Security Authority.\nThis logon process will be trusted to submit logon requests.\n\nSubject:\n\nSecurity ID:  SYSTEM\nAccount Name:  MS4\\u001e$\nAccount Domain:  WORKGROUP\nLogon ID:  0x3e7\nLogon Process Name:  IKE', 'event.created': '2022-03-17T12:09:06.596Z', 'event.code': '10', 'event.kind': 'event', 'event.provider': 'SystemTest_93e80', 'log.level': 'information', 'host.name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'ecs.version': '8.0.0', 'agent.id': 'd0200409-835f-44f7-a7cc-8a7b7b1ffeb5', 'agent.name': 'beats-ci-immutable-windows-2022-1647518153162920497', 'agent.type': 'winlogbeat', 'agent.version': '8.2.0', 'agent.ephemeral_id': '68712fb8-befd-42ac-8151-f225cbc6f071'}

tests\system\test_wineventlog.py:398: AssertionError

`Extended / winlogbeat-windows-2022-windows-2022 / test_query_multi_param – winlogbeat.tests.system.test_wineventlog.Test`

Expand to view the error details

 KeyError: 'message'

Expand to view the stacktrace

 self = <test_wineventlog.Test testMethod=test_query_multi_param>

    def test_query_multi_param(self):
        """
        wineventlog - Query by multiple params
        """
        self.write_event_log("selected", source=self.otherAppName,
                             eventID=556, level=win32evtlog.EVENTLOG_ERROR_TYPE)
        self.write_event_log("filtered", source=self.otherAppName, eventID=556)
        self.write_event_log(
            "filtered", level=win32evtlog.EVENTLOG_WARNING_TYPE)
        evts = self.read_events(config={
            "event_logs": [
                {
                    "name": self.providerName,
                    "api": self.api,
                    "event_id": "10-20, 30-40, -35, -18, 400-1000, -432",
                    "level": "warn, error",
                    "provider": [self.otherAppName]
                }
            ]
        })
        self.assertTrue(len(evts), 1)
>       self.assertEqual(evts[0]["message"], "selected")
E       KeyError: 'message'

tests\system\test_wineventlog.py:314: KeyError

`Extended / winlogbeat-windows-2022-windows-2022 / test_read_one_event – winlogbeat.tests.system.test_wineventlog.Test`

Expand to view the error details

 KeyError: 'message'

Expand to view the stacktrace

 self = <test_wineventlog.Test testMethod=test_read_one_event>

    def test_read_one_event(self):
        """
        wineventlog - Read one classic event
        """
        msg = "Hello world!"
        self.write_event_log(msg)
        evts = self.read_events()
        self.assertTrue(len(evts), 1)
>       self.assert_common_fields(evts[0], msg=msg, extra={
            "winlog.keywords": ["Classic"],
            "winlog.opcode": "Info",
        })

tests\system\test_wineventlog.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <test_wineventlog.Test testMethod=test_read_one_event>
evt = {'@metadata.beat': 'winlogbeat', '@metadata.type': '_doc', '@metadata.version': '8.2.0', '@timestamp': '2022-03-17T12:09:18.884Z', ...}
msg = 'Hello world!', eventID = '10', sid = None, level = 'information'
extra = {'winlog.keywords': ['Classic'], 'winlog.opcode': 'Info'}

    def assert_common_fields(self, evt, msg=None, eventID="10", sid=None,
                             level="information", extra=None):
    
        assert host_name(evt["winlog.computer_name"]).lower() == host_name(platform.node()).lower()
        assert "winlog.record_id" in evt
        expected = {
            "winlog.event_id": eventID,
            "event.code": eventID,
            "log.level": level.lower(),
            "winlog.channel": self.providerName,
            "winlog.provider_name": self.applicationName,
            "winlog.api": self.api,
        }
        assert expected.items() <= evt.items()
    
        if msg is None:
            assert "message" not in evt
        else:
>           self.assertEqual(evt["message"], msg)
E           KeyError: 'message'

tests\system\winlogbeat.py:156: KeyError

`Extended / winlogbeat-windows-2022-windows-2022 / test_read_unknown_event_id – winlogbeat.tests.system.test_wineventlog.Test`

Expand to view the error details

 AssertionError

Expand to view the stacktrace

 self = <test_wineventlog.Test testMethod=test_read_unknown_event_id>

    def test_read_unknown_event_id(self):
        """
        wineventlog - Read unknown event ID
        """
        msg = "Unknown event ID"
        self.write_event_log(msg, eventID=1111)
        evts = self.read_events()
        self.assertTrue(len(evts), 1)
>       self.assert_common_fields(evts[0], eventID="1111", extra={
            "winlog.keywords": ["Classic"],
            "winlog.opcode": "Info",
        })

tests\system\test_wineventlog.py:74: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <test_wineventlog.Test testMethod=test_read_unknown_event_id>
evt = {'@metadata.beat': 'winlogbeat', '@metadata.type': '_doc', '@metadata.version': '8.2.0', '@timestamp': '2022-03-17T12:09:20.198Z', ...}
msg = None, eventID = '1111', sid = None, level = 'information'
extra = {'winlog.keywords': ['Classic'], 'winlog.opcode': 'Info'}

    def assert_common_fields(self, evt, msg=None, eventID="10", sid=None,
                             level="information", extra=None):
    
        assert host_name(evt["winlog.computer_name"]).lower() == host_name(platform.node()).lower()
        assert "winlog.record_id" in evt
        expected = {
            "winlog.event_id": eventID,
            "event.code": eventID,
            "log.level": level.lower(),
            "winlog.channel": self.providerName,
            "winlog.provider_name": self.applicationName,
            "winlog.api": self.api,
        }
        assert expected.items() <= evt.items()
    
        if msg is None:
            assert "message" not in evt
        else:
            self.assertEqual(evt["message"], msg)
            self.assertEqual(msg, evt.get("winlog.event_data.param1"))
    
        if sid is None:
            self.assertEqual(evt["winlog.user.identifier"], self.get_sid_string())
            self.assertEqual(evt["winlog.user.name"].lower(),
                             win32api.GetUserName().lower())
            self.assertEqual(evt["winlog.user.type"], "User")
            assert "winlog.user.domain" in evt
        else:
            self.assertEqual(evt["winlog.user.identifier"], sid)
            assert "winlog.user.name" not in evt
            assert "winlog.user.type" not in evt
    
        if extra is not None:
>           assert extra.items() <= evt.items()
E           AssertionError

tests\system\winlogbeat.py:171: AssertionError

`Extended / winlogbeat-windows-2022-windows-2022 / test_read_unknown_sid – winlogbeat.tests.system.test_wineventlog.Test`

Expand to view the error details

 KeyError: 'message'

Expand to view the stacktrace

 self = <test_wineventlog.Test testMethod=test_read_unknown_sid>

    def test_read_unknown_sid(self):
        """
        wineventlog - Read event with unknown SID
        """
        # Fake SID that was made up.
        accountIdentifier = "S-1-5-21-3623811015-3361044348-30300820-1013"
        sid = win32security.ConvertStringSidToSid(accountIdentifier)
    
        msg = "Unknown SID " + accountIdentifier
        self.write_event_log(msg, sid=sid)
        evts = self.read_events()
        self.assertTrue(len(evts), 1)
>       self.assert_common_fields(evts[0], msg=msg, sid=accountIdentifier, extra={
            "winlog.keywords": ["Classic"],
            "winlog.opcode": "Info",
        })

tests\system\test_wineventlog.py:93: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <test_wineventlog.Test testMethod=test_read_unknown_sid>
evt = {'@metadata.beat': 'winlogbeat', '@metadata.type': '_doc', '@metadata.version': '8.2.0', '@timestamp': '2022-03-17T12:09:21.508Z', ...}
msg = 'Unknown SID S-1-5-21-3623811015-3361044348-30300820-1013', eventID = '10'
sid = 'S-1-5-21-3623811015-3361044348-30300820-1013', level = 'information'
extra = {'winlog.keywords': ['Classic'], 'winlog.opcode': 'Info'}

    def assert_common_fields(self, evt, msg=None, eventID="10", sid=None,
                             level="information", extra=None):
    
        assert host_name(evt["winlog.computer_name"]).lower() == host_name(platform.node()).lower()
        assert "winlog.record_id" in evt
        expected = {
            "winlog.event_id": eventID,
            "event.code": eventID,
            "log.level": level.lower(),
            "winlog.channel": self.providerName,
            "winlog.provider_name": self.applicationName,
            "winlog.api": self.api,
        }
        assert expected.items() <= evt.items()
    
        if msg is None:
            assert "message" not in evt
        else:
>           self.assertEqual(evt["message"], msg)
E           KeyError: 'message'

tests\system\winlogbeat.py:156: KeyError

`Extended / winlogbeat-windows-2022-windows-2022 / test_resume_reading_events – winlogbeat.tests.system.test_wineventlog.Test`

Expand to view the error details

 KeyError: 'message'

Expand to view the stacktrace

 self = <test_wineventlog.Test testMethod=test_resume_reading_events>

    def test_resume_reading_events(self):
        """
        wineventlog - Resume reading events
        """
        msg = "First event"
        self.write_event_log(msg)
        evts = self.read_events()
        self.assertTrue(len(evts), 1)
>       self.assert_common_fields(evts[0], msg=msg, extra={
            "winlog.keywords": ["Classic"],
            "winlog.opcode": "Info",
        })

tests\system\test_wineventlog.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <test_wineventlog.Test testMethod=test_resume_reading_events>
evt = {'@metadata.beat': 'winlogbeat', '@metadata.type': '_doc', '@metadata.version': '8.2.0', '@timestamp': '2022-03-17T12:09:24.134Z', ...}
msg = 'First event', eventID = '10', sid = None, level = 'information'
extra = {'winlog.keywords': ['Classic'], 'winlog.opcode': 'Info'}

    def assert_common_fields(self, evt, msg=None, eventID="10", sid=None,
                             level="information", extra=None):
    
        assert host_name(evt["winlog.computer_name"]).lower() == host_name(platform.node()).lower()
        assert "winlog.record_id" in evt
        expected = {
            "winlog.event_id": eventID,
            "event.code": eventID,
            "log.level": level.lower(),
            "winlog.channel": self.providerName,
            "winlog.provider_name": self.applicationName,
            "winlog.api": self.api,
        }
        assert expected.items() <= evt.items()
    
        if msg is None:
            assert "message" not in evt
        else:
>           self.assertEqual(evt["message"], msg)
E           KeyError: 'message'

tests\system\winlogbeat.py:156: KeyError

`Extended / winlogbeat-windows-2022-windows-2022 / test_utf16_characters – winlogbeat.tests.system.test_wineventlog.Test`

Expand to view the error details

 KeyError: 'message'

Expand to view the stacktrace

 self = <test_wineventlog.Test testMethod=test_utf16_characters>

    def test_utf16_characters(self):
        """
        wineventlog - UTF-16 characters
        """
        msg = (u'\u89E3\u51CD\u3057\u305F\u30D5\u30A9\u30EB\u30C0\u306E'
               u'\u30A4\u30F3\u30B9\u30C8\u30FC\u30EB\u30B9\u30AF\u30EA'
               u'\u30D7\u30C8\u3092\u5B9F\u884C\u3057'
               u'\u8C61\u5F62\u5B57')
        self.write_event_log(str(msg))
        evts = self.read_events(config={
            "event_logs": [
                {
                    "name": self.providerName,
                    "api": self.api,
                    "include_xml": True,
                }
            ]
        })
        self.assertTrue(len(evts), 1)
>       self.assertEqual(evts[0]["message"], msg)
E       KeyError: 'message'

tests\system\test_wineventlog.py:335: KeyError

Steps errors

Expand to view the steps failures

`winlogbeat-windows-2022-windows-2022 - mage build unitTest`

Took 4 min 6 sec . View more details here
Description: mage build unitTest

`winlogbeat-windows-2022-windows-2022 - mage build unitTest`

Took 1 min 7 sec . View more details here
Description: mage build unitTest

`winlogbeat-windows-2022-windows-2022 - mage build unitTest`

Took 1 min 8 sec . View more details here
Description: mage build unitTest

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Extended / winlogbeat-windows-2022-windows-2022 / test_fields_not_under_root – winlogbeat.tests.system.test_wineventlog.Test
Name: Extended / winlogbeat-windows-2022-windows-2022 / test_fields_under_root – winlogbeat.tests.system.test_wineventlog.Test
Name: Extended / winlogbeat-windows-2022-windows-2022 / test_include_xml – winlogbeat.tests.system.test_wineventlog.Test
Name: Extended / winlogbeat-windows-2022-windows-2022 / test_multiline_events – winlogbeat.tests.system.test_wineventlog.Test
Name: Extended / winlogbeat-windows-2022-windows-2022 / test_query_multi_param – winlogbeat.tests.system.test_wineventlog.Test
Name: Extended / winlogbeat-windows-2022-windows-2022 / test_read_one_event – winlogbeat.tests.system.test_wineventlog.Test
Name: Extended / winlogbeat-windows-2022-windows-2022 / test_read_unknown_event_id – winlogbeat.tests.system.test_wineventlog.Test
Name: Extended / winlogbeat-windows-2022-windows-2022 / test_read_unknown_sid – winlogbeat.tests.system.test_wineventlog.Test
Name: Extended / winlogbeat-windows-2022-windows-2022 / test_resume_reading_events – winlogbeat.tests.system.test_wineventlog.Test
Name: Extended / winlogbeat-windows-2022-windows-2022 / test_utf16_characters – winlogbeat.tests.system.test_wineventlog.Test

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

v1v added 2 commits

March 1, 2022 15:03


          ci: enable windows-2022 support

f146a3a


          Revert "ci: enable windows-2022 for winlogbeat"

a647625

This reverts commit 169db96.

v1v commented

View reviewed changes

winlogbeat/Jenkinsfile.yml Outdated Show resolved Hide resolved


          Update winlogbeat/Jenkinsfile.yml

ca99055

andrewkroh mentioned this pull request

winlogbeat: system.test_wineventlog failures for Windows-2022 #30621

Closed

Member

andrewkroh commented Mar 1, 2022

I checked one of the tests and message is missing from the output document. IIRC these tests rely on a Windows application at %systemroot%\\system32\\EventCreate.exe. Maybe something change w.r.t. that application.

test_wineventlog.Test.test_utf16_characters akroh$ jq .message output/winlogbeat-20220301.ndjson
null

Member Author

v1v commented Mar 17, 2022

/test

mtojek added the Team:Automation label

botelastic bot removed the needs_team label

efd6 mentioned this pull request

winlogbeat/sys/wineventlog: extend testing and fix bugs #30942

Merged

4 tasks

efd6 added a commit that referenced this pull request


          winlogbeat: fix event handling for Windows 2022 (#30942)

34bdc3d

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]#30621
[2]#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

mergify bot pushed a commit that referenced this pull request


          winlogbeat: fix event handling for Windows 2022 (#30942)

f1c49d1

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]#30621
[2]#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit 34bdc3d)

# Conflicts:
#	winlogbeat/beater/winlogbeat.go
#	winlogbeat/eventlog/wineventlog.go
#	winlogbeat/eventlog/wineventlog_experimental.go
#	winlogbeat/sys/wineventlog/wineventlog_windows.go
#	x-pack/winlogbeat/Jenkinsfile.yml

Member Author

v1v commented Mar 31, 2022

#30942 has enabled this stage, so I'll close this now

v1v closed this

efd6 added a commit that referenced this pull request


          [8.1](backport #30942) winlogbeat/sys/wineventlog: extend testing and…

73f93c4

… fix bugs (#31075)

* winlogbeat: fix event handling for Windows 2022 (#30942)

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]#30621
[2]#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit 34bdc3d)

# Conflicts:
#	winlogbeat/beater/winlogbeat.go
#	winlogbeat/eventlog/wineventlog.go
#	winlogbeat/eventlog/wineventlog_experimental.go
#	winlogbeat/sys/wineventlog/wineventlog_windows.go
#	x-pack/winlogbeat/Jenkinsfile.yml

* fix conflict

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

mergify bot pushed a commit that referenced this pull request


          winlogbeat: fix event handling for Windows 2022 (#30942)

f0be195

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]#30621
[2]#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit 34bdc3d)

# Conflicts:
#	winlogbeat/Jenkinsfile.yml
#	winlogbeat/beater/winlogbeat.go
#	winlogbeat/eventlog/wineventlog.go
#	winlogbeat/eventlog/wineventlog_experimental.go
#	winlogbeat/sys/wineventlog/wineventlog_windows.go
#	x-pack/winlogbeat/Jenkinsfile.yml

mergify bot pushed a commit that referenced this pull request


          winlogbeat: fix event handling for Windows 2022 (#30942)

cbe32e0

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]#30621
[2]#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit 34bdc3d)

emilioalvap pushed a commit to emilioalvap/beats that referenced this pull request


          winlogbeat: fix event handling for Windows 2022 (elastic#30942)

3bf6d25

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]elastic#30621
[2]elastic#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

efd6 added a commit that referenced this pull request


          winlogbeat: fix event handling for Windows 2022 (#30942) (#31097)

0af33ec

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]#30621
[2]#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit 34bdc3d)

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

kush-elastic pushed a commit to kush-elastic/beats that referenced this pull request


          winlogbeat: fix event handling for Windows 2022 (elastic#30942)

eb58085

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]elastic#30621
[2]elastic#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

efd6 added a commit that referenced this pull request


          [7.17](backport #30942) winlogbeat/sys/wineventlog: extend testing an…

953f799

…d fix bugs (#31096)

* winlogbeat: fix event handling for Windows 2022 (#30942)

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]#30621
[2]#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit 34bdc3d)

# Conflicts:
#	winlogbeat/Jenkinsfile.yml
#	winlogbeat/beater/winlogbeat.go
#	winlogbeat/eventlog/wineventlog.go
#	winlogbeat/eventlog/wineventlog_experimental.go
#	winlogbeat/sys/wineventlog/wineventlog_windows.go
#	x-pack/winlogbeat/Jenkinsfile.yml

* clean up conflict scars

* cherrypick winlogbeat test fixes relevant to 7.17 from #31464

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

chrisberkhout pushed a commit that referenced this pull request


          winlogbeat: fix event handling for Windows 2022 (#30942)

d817941

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]#30621
[2]#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-skip Team:Automation