winlogbeat/sys/wineventlog: extend testing and fix bugs#30942
winlogbeat/sys/wineventlog: extend testing and fix bugs#30942efd6 merged 19 commits intoelastic:mainfrom
Conversation
|
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
e5f159b to
15ccc7e
Compare
|
I pushed in a change to see how the tests react with Windows 2022. |
🤣 I'll fix this. Note: funny that the linter gets the format string wrong |
|
It looks like there is progress on the |
There was a problem hiding this comment.
IIRC the modulo on L186 is bad. The eventcreate event ID values should range on [1-1000] instead of [0-1000]. Saw this too when I was hacking.
Specifies the event ID for the event. A valid ID is any number from 1 to 1000.
https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/eventcreate
Yea. It's not clear to me how to get that. The change in 7c44687 does the majority of that, but the keywords are not obtainable through that path. |
|
This pull request is now in conflicts. Could you fix it? 🙏 |
New events in testing as rendered by the Event Viewer: ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="EventCreate" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" /> <EventRecordID>316</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>My custom error event for the application log</Data> </EventData> </Event> ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="WinWord" /> <EventID Qualifiers="0">999</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" /> <EventRecordID>317</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Winword event 999 happened due to low diskspace</Data> </EventData> </Event> ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="system" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" /> <EventRecordID>1413</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Catastrophe!</Data> </EventData> </Event> ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Backup" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" /> <EventRecordID>1414</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Backup failure</Data> </EventData> </Event>
andrewkroh
left a comment
There was a problem hiding this comment.
The tests are passing on Windows 2022 🍾
|
Because of the amount of delint spam, I'll write a fairly detailed commit message since the actual fix was only really two minor changes. |
|
That would be helpful, and can you add that message somewhere in the PR description too. |
This fixes failures in event handling on Windows 2022[1] where parts of events available from the Windows API are not reflected in the events recovered by winlogbeat[2]. There is unfortunately quite a lot of movement in this change due to the need to satisfy linter requirements. Beyond those changes, the substantive changes here are: 1. Addition of new testing infrastructure to allow addition of evtx files and comparison with there expected XML renderings, and adding some test cases (shown below). 2. Fixing a buffer length parameter in the call to _EvtFormatMessage in evtFormatMessage that was the result of a lack of clarity in the API documentation for that syscall. 3. Fixing a var shadowing decl of the publisher handle EvtHandle in FormatEventString. 4. Providing a call back for the legacy (non-experimental) API through wineventlog.Message to allow it to obtain the event message in the case that the RenderingInfo element is not available via the Windows API. 5. Ensure that keyword, opcode and level are obtained by the non-experimental API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML. This change also required making winevent.Event.OpcodeRaw a pointer to allow an absent System>Opcode element to be distinquished from the zero, but present element. The change also enables testing on Windows 2022. [1]#30621 [2]#30622 (comment) New events in testing as rendered by the Event Viewer: ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="EventCreate" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" /> <EventRecordID>316</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>My custom error event for the application log</Data> </EventData> </Event> ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="WinWord" /> <EventID Qualifiers="0">999</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" /> <EventRecordID>317</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Winword event 999 happened due to low diskspace</Data> </EventData> </Event> ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="system" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" /> <EventRecordID>1413</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Catastrophe!</Data> </EventData> </Event> ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Backup" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" /> <EventRecordID>1414</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Backup failure</Data> </EventData> </Event> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 34bdc3d) # Conflicts: # winlogbeat/beater/winlogbeat.go # winlogbeat/eventlog/wineventlog.go # winlogbeat/eventlog/wineventlog_experimental.go # winlogbeat/sys/wineventlog/wineventlog_windows.go # x-pack/winlogbeat/Jenkinsfile.yml
|
Do we want to backport to 7.17 and 8.2 as well now? |
…ments * upstream/main: winlogbeat: fix event handling for Windows 2022 (elastic#30942) [Release] add-backport-next (elastic#31056)
… fix bugs (#31075) * winlogbeat: fix event handling for Windows 2022 (#30942) This fixes failures in event handling on Windows 2022[1] where parts of events available from the Windows API are not reflected in the events recovered by winlogbeat[2]. There is unfortunately quite a lot of movement in this change due to the need to satisfy linter requirements. Beyond those changes, the substantive changes here are: 1. Addition of new testing infrastructure to allow addition of evtx files and comparison with there expected XML renderings, and adding some test cases (shown below). 2. Fixing a buffer length parameter in the call to _EvtFormatMessage in evtFormatMessage that was the result of a lack of clarity in the API documentation for that syscall. 3. Fixing a var shadowing decl of the publisher handle EvtHandle in FormatEventString. 4. Providing a call back for the legacy (non-experimental) API through wineventlog.Message to allow it to obtain the event message in the case that the RenderingInfo element is not available via the Windows API. 5. Ensure that keyword, opcode and level are obtained by the non-experimental API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML. This change also required making winevent.Event.OpcodeRaw a pointer to allow an absent System>Opcode element to be distinquished from the zero, but present element. The change also enables testing on Windows 2022. [1]#30621 [2]#30622 (comment) New events in testing as rendered by the Event Viewer: ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="EventCreate" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" /> <EventRecordID>316</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>My custom error event for the application log</Data> </EventData> </Event> ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="WinWord" /> <EventID Qualifiers="0">999</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" /> <EventRecordID>317</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Winword event 999 happened due to low diskspace</Data> </EventData> </Event> ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="system" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" /> <EventRecordID>1413</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Catastrophe!</Data> </EventData> </Event> ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Backup" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" /> <EventRecordID>1414</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Backup failure</Data> </EventData> </Event> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 34bdc3d) # Conflicts: # winlogbeat/beater/winlogbeat.go # winlogbeat/eventlog/wineventlog.go # winlogbeat/eventlog/wineventlog_experimental.go # winlogbeat/sys/wineventlog/wineventlog_windows.go # x-pack/winlogbeat/Jenkinsfile.yml * fix conflict Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
|
Yeah, let's backport this bugfix into both 8.2 and 7.17. |
This fixes failures in event handling on Windows 2022[1] where parts of events available from the Windows API are not reflected in the events recovered by winlogbeat[2]. There is unfortunately quite a lot of movement in this change due to the need to satisfy linter requirements. Beyond those changes, the substantive changes here are: 1. Addition of new testing infrastructure to allow addition of evtx files and comparison with there expected XML renderings, and adding some test cases (shown below). 2. Fixing a buffer length parameter in the call to _EvtFormatMessage in evtFormatMessage that was the result of a lack of clarity in the API documentation for that syscall. 3. Fixing a var shadowing decl of the publisher handle EvtHandle in FormatEventString. 4. Providing a call back for the legacy (non-experimental) API through wineventlog.Message to allow it to obtain the event message in the case that the RenderingInfo element is not available via the Windows API. 5. Ensure that keyword, opcode and level are obtained by the non-experimental API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML. This change also required making winevent.Event.OpcodeRaw a pointer to allow an absent System>Opcode element to be distinquished from the zero, but present element. The change also enables testing on Windows 2022. [1]#30621 [2]#30622 (comment) New events in testing as rendered by the Event Viewer: ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="EventCreate" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" /> <EventRecordID>316</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>My custom error event for the application log</Data> </EventData> </Event> ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="WinWord" /> <EventID Qualifiers="0">999</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" /> <EventRecordID>317</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Winword event 999 happened due to low diskspace</Data> </EventData> </Event> ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="system" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" /> <EventRecordID>1413</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Catastrophe!</Data> </EventData> </Event> ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Backup" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" /> <EventRecordID>1414</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Backup failure</Data> </EventData> </Event> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 34bdc3d) # Conflicts: # winlogbeat/Jenkinsfile.yml # winlogbeat/beater/winlogbeat.go # winlogbeat/eventlog/wineventlog.go # winlogbeat/eventlog/wineventlog_experimental.go # winlogbeat/sys/wineventlog/wineventlog_windows.go # x-pack/winlogbeat/Jenkinsfile.yml
This fixes failures in event handling on Windows 2022[1] where parts of events available from the Windows API are not reflected in the events recovered by winlogbeat[2]. There is unfortunately quite a lot of movement in this change due to the need to satisfy linter requirements. Beyond those changes, the substantive changes here are: 1. Addition of new testing infrastructure to allow addition of evtx files and comparison with there expected XML renderings, and adding some test cases (shown below). 2. Fixing a buffer length parameter in the call to _EvtFormatMessage in evtFormatMessage that was the result of a lack of clarity in the API documentation for that syscall. 3. Fixing a var shadowing decl of the publisher handle EvtHandle in FormatEventString. 4. Providing a call back for the legacy (non-experimental) API through wineventlog.Message to allow it to obtain the event message in the case that the RenderingInfo element is not available via the Windows API. 5. Ensure that keyword, opcode and level are obtained by the non-experimental API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML. This change also required making winevent.Event.OpcodeRaw a pointer to allow an absent System>Opcode element to be distinquished from the zero, but present element. The change also enables testing on Windows 2022. [1]#30621 [2]#30622 (comment) New events in testing as rendered by the Event Viewer: ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="EventCreate" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" /> <EventRecordID>316</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>My custom error event for the application log</Data> </EventData> </Event> ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="WinWord" /> <EventID Qualifiers="0">999</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" /> <EventRecordID>317</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Winword event 999 happened due to low diskspace</Data> </EventData> </Event> ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="system" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" /> <EventRecordID>1413</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Catastrophe!</Data> </EventData> </Event> ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Backup" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" /> <EventRecordID>1414</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Backup failure</Data> </EventData> </Event> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 34bdc3d)
This reverts commit 34bdc3d.
This fixes failures in event handling on Windows 2022[1] where parts of events available from the Windows API are not reflected in the events recovered by winlogbeat[2]. There is unfortunately quite a lot of movement in this change due to the need to satisfy linter requirements. Beyond those changes, the substantive changes here are: 1. Addition of new testing infrastructure to allow addition of evtx files and comparison with there expected XML renderings, and adding some test cases (shown below). 2. Fixing a buffer length parameter in the call to _EvtFormatMessage in evtFormatMessage that was the result of a lack of clarity in the API documentation for that syscall. 3. Fixing a var shadowing decl of the publisher handle EvtHandle in FormatEventString. 4. Providing a call back for the legacy (non-experimental) API through wineventlog.Message to allow it to obtain the event message in the case that the RenderingInfo element is not available via the Windows API. 5. Ensure that keyword, opcode and level are obtained by the non-experimental API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML. This change also required making winevent.Event.OpcodeRaw a pointer to allow an absent System>Opcode element to be distinquished from the zero, but present element. The change also enables testing on Windows 2022. [1]elastic#30621 [2]elastic#30622 (comment) New events in testing as rendered by the Event Viewer: ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="EventCreate" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" /> <EventRecordID>316</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>My custom error event for the application log</Data> </EventData> </Event> ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="WinWord" /> <EventID Qualifiers="0">999</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" /> <EventRecordID>317</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Winword event 999 happened due to low diskspace</Data> </EventData> </Event> ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="system" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" /> <EventRecordID>1413</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Catastrophe!</Data> </EventData> </Event> ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Backup" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" /> <EventRecordID>1414</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Backup failure</Data> </EventData> </Event> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
This fixes failures in event handling on Windows 2022[1] where parts of events available from the Windows API are not reflected in the events recovered by winlogbeat[2]. There is unfortunately quite a lot of movement in this change due to the need to satisfy linter requirements. Beyond those changes, the substantive changes here are: 1. Addition of new testing infrastructure to allow addition of evtx files and comparison with there expected XML renderings, and adding some test cases (shown below). 2. Fixing a buffer length parameter in the call to _EvtFormatMessage in evtFormatMessage that was the result of a lack of clarity in the API documentation for that syscall. 3. Fixing a var shadowing decl of the publisher handle EvtHandle in FormatEventString. 4. Providing a call back for the legacy (non-experimental) API through wineventlog.Message to allow it to obtain the event message in the case that the RenderingInfo element is not available via the Windows API. 5. Ensure that keyword, opcode and level are obtained by the non-experimental API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML. This change also required making winevent.Event.OpcodeRaw a pointer to allow an absent System>Opcode element to be distinquished from the zero, but present element. The change also enables testing on Windows 2022. [1]#30621 [2]#30622 (comment) New events in testing as rendered by the Event Viewer: ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="EventCreate" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" /> <EventRecordID>316</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>My custom error event for the application log</Data> </EventData> </Event> ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="WinWord" /> <EventID Qualifiers="0">999</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" /> <EventRecordID>317</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Winword event 999 happened due to low diskspace</Data> </EventData> </Event> ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="system" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" /> <EventRecordID>1413</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Catastrophe!</Data> </EventData> </Event> ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Backup" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" /> <EventRecordID>1414</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Backup failure</Data> </EventData> </Event> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 34bdc3d) Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This fixes failures in event handling on Windows 2022[1] where parts of events available from the Windows API are not reflected in the events recovered by winlogbeat[2]. There is unfortunately quite a lot of movement in this change due to the need to satisfy linter requirements. Beyond those changes, the substantive changes here are: 1. Addition of new testing infrastructure to allow addition of evtx files and comparison with there expected XML renderings, and adding some test cases (shown below). 2. Fixing a buffer length parameter in the call to _EvtFormatMessage in evtFormatMessage that was the result of a lack of clarity in the API documentation for that syscall. 3. Fixing a var shadowing decl of the publisher handle EvtHandle in FormatEventString. 4. Providing a call back for the legacy (non-experimental) API through wineventlog.Message to allow it to obtain the event message in the case that the RenderingInfo element is not available via the Windows API. 5. Ensure that keyword, opcode and level are obtained by the non-experimental API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML. This change also required making winevent.Event.OpcodeRaw a pointer to allow an absent System>Opcode element to be distinquished from the zero, but present element. The change also enables testing on Windows 2022. [1]elastic#30621 [2]elastic#30622 (comment) New events in testing as rendered by the Event Viewer: ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="EventCreate" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" /> <EventRecordID>316</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>My custom error event for the application log</Data> </EventData> </Event> ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="WinWord" /> <EventID Qualifiers="0">999</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" /> <EventRecordID>317</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Winword event 999 happened due to low diskspace</Data> </EventData> </Event> ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="system" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" /> <EventRecordID>1413</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Catastrophe!</Data> </EventData> </Event> ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Backup" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" /> <EventRecordID>1414</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Backup failure</Data> </EventData> </Event> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
…d fix bugs (#31096) * winlogbeat: fix event handling for Windows 2022 (#30942) This fixes failures in event handling on Windows 2022[1] where parts of events available from the Windows API are not reflected in the events recovered by winlogbeat[2]. There is unfortunately quite a lot of movement in this change due to the need to satisfy linter requirements. Beyond those changes, the substantive changes here are: 1. Addition of new testing infrastructure to allow addition of evtx files and comparison with there expected XML renderings, and adding some test cases (shown below). 2. Fixing a buffer length parameter in the call to _EvtFormatMessage in evtFormatMessage that was the result of a lack of clarity in the API documentation for that syscall. 3. Fixing a var shadowing decl of the publisher handle EvtHandle in FormatEventString. 4. Providing a call back for the legacy (non-experimental) API through wineventlog.Message to allow it to obtain the event message in the case that the RenderingInfo element is not available via the Windows API. 5. Ensure that keyword, opcode and level are obtained by the non-experimental API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML. This change also required making winevent.Event.OpcodeRaw a pointer to allow an absent System>Opcode element to be distinquished from the zero, but present element. The change also enables testing on Windows 2022. [1]#30621 [2]#30622 (comment) New events in testing as rendered by the Event Viewer: ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="EventCreate" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" /> <EventRecordID>316</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>My custom error event for the application log</Data> </EventData> </Event> ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="WinWord" /> <EventID Qualifiers="0">999</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" /> <EventRecordID>317</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Winword event 999 happened due to low diskspace</Data> </EventData> </Event> ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="system" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" /> <EventRecordID>1413</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Catastrophe!</Data> </EventData> </Event> ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Backup" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" /> <EventRecordID>1414</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Backup failure</Data> </EventData> </Event> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 34bdc3d) # Conflicts: # winlogbeat/Jenkinsfile.yml # winlogbeat/beater/winlogbeat.go # winlogbeat/eventlog/wineventlog.go # winlogbeat/eventlog/wineventlog_experimental.go # winlogbeat/sys/wineventlog/wineventlog_windows.go # x-pack/winlogbeat/Jenkinsfile.yml * clean up conflict scars * cherrypick winlogbeat test fixes relevant to 7.17 from #31464 Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com> Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
This fixes failures in event handling on Windows 2022[1] where parts of events available from the Windows API are not reflected in the events recovered by winlogbeat[2]. There is unfortunately quite a lot of movement in this change due to the need to satisfy linter requirements. Beyond those changes, the substantive changes here are: 1. Addition of new testing infrastructure to allow addition of evtx files and comparison with there expected XML renderings, and adding some test cases (shown below). 2. Fixing a buffer length parameter in the call to _EvtFormatMessage in evtFormatMessage that was the result of a lack of clarity in the API documentation for that syscall. 3. Fixing a var shadowing decl of the publisher handle EvtHandle in FormatEventString. 4. Providing a call back for the legacy (non-experimental) API through wineventlog.Message to allow it to obtain the event message in the case that the RenderingInfo element is not available via the Windows API. 5. Ensure that keyword, opcode and level are obtained by the non-experimental API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML. This change also required making winevent.Event.OpcodeRaw a pointer to allow an absent System>Opcode element to be distinquished from the zero, but present element. The change also enables testing on Windows 2022. [1]#30621 [2]#30622 (comment) New events in testing as rendered by the Event Viewer: ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="EventCreate" /> <EventID Qualifiers="0">1000</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" /> <EventRecordID>316</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>My custom error event for the application log</Data> </EventData> </Event> ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="WinWord" /> <EventID Qualifiers="0">999</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" /> <EventRecordID>317</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Winword event 999 happened due to low diskspace</Data> </EventData> </Event> ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="system" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" /> <EventRecordID>1413</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Catastrophe!</Data> </EventData> </Event> ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure" - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> - <System> <Provider Name="Backup" /> <EventID Qualifiers="0">5</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" /> <EventRecordID>1414</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>System</Channel> <Computer>vagrant</Computer> <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" /> </System> - <EventData> <Data>Backup failure</Data> </EventData> </Event> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
What does this PR do?
This fixes a couple of bug identified in #30621 and extends the test fixture to allow addition of further test cases.
This fixes failures in event handling on Windows 2022 where parts of
events available from the Windows API are not reflected in the events recovered by winlogbeat.
There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:
and comparison with there expected XML renderings, and adding some test
cases.
evtFormatMessage that was the result of a lack of clarity in the API
documentation for that syscall.
FormatEventString.
wineventlog.Message to allow it to obtain the event message in the case
that the RenderingInfo element is not available via the Windows API.
API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
This change also required making winevent.Event.OpcodeRaw a pointer to
allow an absent System>Opcode element to be distinquished from the zero,
but present element.
The change also enables testing on Windows 2022.
Why is it important?
The bugs look to have caused CI failure when adding Windows2022 to the build matrix #30622.
Checklist
- [ ] I have made corresponding changes to the documentation- [ ] I have made corresponding change to the default configuration filesCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs