This is a moderately involved change. The process is described in the steps
below, but the reader should refer to the commits in the PR to see exactly
what was done and when.

* port forward semi-processed events for tests

This brings partially processed event data from a modified version of the testing
code at 8896fd3 (the commit immediately prior to
the removal of the javascript processing pipeline in #29435: commit at
2f3b0c5cbe9cfdd10e11fd52e2a259e564001100).

The evtx.golden.json files were generated by removing the js processing call at
https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135
to match the code in that file as it appears here and then run go test -update
in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019
host.

The test package for each of the modules is also resurrected with modifications
reflecting the loss of the javascript processor.

Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this
commit.

* make sure metadata is available for enrichment of raw values

This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in
sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in
33acb3c (2022-01-25) after the origin of the
forward port origin (2021-12-02).

* update golden file for sysmon-11-filedeletedetected.evtx

* ignore opcode field on Windows 2022

Also prohibit generating golden files for PowerShell on Windows 2022 to
prevent unnecessary work in discovering that this will fail on other
versions.

* defer event field filtering until value comparison

This will result in additional diff noise if golden values are generated on multiple
versions of windows so it may be worth keeping the version used reasonably constant.
The version used here was 2019.

(cherry picked from commit f22abe3)

# Conflicts:
#	x-pack/winlogbeat/module/security/test/testdata/security-windows2019_4688_Process_Created.evtx.golden.json
#	x-pack/winlogbeat/module/testing_windows.go

…t processing (#31491)

* {,x-pack/}winlogbeat: resurrect tests for event processing (#31464)

This is a moderately involved change. The process is described in the steps
below, but the reader should refer to the commits in the PR to see exactly
what was done and when.

* port forward semi-processed events for tests

This brings partially processed event data from a modified version of the testing
code at 8896fd3 (the commit immediately prior to
the removal of the javascript processing pipeline in #29435: commit at
2f3b0c5cbe9cfdd10e11fd52e2a259e564001100).

The evtx.golden.json files were generated by removing the js processing call at
https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135
to match the code in that file as it appears here and then run go test -update
in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019
host.

The test package for each of the modules is also resurrected with modifications
reflecting the loss of the javascript processor.

Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this
commit.

* make sure metadata is available for enrichment of raw values

This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in
sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in
33acb3c (2022-01-25) after the origin of the
forward port origin (2021-12-02).

* update golden file for sysmon-11-filedeletedetected.evtx

* ignore opcode field on Windows 2022

Also prohibit generating golden files for PowerShell on Windows 2022 to
prevent unnecessary work in discovering that this will fail on other
versions.

* defer event field filtering until value comparison

This will result in additional diff noise if golden values are generated on multiple
versions of windows so it may be worth keeping the version used reasonably constant.
The version used here was 2019.

(cherry picked from commit f22abe3)

# Conflicts:
#	x-pack/winlogbeat/module/security/test/testdata/security-windows2019_4688_Process_Created.evtx.golden.json
#	x-pack/winlogbeat/module/testing_windows.go

* fix conflict scars

* silence bad linter

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

…d fix bugs (#31096)

* winlogbeat: fix event handling for Windows 2022 (#30942)

This fixes failures in event handling on Windows 2022[1] where parts of
events available from the Windows API are not reflected in the events
recovered by winlogbeat[2].

There is unfortunately quite a lot of movement in this change due to the
need to satisfy linter requirements. Beyond those changes, the substantive
changes here are:

1. Addition of new testing infrastructure to allow addition of evtx files
   and comparison with there expected XML renderings, and adding some test
   cases (shown below).
2. Fixing a buffer length parameter in the call to _EvtFormatMessage in
   evtFormatMessage that was the result of a lack of clarity in the API
   documentation for that syscall.
3. Fixing a var shadowing decl of the publisher handle EvtHandle in
   FormatEventString.
4. Providing a call back for the legacy (non-experimental) API through
   wineventlog.Message to allow it to obtain the event message in the case
   that the RenderingInfo element is not available via the Windows API.
5. Ensure that keyword, opcode and level are obtained by the non-experimental
   API by calling winevent.EnrichRawValuesWithNames in buildRecordFromXML.
   This change also required making winevent.Event.OpcodeRaw a pointer to
   allow an absent System>Opcode element to be distinquished from the zero,
   but present element.

The change also enables testing on Windows 2022.

[1]#30621
[2]#30622 (comment)

New events in testing as rendered by the Event Viewer:

ec1: eventcreate /id 1000 /t error /l application /d "My custom error event for the application log"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="EventCreate" />
          <EventID Qualifiers="0">1000</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.3106672Z" />
          <EventRecordID>316</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>My custom error event for the application log</Data>
      </EventData>
  </Event>

ec2: eventcreate /id 999 /t error /l application /so WinWord /d "Winword event 999 happened due to low diskspace"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="WinWord" />
          <EventID Qualifiers="0">999</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5132246Z" />
          <EventRecordID>317</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>Application</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Winword event 999 happened due to low diskspace</Data>
      </EventData>
  </Event>

ec3: eventcreate /id 5 /t error /l system /d "Catastrophe!"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="system" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.5455572Z" />
          <EventRecordID>1413</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Catastrophe!</Data>
      </EventData>
  </Event>

ec4: eventcreate /id 5 /t error /l system /so Backup /d "Backup failure"

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
          <Provider Name="Backup" />
          <EventID Qualifiers="0">5</EventID>
          <Version>0</Version>
          <Level>2</Level>
          <Task>0</Task>
          <Opcode>0</Opcode>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2022-03-22T02:03:11.8616638Z" />
          <EventRecordID>1414</EventRecordID>
          <Correlation />
          <Execution ProcessID="0" ThreadID="0" />
          <Channel>System</Channel>
          <Computer>vagrant</Computer>
          <Security UserID="S-1-5-21-2297499104-2362337018-4092230427-1000" />
      </System>
    - <EventData>
          <Data>Backup failure</Data>
      </EventData>
  </Event>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit 34bdc3d)

# Conflicts:
#	winlogbeat/Jenkinsfile.yml
#	winlogbeat/beater/winlogbeat.go
#	winlogbeat/eventlog/wineventlog.go
#	winlogbeat/eventlog/wineventlog_experimental.go
#	winlogbeat/sys/wineventlog/wineventlog_windows.go
#	x-pack/winlogbeat/Jenkinsfile.yml

* clean up conflict scars

* cherrypick winlogbeat test fixes relevant to 7.17 from #31464

Co-authored-by: Dan Kortschak <90160302+efd6@users.noreply.github.com>
Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

This is a moderately involved change. The process is described in the steps
below, but the reader should refer to the commits in the PR to see exactly
what was done and when.

* port forward semi-processed events for tests

This brings partially processed event data from a modified version of the testing
code at 8896fd3 (the commit immediately prior to
the removal of the javascript processing pipeline in #29435: commit at
2f3b0c5cbe9cfdd10e11fd52e2a259e564001100).

The evtx.golden.json files were generated by removing the js processing call at
https://github.com/elastic/beats/blob/8896fd319a257f3e0783119a7dd8d0978ef62197/x-pack/winlogbeat/module/testing_windows.go#L132-L135
to match the code in that file as it appears here and then run go test -update
in x-pack/winlogbeat/module/{powershell,security,sysmon}/test on a windows 2019
host.

The test package for each of the modules is also resurrected with modifications
reflecting the loss of the javascript processor.

Tests in x-pack/winlogbeat/module/{security,sysmon}/test fail in this
commit.

* make sure metadata is available for enrichment of raw values

This fixes failing tests in x-pack/winlogbeat/module/security/test, but tests in
sysmon continue to fail because sysmon-11-filedeletedetected.evtx was added in
33acb3c (2022-01-25) after the origin of the
forward port origin (2021-12-02).

* update golden file for sysmon-11-filedeletedetected.evtx

* ignore opcode field on Windows 2022

Also prohibit generating golden files for PowerShell on Windows 2022 to
prevent unnecessary work in discovering that this will fail on other
versions.

* defer event field filtering until value comparison

This will result in additional diff noise if golden values are generated on multiple
versions of windows so it may be worth keeping the version used reasonably constant.
The version used here was 2019.