Skip to content

Win eventlog input plugin#7020

Closed
anuar45 wants to merge 6 commits intoinfluxdata:masterfrom
anuar45:win_eventlog-input-plugin
Closed

Win eventlog input plugin#7020
anuar45 wants to merge 6 commits intoinfluxdata:masterfrom
anuar45:win_eventlog-input-plugin

Conversation

@anuar45
Copy link
Copy Markdown
Contributor

@anuar45 anuar45 commented Feb 13, 2020

Required for all PRs:

  • Signed CLA.
  • Associated README.md updated.
  • Has appropriate unit tests.

Feature #4525

@anuar45 anuar45 requested a review from danielnelson February 17, 2020 16:17
@natejgardner
Copy link
Copy Markdown

@danielnelson we're really looking forward to this feature

@timhallinflux timhallinflux added this to the 1.15.0 milestone Apr 29, 2020
@vavdoshka
Copy link
Copy Markdown

guys, is there any plan about merging this?

@anuar45 anuar45 marked this pull request as draft June 4, 2020 20:13
@anuar45 anuar45 force-pushed the win_eventlog-input-plugin branch from 16d2ae4 to 572979e Compare June 23, 2020 20:56
@anuar45 anuar45 marked this pull request as ready for review June 23, 2020 20:59
@sjwang90 sjwang90 modified the milestones: 1.15.0, 1.16.0 Jun 26, 2020
@utsah87
Copy link
Copy Markdown

utsah87 commented Jul 6, 2020

@anuar45 Can this plugin be updated to have a user defined filter to filter events on the basis of source and event ids. i.e. if we want to have winlog events for a particular source and for any defined EventIDs.

@anuar45
Copy link
Copy Markdown
Contributor Author

anuar45 commented Jul 7, 2020

@utsah87 It is already supported, for example to query for specific EventID you can use following config parameter xpath_query = "Event/System[EventID=999]" and so on...

reference https://docs.microsoft.com/en-us/windows/win32/wes/consuming-events

@utsah87
Copy link
Copy Markdown

utsah87 commented Jul 8, 2020

@utsah87 It is already supported, for example to query for specific EventID you can use following config parameter xpath_query = "Event/System[EventID=999]" and so on...

reference https://docs.microsoft.com/en-us/windows/win32/wes/consuming-events

Thanks for the reply. Am able to use this correctly now.
One more question is that we are getting the timestamp for application logs but not for System logs, is this as per design or need to be fixed. Below are set of data for both channels logs:
win_eventlog,eventlog_name=Application,host=abcd.com,level=4 record_id=34287i,event_id=105i,description="Test event" 159421080000000000
win_eventlog,eventlog_name=System,host=abcd.com,level=4 record_id=12239i,event_id=7036i,description="Test winlog event"

@anuar45
Copy link
Copy Markdown
Contributor Author

anuar45 commented Jul 8, 2020

@utsah87 It is already supported, for example to query for specific EventID you can use following config parameter xpath_query = "Event/System[EventID=999]" and so on...
reference https://docs.microsoft.com/en-us/windows/win32/wes/consuming-events

Thanks for the reply. Am able to use this correctly now.
One more question is that we are getting the timestamp for application logs but not for System logs, is this as per design or need to be fixed. Below are set of data for both channels logs:
win_eventlog,eventlog_name=Application,host=abcd.com,level=4 record_id=34287i,event_id=105i,description="Test event" 159421080000000000
win_eventlog,eventlog_name=System,host=abcd.com,level=4 record_id=12239i,event_id=7036i,description="Test winlog event"

Hi @utsah87 thanks for reporting, but I dont see such issue on my test vm
win_eventlog,eventlog_name=System,host=DESKTOP-RP3TODN,level=0 description="TestEvent526",rec ord_id=1733i,event_id=999i 1594241630000000000

@utsah87
Copy link
Copy Markdown

utsah87 commented Jul 10, 2020

@utsah87 It is already supported, for example to query for specific EventID you can use following config parameter xpath_query = "Event/System[EventID=999]" and so on...
reference https://docs.microsoft.com/en-us/windows/win32/wes/consuming-events

Thanks for the reply. Am able to use this correctly now.
One more question is that we are getting the timestamp for application logs but not for System logs, is this as per design or need to be fixed. Below are set of data for both channels logs:
win_eventlog,eventlog_name=Application,host=abcd.com,level=4 record_id=34287i,event_id=105i,description="Test event" 159421080000000000
win_eventlog,eventlog_name=System,host=abcd.com,level=4 record_id=12239i,event_id=7036i,description="Test winlog event"

Hi @utsah87 thanks for reporting, but I dont see such issue on my test vm
win_eventlog,eventlog_name=System,host=DESKTOP-RP3TODN,level=0 description="TestEvent526",rec ord_id=1733i,event_id=999i 1594241630000000000

Hi @anuar45, not sure why but i am not able to see timestamps for system channel alerts only. have tried that on diff windows servers too. To let you know i am trying it on windows server 2012 R2, standard x64 edition.

@anuar45
Copy link
Copy Markdown
Contributor Author

anuar45 commented Jul 13, 2020

@utsah87 It is already supported, for example to query for specific EventID you can use following config parameter xpath_query = "Event/System[EventID=999]" and so on...
reference https://docs.microsoft.com/en-us/windows/win32/wes/consuming-events

Thanks for the reply. Am able to use this correctly now.
One more question is that we are getting the timestamp for application logs but not for System logs, is this as per design or need to be fixed. Below are set of data for both channels logs:
win_eventlog,eventlog_name=Application,host=abcd.com,level=4 record_id=34287i,event_id=105i,description="Test event" 159421080000000000
win_eventlog,eventlog_name=System,host=abcd.com,level=4 record_id=12239i,event_id=7036i,description="Test winlog event"

Hi @utsah87 thanks for reporting, but I dont see such issue on my test vm
win_eventlog,eventlog_name=System,host=DESKTOP-RP3TODN,level=0 description="TestEvent526",rec ord_id=1733i,event_id=999i 1594241630000000000

Hi @anuar45, not sure why but i am not able to see timestamps for system channel alerts only. have tried that on diff windows servers too. To let you know i am trying it on windows server 2012 R2, standard x64 edition.

Hi @utsah87, thank you. Let me test it on windows 2012 r2.

@utsah87
Copy link
Copy Markdown

utsah87 commented Jul 20, 2020

@utsah87 It is already supported, for example to query for specific EventID you can use following config parameter xpath_query = "Event/System[EventID=999]" and so on...
reference https://docs.microsoft.com/en-us/windows/win32/wes/consuming-events

Thanks for the reply. Am able to use this correctly now.
One more question is that we are getting the timestamp for application logs but not for System logs, is this as per design or need to be fixed. Below are set of data for both channels logs:
win_eventlog,eventlog_name=Application,host=abcd.com,level=4 record_id=34287i,event_id=105i,description="Test event" 159421080000000000
win_eventlog,eventlog_name=System,host=abcd.com,level=4 record_id=12239i,event_id=7036i,description="Test winlog event"

Hi @utsah87 thanks for reporting, but I dont see such issue on my test vm
win_eventlog,eventlog_name=System,host=DESKTOP-RP3TODN,level=0 description="TestEvent526",rec ord_id=1733i,event_id=999i 1594241630000000000

Hi @anuar45, not sure why but i am not able to see timestamps for system channel alerts only. have tried that on diff windows servers too. To let you know i am trying it on windows server 2012 R2, standard x64 edition.

Hi @utsah87, thank you. Let me test it on windows 2012 r2.

Hi @anuar45 .. while retesting today the same scenario I noticed 1 thing, we are actually getting the timestamps too but then there is catch: in channel "System" , description picks the messages from param1 and param2, now what happens as descriptions clubs both the params the delimiter which comes between both is a new line charachter, therefore what we see is that param1 comes with description field, while param2 comes in a new line and has timestamp appended to it. So what needs to be fixed is to either pick on 1 param as description or to club both the params by comma ','.
image
image2

@utsah87
Copy link
Copy Markdown

utsah87 commented Jul 20, 2020

Hi @anuar45 . Apart above issue, i need to understand if we can make a xpath_query which is applicable for all the log channels i.e. if we can use eventlog_name = "*" instead of any particular channel name.

@anuar45
Copy link
Copy Markdown
Contributor Author

anuar45 commented Jul 22, 2020

Hi @anuar45 . Apart above issue, i need to understand if we can make a xpath_query which is applicable for all the log channels i.e. if we can use eventlog_name = "*" instead of any particular channel name.

Hi @utsah87, first of all thank you for you findings. I think will concatenate all log messages with space.

Regarding wildcard for eventlog_name, not sure this possible, but I will check.

@anuar45
Copy link
Copy Markdown
Contributor Author

anuar45 commented Jul 28, 2020

Hi @anuar45 . Apart above issue, i need to understand if we can make a xpath_query which is applicable for all the log channels i.e. if we can use eventlog_name = "*" instead of any particular channel name.

Hi @utsah87 . So I have added space between event data fields, now nothing should be truncated.
Regarding wildcard mask in xpath_query, I am afraid, this is not possible in win event api. But it is possible to use special query XML, where you can specify several sources https://docs.microsoft.com/en-us/windows/win32/wes/consuming-events, in this case eventlog_name field will be ignored(use triple quote for xml '''XML''')

```

Alternatively it is possible to use special XML query documented [here](https://docs.microsoft.com/en-us/windows/win32/wes/consuming-events), in this case eventlog_name field will be ignored (use triple quote for xml in telegraf config file).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please give an example for how to use this after ignoring eventlog_name

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anuar45 : Is there any possibility to have filter on the basis of Event Data or message of the event log.
Also can you please help with an example for below:
Alternatively it is possible to use special XML query documented here, in this case eventlog_name field will be ignored (use triple quote for xml in telegraf config file).

@anuar45
Copy link
Copy Markdown
Contributor Author

anuar45 commented Aug 18, 2020 via email

@simnv
Copy link
Copy Markdown
Contributor

simnv commented Aug 18, 2020

Hello!

Another example of xpath_query:

  xpath_query = '''
  <QueryList>
    <Query Id="0" Path="Security">
      <Select Path="Security">*</Select>
      <Suppress Path="Security">*[System[( (EventID &gt;= 5152 and EventID &lt;= 5158) or EventID=5379 or EventID=4672)]]</Suppress>
    </Query>
    <Query Id="1" Path="Application">
      <Select Path="Application">*[System[(Level &lt; 4)]]</Select>
      <Select Path="OpenSSH/Admin">*[System[(Level &lt; 4)]]</Select>
      <Select Path="Windows PowerShell">*[System[(Level &lt; 4)]]</Select>
      <Select Path="Key Management Service">*[System[(Level &lt; 4)]]</Select>
      <Select Path="HardwareEvents">*[System[(Level &lt; 4)]]</Select>
    </Query>
    <Query Id="2" Path="Windows PowerShell">
      <Select Path="Windows PowerShell">*[System[(Level &lt; 4)]]</Select>
    </Query>
    <Query Id="3" Path="System">
      <Select Path="System">*</Select>
    </Query>
    <Query Id="4" Path="Setup">
      <Select Path="Setup">*</Select>
    </Query>
  </QueryList>
  '''

@anuar45 I think it is expected of you to add xml query example to README.md

@danielnelson What are the prospects of merging this PR? I have some additions, but can't add them without making additional pull request.

@anuar45
Copy link
Copy Markdown
Contributor Author

anuar45 commented Aug 18, 2020

Thank you all, but I found major issue in rendering events. There is special win api function which can format event data to full event message. I will update pr shortly.

@simnv
Copy link
Copy Markdown
Contributor

simnv commented Aug 18, 2020

@anuar45 I've created PR #8000 with your commits and my changes. Your commits are preserved in that PR and will be in git blame. I think it will be more productive to go with that version. Thank you very much for the initial work!

@anuar45
Copy link
Copy Markdown
Contributor Author

anuar45 commented Aug 18, 2020

@anuar45 I've created PR #8000 with your commits and my changes. Your commits are preserved in that PR and will be in git blame. I think it will be more productive to go with that version. Thank you very much for the initial work!

Thank you @simnv, that's the best thing that could happen to this piece of code.

PS. I also started go module anuar45/winlog, if you have interest you can take it also.

@sjwang90
Copy link
Copy Markdown
Contributor

Closing this issue in favor of PR #8000. Thank you @anuar45 for all the great work so far for with this plugin and @simnv for pick it up!

@sjwang90 sjwang90 closed this Aug 21, 2020
@simnv
Copy link
Copy Markdown
Contributor

simnv commented Aug 21, 2020

PS. I also started go module anuar45/winlog, if you have interest you can take it also.

@anuar45 Looks like someone has beaten us to it: AtakanColak/gowinlog

ssoroka pushed a commit that referenced this pull request Sep 28, 2020
Co-authored-by: Anuar Serdaliyev <serdaliyev.anuar@gmail.com>
arstercz pushed a commit to arstercz/telegraf that referenced this pull request Mar 5, 2023
…8000)

Co-authored-by: Anuar Serdaliyev <serdaliyev.anuar@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants