Skip to content

Python3 Windows and Winlogbeat updates#15831

Merged
andrewkroh merged 3 commits intoelastic:feature/python3from
andrewkroh:feature/python3-windows
Jan 27, 2020
Merged

Python3 Windows and Winlogbeat updates#15831
andrewkroh merged 3 commits intoelastic:feature/python3from
andrewkroh:feature/python3-windows

Conversation

@andrewkroh
Copy link
Copy Markdown
Member

This updates the Vagrantfile and addresses a few issues. I installed Python3 on Windows
via chocolotey and the python MSI and neither provided a python3 command so I adjusted
the build tooling to use python on Windows.

Another issue I hit was that under Vagrant with the shared filesystem, pip would not
install into the virtual environment. It only will install when moving the PYTHON_ENV
off of the shared filesystem. So when provisining the Vagrant VM it will now put
PYTHON_ENV in C:\beats-python-env. Remember this if you need need to do a completely
clean build on Windows+Vagrant.

There were some encoding issues I found while reading output files containing unicode.
So I modified the libbeat's beat.py to specify utf_8 encoding when opening output files.
I was seeing errors like:

======================================================================
ERROR: eventlogging - UTF-16 characters
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Gopath\src\github.com\elastic\beats\winlogbeat\tests\system\test_eventlogging.py", line 187, in test_utf16_characters
    evts = self.read_events(config={
  File "C:\Gopath\src\github.com\elastic\beats\winlogbeat\tests\system\winlogbeat.py", line 110, in read_events
    return self.read_output()
  File "C:\Gopath\src\github.com\elastic\beats\winlogbeat\tests\system\../../../libbeat/tests/system\beat\beat.py", line 260, in read_output
    for line in f:
  File "C:\Python38\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 277: character maps to <undefined>

This updates the Vagrantfile and addresses a few issues. I installed Python3 on Windows
via chocolotey and the python MSI and neither provided a `python3` command so I adjusted
the build tooling to use `python` on Windows.

Another issue I hit was that under Vagrant with the shared filesystem, `pip` would not
install into the virtual environment. It only will install when moving the PYTHON_ENV
off of the shared filesystem. So when provisining the Vagrant VM it will now put
PYTHON_ENV in C:\beats-python-env. Remember this if you need need to do a completely
clean build on Windows+Vagrant.

There were some encoding issues I found while reading output files containing unicode.
So I modified the libbeat's beat.py to specify `utf_8` encoding when opening output files.
I was seeing errors like:

```
======================================================================
ERROR: eventlogging - UTF-16 characters
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Gopath\src\github.com\elastic\beats\winlogbeat\tests\system\test_eventlogging.py", line 187, in test_utf16_characters
    evts = self.read_events(config={
  File "C:\Gopath\src\github.com\elastic\beats\winlogbeat\tests\system\winlogbeat.py", line 110, in read_events
    return self.read_output()
  File "C:\Gopath\src\github.com\elastic\beats\winlogbeat\tests\system\../../../libbeat/tests/system\beat\beat.py", line 260, in read_output
    for line in f:
  File "C:\Python38\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 277: character maps to <undefined>
```

I bumped the Pillow version in the pip requirements because the newer version provides a
precompiled wheel for Python 3 on Windows. This avoids having to compiling it. I bumped
the Docker versions while debugging and just left them at the latest version since nothing broke.

I tested Winlogbeat and Metricbeat's `mage pythonUnitTest` on Windows 2019 and all tests passed.
@andrewkroh andrewkroh requested a review from jsoriano January 24, 2020 16:54
@jsoriano jsoriano mentioned this pull request Jan 24, 2020
63 tasks
Copy link
Copy Markdown
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

@andrewkroh
Copy link
Copy Markdown
Member Author

Needs formatted.

fmt - goimports: Formatting Go code
mage/pytest.go

@andrewkroh andrewkroh merged commit 7a72c55 into elastic:feature/python3 Jan 27, 2020
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
* Python3 Windows and Winlogbeat updates

This updates the Vagrantfile and addresses a few issues. I installed Python3 on Windows
via chocolotey and the python MSI and neither provided a `python3` command so I adjusted
the build tooling to use `python` on Windows.

Another issue I hit was that under Vagrant with the shared filesystem, `pip` would not
install into the virtual environment. It only will install when moving the PYTHON_ENV
off of the shared filesystem. So when provisining the Vagrant VM it will now put
PYTHON_ENV in C:\beats-python-env. Remember this if you need need to do a completely
clean build on Windows+Vagrant.

There were some encoding issues I found while reading output files containing unicode.
So I modified the libbeat's beat.py to specify `utf_8` encoding when opening output files.
I was seeing errors like:

```
======================================================================
ERROR: eventlogging - UTF-16 characters
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Gopath\src\github.com\elastic\beats\winlogbeat\tests\system\test_eventlogging.py", line 187, in test_utf16_characters
    evts = self.read_events(config={
  File "C:\Gopath\src\github.com\elastic\beats\winlogbeat\tests\system\winlogbeat.py", line 110, in read_events
    return self.read_output()
  File "C:\Gopath\src\github.com\elastic\beats\winlogbeat\tests\system\../../../libbeat/tests/system\beat\beat.py", line 260, in read_output
    for line in f:
  File "C:\Python38\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 277: character maps to <undefined>
```

I bumped the Pillow version in the pip requirements because the newer version provides a
precompiled wheel for Python 3 on Windows. This avoids having to compiling it. I bumped
the Docker versions while debugging and just left them at the latest version since nothing broke.

I tested Winlogbeat and Metricbeat's `mage pythonUnitTest` on Windows 2019 and all tests passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants