Migrate scripts to python 3 by jsoriano · Pull Request #14798 · elastic/beats

jsoriano · 2019-11-26T18:25:31Z

PR to identify and fix issues with python 3.

If you want to help:

Pick up an issue from the list of issues identified
If you have identified an issue that is not in the list, add it there
Add your name next to the item in the list (if you cannot edit this description, add a comment)
When you have a fix, open a PR, with feature/python3 as base branch
Once the issue is solved and the branch merged, check it in the list

How to use Python 3 locally:

If Python 3 is not the default in your system you may need to:

Remove existing virtual environments you have for beats (you can selectively remove build/python-env and build/ve directories, or just run make clean)
Install Python 3 in your system if it was not already installed

Issues:

Log of changes that we may consider to revert before merging:

Forced use of python3 using PYTHON_EXE env var
Don't check vendor and build directories on make check from the root directory
Move check to the test stage in Travis so failures there don't block other jobs so we can see their problems.

Log of issues identified/done:

To do when build is green, before merging:

Prepare migration guidelines for contributors working with Python 2 @jsoriano Update devguide with information about Python 3 and venv #15833
Sync with APM Server to decide when to merge @jsoriano

Fixes #5950.

This reverts commit c7b67d1.

This reverts commit 933f1af.

…sts execution

kvch · 2019-12-03T09:31:48Z

Opened small PR: #14900

System tests running in docker are using the same virtual environment as the host. If version of python in the virtual environment is not available in the guest, it will use the default in the system, that is so far Python 2.

jsoriano · 2019-12-04T10:06:51Z

jenkins, test this again to see if python 3 is already available in workers

beniwohli · 2019-12-06T13:54:38Z

Great stuff, fantastic to see some more Python 3 adoption!

Is the plan to switch over from Python 2 to Python 3, or to support both at the same time? If it's the former, there's a lot of changes in the PR that aren't really necessary (I assume those were introduced by 2to3 or futurize). Some examples:

all from __future__ imports
all from builtins imports
class SomeClass: -> class SomeClass(object) (in Python 3, all classes inherit from object, there aren't old-style classes anymore)
many of the the instances where an iterator is cast to a list, only to iterate over it, e.g.
```
 for _, deps in list(dependencies.items()):
```
```
 from future import standard_library
 standard_library.install_aliases()
```
This installs aliases in Python 2 for modules that moved around in Python 3, so not needed if you target Python 3 only

While these changes don't really hurt, it makes the diff quite a bit larger than it could be, and the code itself is not quite as readable.

If it helps, I can open a PR that removes these unnecessary changes (again, assuming you don't plan to support running the tests with Python 2 going forward).

andrewkroh · 2020-02-12T21:55:21Z

Ignore the OSX packaging job. The OSX packaging build needs to be disabled/removed until there's a more powerful worker available. It doesn't have enough memory to complete the build.

jsoriano · 2020-02-12T21:58:06Z

jenkins, test this again please

jsoriano · 2020-02-13T12:08:10Z

Opened PR with the squashed changes to prepare the backport to 7.x #16302

urso · 2020-02-15T19:49:54Z

x-pack/metricbeat did fail on travis on go vet running out of memory.

Checking beats-ci and travis-ci, it looks like each testsuite did succeed at least once.

urso · 2020-02-15T20:00:53Z

x-pack/metricbeat/module/appsearch/docker-compose.yml

    build:
      context: ./_meta
+      args:
+        APPSEARCH_VERSION: ${APPSEARCH_VERSION:-7.5.0}


The default version should be 8.0.0-SNAPSHOT, so to test against the app-search master branch.

Appsearch version is set to 7.5.0 also in master, we will review it as a follow up.

andrewkroh

LGTM. No changes needed - just pointing out my own typo.

andrewkroh · 2020-02-17T02:31:55Z

dev-tools/mage/pytest.go

+// platforms. So do verify the version with python.exe --version.
+//
+// Setting up a python virtual environment on a network drive does not work
+// well. So if this applies to your development environment set PYTHON_EXE


Suggested change

// well. So if this applies to your development environment set PYTHON_EXE

// well. So if this applies to your development environment set PYTHON_ENV

Fixed in #16364, where I am also adding a changelog entry for this.

Several changes in test and tools code to migrate to Python 3 and stop supporting Python 2. From now on, Python 3 is required to run tests and many make and mage targets. Switch over to `python3 -m venv` instead of `virtualenv`. It's the recommended way to create virtual Python environments in 3.4 and above. Remove any direct references to python to always allow a specific python binary or version to be used with the PYTHON_EXE environmet variable. Use python3 in the shebang for python scripts. python is going to be reserved for python2 in Ubuntu, and PEP394 mentions that python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3. Update documentation to reflect these changes and add information about the use of Python for beats development. Prepare the tests for checks based on the local ip, this IP can be [::1] if ipv4 is not available or 127.0.1.1 in some machines, as in the newer Travis images. Change checks based on sys.platform, as the values for linux have changed in Python 3. Several assertions in tests have been adapted to its newer versions. Some flaky tests that are more flaky with Python 3 have been fixed or skipped. Replace use of nosetests with mage wrapper in script for Windows CI workers. Mage manages their own python virtual environments, this is preferred to directly call python commands. (cherry picked from 9375fb9) Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Benjamin Wohlwend <bw@piquadrat.ch> Co-authored-by: Michael Madden <mikemadden42@users.noreply.github.com> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Since the migration to Python 3 (#14798) these scripts fail in environments not configured with unicode locales with this error. We saw this in the new jenkins pipelines, and can be reproduced at least on Linux running make check with LANG=C.

Since the migration to Python 3 (elastic#14798) these scripts fail in environments not configured with unicode locales with this error. We saw this in the new jenkins pipelines, and can be reproduced at least on Linux running make check with LANG=C. (cherry picked from commit c2f6358)

Since the migration to Python 3 (#14798) these scripts fail in environments not configured with unicode locales with this error. We saw this in the new jenkins pipelines, and can be reproduced at least on Linux running make check with LANG=C. (cherry picked from commit c2f6358)

Several changes in test and tools code to migrate to Python 3 and stop supporting Python 2. From now on, Python 3 is required to run tests and many make and mage targets. Switch over to `python3 -m venv` instead of `virtualenv`. It's the recommended way to create virtual Python environments in 3.4 and above. Remove any direct references to python to always allow a specific python binary or version to be used with the PYTHON_EXE environmet variable. Use python3 in the shebang for python scripts. python is going to be reserved for python2 in Ubuntu, and PEP394 mentions that python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3. Update documentation to reflect these changes and add information about the use of Python for beats development. Prepare the tests for checks based on the local ip, this IP can be [::1] if ipv4 is not available, or 127.0.1.1 in some machines, as in the latest Ubuntu images used in Travis. Change checks based on sys.platform, as the values for linux have changed in Python 3. Several assertions in tests have been adapted to its newer versions. Some flaky tests that are more flaky with Python 3 have been fixed or skipped. Replace use of nosetests with mage wrapper in script for Windows CI workers. Mage manages their own python virtual environments, this is preferred to directly call python commands. Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Benjamin Wohlwend <bw@piquadrat.ch> Co-authored-by: Michael Madden <mikemadden42@users.noreply.github.com> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Since the migration to Python 3 (elastic#14798) these scripts fail in environments not configured with unicode locales with this error. We saw this in the new jenkins pipelines, and can be reproduced at least on Linux running make check with LANG=C.

jsoriano added 5 commits November 26, 2019 17:47

Set PYTHON_EXE to python3

34a0df8

Remove backported library from requirements.txt

5fa5430

Run 2to3 tool on all python files

c93c97f

Don't check vendor and build folders for PEP8

f05d17e

Fix quotes conversion in notice generator

f77869d

jsoriano added review refactoring labels Nov 26, 2019

jsoriano self-assigned this Nov 26, 2019

jsoriano added the help wanted Indicates that a maintainer wants help on an issue or pull request label Nov 26, 2019

jsoriano added 2 commits November 26, 2019 21:33

Try again with the bytes to strings

ce3dcc8

Set PYTHON_EXE also in travis

73fca9d

EricFromCanada mentioned this pull request Nov 27, 2019

Formulas still using Python 2 Homebrew/homebrew-core#47050

Closed

62 tasks

jsoriano added 7 commits November 28, 2019 18:51

Set version of python for travis

933f1af

Upgrade base image in travis

c7b67d1

Revert "Upgrade base image in travis"

3f0aefd

This reverts commit c7b67d1.

Revert "Set version of python for travis"

5cc56aa

This reverts commit 933f1af.

Temporarily move check stage to test in travis so it doesn't block te…

d46ac2c

…sts execution

Typo

4d0695a

Merge remote-tracking branch 'origin/master' into feature/python3

713c75f

jsoriano added the in progress Pull request is currently in progress. label Nov 29, 2019

jsoriano added 3 commits December 2, 2019 17:49

Merge remote-tracking branch 'origin/master' into feature/python3

dd5ce17

Use future and pylint

ca3958e

Add future dependency

28ffc35

kvch and others added 2 commits December 3, 2019 11:55

Migrate cherrypick_pr to python3 (#14900)

20e1ecc

Make independent python virtualenvs in docker (#14903)

360d5e1

System tests running in docker are using the same virtual environment as the host. If version of python in the virtual environment is not available in the guest, it will use the default in the system, that is so far Python 2.

jsoriano mentioned this pull request Dec 3, 2019

Pass PYTHON_EXE when integration tests are executed from mage #14912

Merged

Pass PYTHON_EXE when integration tests are executed from mage (#14912)

21dd3d3

jsoriano removed the in progress Pull request is currently in progress. label Feb 13, 2020

jsoriano mentioned this pull request Feb 13, 2020

Backport migration to python 3 to 7.x #16302

Merged

Merge remote-tracking branch 'origin/master' into HEAD

bd6d8cf

urso approved these changes Feb 15, 2020

View reviewed changes

urso reviewed Feb 15, 2020

View reviewed changes

andrewkroh approved these changes Feb 17, 2020

View reviewed changes

jsoriano merged commit 9375fb9 into master Feb 17, 2020

jsoriano mentioned this pull request Feb 18, 2020

Cherry-pick #16376 to 7.x: Enforce utf8 when opening files in python scripts #16384

Merged

simitt mentioned this pull request Feb 18, 2020

Upgrade to Python3 elastic/apm-server#3357

Merged

4 tasks

jsoriano mentioned this pull request May 12, 2020

CI fails when new version of virtualenv is used #16242

Closed

cachedout mentioned this pull request May 26, 2020

Fix bdist during installation of Python requirements #18745

Merged

6 tasks

leehinman mentioned this pull request Jul 8, 2020

Cherry-pick #16376 to 7.x: Enforce utf8 when opening files in python scripts #19735

Closed

marc-gr mentioned this pull request Jul 14, 2020

Cherry-pick #18745 to 7.x: Fix bdist during installation of Python requirements #19905

Merged

6 tasks

This was referenced Aug 5, 2020

Remove pillow from testing requirements.txt #20407

Merged

Cherry-pick #20407 to 7.x: Remove pillow from testing requirements.txt #20444

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate scripts to python 3#14798

Migrate scripts to python 3#14798
jsoriano merged 58 commits intomasterfrom
feature/python3

jsoriano commented Nov 26, 2019 •

edited

Loading

Uh oh!

kvch commented Dec 3, 2019

Uh oh!

jsoriano commented Dec 4, 2019

Uh oh!

beniwohli commented Dec 6, 2019 •

edited

Loading

Uh oh!

andrewkroh commented Feb 12, 2020

Uh oh!

jsoriano commented Feb 12, 2020

Uh oh!

jsoriano commented Feb 13, 2020

Uh oh!

urso commented Feb 15, 2020

Uh oh!

urso Feb 15, 2020

Uh oh!

jsoriano Feb 15, 2020

Uh oh!

andrewkroh left a comment

Uh oh!

andrewkroh Feb 17, 2020

Uh oh!

jsoriano Feb 17, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

	// well. So if this applies to your development environment set PYTHON_EXE
	// well. So if this applies to your development environment set PYTHON_ENV

Conversation

jsoriano commented Nov 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kvch commented Dec 3, 2019

Uh oh!

jsoriano commented Dec 4, 2019

Uh oh!

beniwohli commented Dec 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewkroh commented Feb 12, 2020

Uh oh!

jsoriano commented Feb 12, 2020

Uh oh!

jsoriano commented Feb 13, 2020

Uh oh!

urso commented Feb 15, 2020

Uh oh!

urso Feb 15, 2020

Choose a reason for hiding this comment

Uh oh!

jsoriano Feb 15, 2020

Choose a reason for hiding this comment

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

andrewkroh Feb 17, 2020

Choose a reason for hiding this comment

Uh oh!

jsoriano Feb 17, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

jsoriano commented Nov 26, 2019 •

edited

Loading

beniwohli commented Dec 6, 2019 •

edited

Loading