@josephsl wrote:

Hi, note that Py2Exe may insist on installing Numpy, so be careful with this approach.

While Numpy is indeed installed on AppVeyor during NVDA's build process it is not a dependency of py2exe, rather it is required by robotframework-screencaplibrary

@josephsl wrote:

Hi, true, but I found that if Numpy is installed, Py2Exe will copy Numpy libs to NVDA installation. Thanks.

thankfully robotframework-screencaplibrary is installed after building launcher so Numpy is not available at this point. Thanks for letting me know about this problem. For reference if we would ever need to look into this more thoroughly it is not py2exe that requires Numpy rather there is optional support for Numpy arrays in ComTypes and that is causing py2exe to bundle these files.

Could you elaborate on:

• What happens if py2exe is on the system, but the version is older?
• • What happens if py2exe is on the system, but the version is newer?

@LeonarddeR Wrote:

Could you elaborate on:

* What happens if py2exe is on the system, but the version is older?

* * What happens if py2exe is on the system, but the version is newer?

In both these cases the version that is currently installed is uninstalled and replaced with whatever version is defined in the Sconstruct file.

So if I understand correctly, packages will be installed / replaced globally, unless the user has set up a virtual environment?
I don't really like the idea that globally installed packages could be affected, unless the user has taken a specific action of setting up a virtual environment.
I think I would prefer that we add a particular directory within our repo to the python path, and have pip install into their with --target.
This assumes though that pkg_resources will correctly detect packages this way.
If we must support virtual environments, perhaps we could somehow detect if running in a virtual environment and fall back to installing globally if so?

I think I would prefer that we add a particular directory within our repo to the python path, and have pip install into their with --target.

I like this idea.
Note that @feerrenrut did some research into the several possibilities when implementing Flake8.

@michaelDCurran wrote:

So if I understand correctly, packages will be installed / replaced globally, unless the user has set up a virtual environment?

That's true but it is not a new behaviour - the same thing happens when installing dependencies for Linting or building dev docs i.e. global packages are overwritten unless user explicitly activates a virtual env before initiating instal.

I don't really like the idea that globally installed packages could be affected, unless the user has taken a specific action of setting up a virtual environment.

Again its not a new thing though I understand why it is not ideal.

I think I would prefer that we add a particular directory within our repo to the python path, and have pip install into their with --target.
This assumes though that pkg_resources will correctly detect packages this way.
If we must support virtual environments, perhaps we could somehow detect if running in a virtual environment and fall back to installing globally if so?

To be extra clear are you suggesting to install packages at user level providing --user to piip if not using a virtual env
and install them globally that is to the currently active virtual env otherwise?

I'm proposing using -t directory, which will install them in to a specific directory of our choice. then, that directory should be added to NVDA's pythonPath. To my knowledge, that does not affect anything globally.
I think --user is used to install into a user-wide Python package directory. Though I have just noticed with devDocs, that it seems to install dependencies into a "~" directory in the NVDA repo, which I think is a bit weird, and possibly a bug?
I was not aware that scons lint automatically installed pip dependencies ... I thought like with system tests, the user had to run pip with requirements.txt manually, but I could be wrong.
In short, if we are to embrace using pip for NVDA dependencies more, I do not support scons automatically installing / replacing pip packages globally.

hi @michaelDCurran
Is it possible to use virtualenv for all NVDA builds? That might solve the above problem.
thanks

• PR introduces Flake8 errors
• Build (for testing PR)

See test results for failed build of commit 1a7d9afb88

I've started implementing the approach which has been requested by @michaelDCurran that is install packages to the specific directory inside NVDA's repo, and while it works for the subsequent runs of SCons the first one fails. I'll look into this further as soon as time allows and mark this pr as a draft for now.

It seems that this build also supports Python 3.9, which is now the latest Python stable.

@michaelDCurran This is now ready for review. As you've suggested py2exe and its dependencies are installed to the installed_packages dir in the NVDA repo without affecting packages installed globally.

If @feerrenrut or @michaelDCurran could merge this ASAP, the work done in this pr could be used to handle stuff like wx as well.

We plan to get several PR's merged into a Python 3.8 branch and get NVDA running on that branch first. We won't use it fix all issues, just those that prevent building and running, so it should be a very short lived branch.

I've created py3.8 for this. The PR can be retargeted. Since @michaelDCurran is already familiar, I'll let him review this again.

Is retargeting really necessary here. While more recent version of Py2exe is indeed needed to make NVDA run under Python 3.8 it is still backwards compatible with 3.7, so merging this PR to Alpha would give testers best chance of catching eventual regressions caused by py2exe upgrade.

Agreed with @lukaszgo1 here.

@LeonarddeR wrote:

If @feerrenrut or @michaelDCurran could merge this ASAP, the work done in this pr could be used to handle stuff like wx as well.

In #11794 you've said that installing WX using pip is not a good idea since it is a runtime dependency. What has changed?

In #11794 you've said that installing WX using pip is not a good idea since it is a runtime dependency. What has changed?

In the initial approach, py2exe was installed globally using pip. The current approach installs py2exe to NVDA's repository. I don't think installing something like wx globally is a good idea, I don't have any objections against installing wx in the way this pr introduces.

I'm currently experimenting with this idea, extending it to fetch scons, wxPython, comtypes, sphinx etc. However, I have hit a problem which is that if the user has previously installed a later version of a package globally (E.g. the user installed sphinx 3.4.3 system-wide) but NVDA specifically requires "sphinx==3.4.1", pkg_resources.working_set.resolve raises a VersionConflict error, and doesn't even run the install callback. Even if it did, or the code was changed to run pip install specifically for sphinx==3.4.1 before this point, and phinx 3.4.1 is definitely in our installed_packages directory, pkg_resources continues to fail to resolve it because it finds sphinx 3.4.3 in site-packages.
In this specific case, we could be less pedantic and just say our requirement is sphinx>=3.4.1 but there are strong arguments as to why a project like NVDA should require specific versions of things (reproducability of errors being the main one).
But this could become a problem if the user has ever installed a later version of any of our required packages where we require a specific version.
Also the reloading of pkg_resources was sometimes causing failures (especially when installing scons) as a newer version of setuptools was being pulled in from pip, but then the reload was causing multiple versions of pkg_resources to be running at the same time causing some strange exceptions.
Even if we just leave this pr for just Py2exe, this will break if the user has already installed a later version of py2exe system-wide.

After playing with this for a couple of days, I think I'm coming to the conclusion that if we do want to use pip at all, we need to fully switch to using a virtual environment, so that before any of our requirements, our Python distribution starts fresh.
I will acknowledge that up until now, I was personally adverse to going down the virtual environment rout where as others were keen. But I think this will be the cleanest way forward now.
I'll play with virtual environments a bit to better understand them and report back, assuming someone else doesn't implement something first.

I'm currently experimenting with this idea, extending it to fetch scons, wxPython, comtypes, sphinx etc. However, I have hit a problem which is that if the user has previously installed a later version of a package globally (E.g. the user installed sphinx 3.4.3 system-wide) but NVDA specifically requires "sphinx==3.4.1", pkg_resources.working_set.resolve raises a VersionConflict error, and doesn't even run the install callback. Even if it did, or the code was changed to run pip install specifically for sphinx==3.4.1 before this point, and phinx 3.4.1 is definitely in our installed_packages directory, pkg_resources continues to fail to resolve it because it finds sphinx 3.4.3 in site-packages.

That is quite interesting - I recall testing this specific scenario back in December with the initial version of this PR and it worked as expected. Have you played with creating a WorkingSet with our InstalledPackages dir as an only entry and using it explicitly rather than rely on the global working_set instance?

In this specific case, we could be less pedantic and just say our requirement is sphinx>=3.4.1 but there are strong arguments as to why a project like NVDA should require specific versions of things (reproducability of errors being the main one).

Agree 100%

Also the reloading of pkg_resources was sometimes causing failures (especially when installing scons) as a newer version of setuptools was being pulled in from pip, but then the reload was causing multiple versions of pkg_resources to be running at the same time causing some strange exceptions.

Assuming the version conflict above would be fixed and this problem affects only install of scons we can always leave it as a submodule for the time being.

I've looked briefly to the virtual envs idea and making it transparent to the user would probably be pretty difficult. The default Python implementation has a convenient method to create a new env programmatically, it even allows to run arbitrary Python code after creation which would allow us to install packages, but there is no easy possibility of activating a newly created env. An alternative would be to use virtualenv that is going to require a separate install though. Would it be acceptable to just ask users to create a virtual environment before building NVDA and explain that building without one is unsupported? Also the py2exe developer pointed in various issues against his repository that he does not provide support when py2exe is used inside virtual environment. I haven't encountered a single failure and I'm building NVDA that way for some time now, but it can certainly be an argument against continuing with this idea.

Using a separate WorkSpace object doesn't really seem to help. Workspace.resolve still seems to do strange things like wanting to fetch setuptools every time, and also even though it has already fetched sphinx 3.4.1, trying to request sphinx_rtd_theme (which depends on "sphinx") causes sphinx 3.4.3 (newest version) to be fetched, even though sphinx 3.4.1 is already there.
I think we might be pushing pkg_resources to its limits here :) and I feel that we are probably still better off working on the best way we can get users to use a virtual environment which we have total control over and does not require tinkering with workspaces.

I think there are a number of benefits to using a virtual environment, I'd like to see it investigated further. I think transparency to the user is important, but not as important as a reliable and reproducible build and runtime environment. That said, I think that we'll be able to create solutions to make it transparent.

I think there are a number of benefits to using a virtual environment, I'd like to see it investigated further. I think transparency to the user is important, but not as important as a reliable and reproducible build and runtime environment. That said, I think that we'll be able to create solutions to make it transparent.

Thinking more about it I'm not sure to what extend we can achieve transparency here. Some issues which come to mind are:

• Venvs are activated differently depending on the shell in use cmd vs PowerShell vs bash
• How we are going to detect that the environment needs to be recreated after upgrade lets say from Python 3.7 to 3.8?
• If we are going to install packages to it using the approach from this pr (I can't see other reasonable way which ensures that install is not attempted every time) we would probably hit issues when installing Scons described above by @michaelDCurran .

While I'm not a fan of the way we bundle dependencies as a git submodules it was certainly much simpler and as far as I understand time is a factor here - without more recent version of Py2exe we cannot move to Python 3.8.

Closing in favor of #12075