Have you contacted the developer about this issue?

No, I did not, I just found the bug and look around for a fix. Not being sure about how much was widespread.
Do you think it is better to ask upstream developers to fix?

I think so. This line was added for a reason, I'm afraid of blindly removing it. What was the error message? I don't think I've personally had any problems in the past.

Update: Just looked at #4154. This might be a more widespread problem. We should definitely try to fix this permanently.

FWIW, I'm tripping over this, although I might be complexificating it.

I have my standard tree of spack things, /path/to/spack/v0.0.10 and I'm using Lmod and etc... to access things in it. Now I'm working in a newly cloned Spack tree and want to run flake8. Usually I'd just spend the time to build and run flake8 from/in the new tree.

Thought I'd try and see if module load py-flake8 did anything useful, but I end up with a stack trace that ends with:

ImportError: No module named backports.configparser

If I remember well , this problem was related to the lack of init.py inside backport
When installing with pip, this was not happening as the installation path is common, while spack install any package into his own folder.
https://pypi.python.org/pypi/backports

But things could be more complicated:
nix-community/pypi2nix#80

I ' ll let other more spack and python savvy to dig.
I was just willing to produce with spack an environment where to be able to run flake8 tests

That makes sense. Although the suggestion you link to says namespace_packages=['backports'] solves the problem, whereas you commented it out. I can try to take a look. This explains why I was having problems in #4347.

I'm reading through: https://packaging.python.org/namespace_packages/

Apparently, there are 3 different ways to create a namespace package. The first method, native namespace packages, says:

This type of namespace package is defined in PEP 420 and is available in Python 3.3 and later. This is recommended if packages in your namespace only ever need to support Python 3 and installation via pip.

On a hunch, I tried installing things with Python 3 and the import test works! So that's the problem. As for the solution, I'll have to keep digging.

Actually, if you look in src/backports/__init__.py, it looks like they are using a combination of options 2 and 3. The developers probably have a better understanding of how this stuff works than I do.

Can we merge, please? I am running into flake issues.

I'll be looking into this. I aim to figure out why this change works before merging it. Let me know if I missed something and in fact the reason is precisely known.

My current understanding is that doing this may cause conflicts with other packages which add to the backports namespace. If this is the way to go, it should be generalized; there are complications with that.

In the meantime, if you spack activate py-flake8 (which writes symlinks into the python installation) then spack flake8 should work.

In more detail:

As mentioned in #4155 (comment), Python has three methods for managing namespace packages: implicit (PEP 420), pkgutils, and pkg_resources.

configparser comes with an __init__.py which conforms to the pkgutils-style method of managing namespaces. Not every python library comes with this and if it doesn't, then setuptools won't generate it.

configparser's setup.py also includes backports in the namespace_packages argument, which is part of the pkg_resources approach to namespaces. In this case py-setuptools will define a ...nspkg.pth file and explicitly remove the __init__.py from the namespace directory (I'll note that this is in contradiction to https://packaging.python.org/guides/packaging-namespace-packages/ which asserts that defining an __init__.py in backports is part of the pkg_resources approach). The docs at https://docs.python.org/2/library/site.html assert that .pth files are only searched in a specific set of directories, and not additional directories even if they appear in PYTHONPATH; therefore even though spack load -r flake8 adds Spack's py-configparser site-packages directory to PYTHONPATH it will not be found.

In the linked nix issue, the suggestion to add configparser to namespace_packages makes sense because nix creates a merged prefix. There were 3 packages in the user's example and 2 of them used the pkg_resources namespace approach. The pkgutil approach and pkg_resources approach conflict with one another, which is why defining namespace_packages for configparser makes the namespace visible there.

Takeaways:

• Because the pkg_resources and pkgutil namespace approaches conflict, enforcing the pkgutils approach here may create conflicts when globally activating with other python libs that use the pkg_resources approach
• The pkgutil and PEP 420 approaches do not conflict.
• I'm pretty sure that the pkg_resources and PEP 420 approaches do not conflict, although this contradicts https://packaging.python.org/guides/packaging-namespace-packages/. The nspkg.pth file appears to have special-case logic for python versions >= 3.5.
• The pkgutil namespace approach is the only way to support a "distributed" installation of python libraries with namespaces (e.g. using Spack's separate installation prefixes with spack load) for python versions <= 3.3 (or maybe <= 3.5)
• setuptools does not implement the pkgutils approach to namespaces, each python package must do this itself
• Packages implementing the pkgutil approach and adding to the namespace will have equivalent __init__.py files at the same path. Spack's python package would need to understand that this conflict is OK to maintain support activation/views.
• Side note: views will only support the pkg_resources namespace approach if sys.prefix is overridden for python in the view

@scheibelp Should this advice to use spack activate py-flake8 be added to the "Flake8 warnings" in https://github.com/spack/spack/blob/develop/lib/spack/docs/contribution_guide.rst? I just ran into the bug in #4154 also, and eventually stumbled into this thread after trying to kludge around the problem.

@goxberry I'll submit a PR to add this advice to the documentation.