Skip to content

Environment modifications tests fail under venvs #14568

@alalazo

Description

@alalazo

The modifications in #14496 make unit tests fail under virtual environment since apparently Python can't find standard libraries. The PR has been merged in an attempt to fix #14491, which was indeed caused by #14252 which tried to fix the absence of the python executable on RHEL 8.

Steps to reproduce the issue

$ python3.7 -m venv py37
$ . py37/bin/activate

(py37) $ ls
py37
(py37) $ git clone https://github.com/spack/spack.git
[ ... ]
(py37) $ spack test -k environment_modifications -x
[ ... ]
============================================================= 2306 tests deselected ==============================================================
============================================== 1 failed, 10 passed, 2306 deselected in 2.02 seconds ==============================================

Error Message

Error message from failing test
============================================================== test session starts ===============================================================
platform linux -- Python 3.7.6, pytest-3.2.5, py-1.4.34, pluggy-0.4.0
rootdir: /home/culpo/tmp/spack-issues/14496/spack/lib/spack/spack/test, inifile: pytest.ini
collected 2340 items                                                                                                                              

environment_modifications.py ..........F
============================================================ short test summary info =============================================================
FAIL environment_modifications.py::test_source_files

=========================================================== slowest 20 test durations ============================================================
0.21s call     environment_modifications.py::test_source_files
0.07s call     environment_modifications.py::test_inspect_path
0.03s call     environment_modifications.py::test_path_manipulation
0.01s call     environment_modifications.py::test_append_flags
0.01s call     environment_modifications.py::test_set
0.01s call     environment_modifications.py::test_set_path
0.00s setup    environment_modifications.py::test_unset
0.00s call     environment_modifications.py::test_extend
0.00s setup    environment_modifications.py::test_inspect_path
0.00s setup    environment_modifications.py::test_set
0.00s teardown environment_modifications.py::test_append_flags
0.00s setup    environment_modifications.py::test_set_path
0.00s call     environment_modifications.py::test_unset
0.00s teardown environment_modifications.py::test_filter_system_paths
0.00s setup    environment_modifications.py::test_source_files
0.00s call     environment_modifications.py::test_extra_arguments
0.00s teardown environment_modifications.py::test_extra_arguments
0.00s setup    environment_modifications.py::test_path_manipulation
0.00s setup    environment_modifications.py::test_exclude_paths_from_inspection
0.00s teardown environment_modifications.py::test_unset
==================================================================== FAILURES ====================================================================
_______________________________________________________________ test_source_files ________________________________________________________________

files_to_be_sourced = ['/home/culpo/tmp/spack-issues/14496/spack/lib/spack/spack/test/data/sourceme_first.sh', '/home/culpo/tmp/spack-issues...data/sourceme_parameters.sh', '/home/culpo/tmp/spack-issues/14496/spack/lib/spack/spack/test/data/sourceme_unicode.sh']

    @pytest.mark.usefixtures('prepare_environment_for_tests')
    def test_source_files(files_to_be_sourced):
        """Tests the construction of a list of environment modifications that are
        the result of sourcing a file.
        """
        env = EnvironmentModifications()
        for filename in files_to_be_sourced:
            if filename.endswith('sourceme_parameters.sh'):
                env.extend(EnvironmentModifications.from_sourcing_file(
                    filename, 'intel64'))
            else:
>               env.extend(EnvironmentModifications.from_sourcing_file(filename))

environment_modifications.py:262: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../util/environment.py:610: in from_sourcing_file
    environment_after_sourcing_files(file_and_args, **kwargs),
../util/environment.py:950: in environment_after_sourcing_files
    f, environment=current_environment
../util/environment.py:930: in _source_single_file
    output = shell(source_file_arguments, output=str, env=environment)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <exe: ['/bin/bash', '-c']>
args = ('source /home/culpo/tmp/spack-issues/14496/spack/lib/spack/spack/test/data/sourceme_first.sh &> /dev/null && PYTHONHO...home/culpo/tmp/spack-issues/14496/py37/bin/python3" -c "import os; import json; print(json.dumps(dict(os.environ)))"',)
kwargs = {'env': {'CLUTTER_IM_MODULE': 'xim', 'COLORTERM': 'truecolor', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'DEFAULTS_PATH': '/usr/share/gconf/ubuntu.default.path', ...}}
env_arg = {'CLUTTER_IM_MODULE': 'xim', 'COLORTERM': 'truecolor', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'DEFAULTS_PATH': '/usr/share/gconf/ubuntu.default.path', ...}
env = {'CLUTTER_IM_MODULE': 'xim', 'COLORTERM': 'truecolor', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'DEFAULTS_PATH': '/usr/share/gconf/ubuntu.default.path', ...}
fail_on_error = True, ignore_errors = (), input = None, output = <class 'str'>, error = None
streamify = <function Executable.__call__.<locals>.streamify at 0x7fa410e66680>, ostream = -1, close_ostream = False

    def __call__(self, *args, **kwargs):
        """Run this executable in a subprocess.
    
            Parameters:
                *args (str): Command-line arguments to the executable to run
    
            Keyword Arguments:
                _dump_env (dict): Dict to be set to the environment actually
                    used (envisaged for testing purposes only)
                env (dict): The environment to run the executable with
                extra_env (dict): Extra items to add to the environment
                    (neither requires nor precludes env)
                fail_on_error (bool): Raise an exception if the subprocess returns
                    an error. Default is True. The return code is available as
                    ``exe.returncode``
                ignore_errors (int or list): A list of error codes to ignore.
                    If these codes are returned, this process will not raise
                    an exception even if ``fail_on_error`` is set to ``True``
                input: Where to read stdin from
                output: Where to send stdout
                error: Where to send stderr
    
            Accepted values for input, output, and error:
    
            * python streams, e.g. open Python file objects, or ``os.devnull``
            * filenames, which will be automatically opened for writing
            * ``str``, as in the Python string type. If you set these to ``str``,
              output and error will be written to pipes and returned as a string.
              If both ``output`` and ``error`` are set to ``str``, then one string
              is returned containing output concatenated with error. Not valid
              for ``input``
    
            By default, the subprocess inherits the parent's file descriptors.
    
            """
        # Environment
        env_arg = kwargs.get('env', None)
        if env_arg is None:
            env = os.environ.copy()
            env.update(self.default_env)
        else:
            env = self.default_env.copy()
            env.update(env_arg)
        env.update(kwargs.get('extra_env', {}))
        if '_dump_env' in kwargs:
            kwargs['_dump_env'].clear()
            kwargs['_dump_env'].update(env)
    
        fail_on_error = kwargs.pop('fail_on_error', True)
        ignore_errors = kwargs.pop('ignore_errors', ())
    
        # If they just want to ignore one error code, make it a tuple.
        if isinstance(ignore_errors, int):
            ignore_errors = (ignore_errors, )
    
        input  = kwargs.pop('input',  None)
        output = kwargs.pop('output', None)
        error  = kwargs.pop('error',  None)
    
        if input is str:
            raise ValueError('Cannot use `str` as input stream.')
    
        def streamify(arg, mode):
            if isinstance(arg, string_types):
                return open(arg, mode), True
            elif arg is str:
                return subprocess.PIPE, False
            else:
                return arg, False
    
        ostream, close_ostream = streamify(output, 'w')
        estream, close_estream = streamify(error,  'w')
        istream, close_istream = streamify(input,  'r')
    
        quoted_args = [arg for arg in args if re.search(r'^"|^\'|"$|\'$', arg)]
        if quoted_args:
            tty.warn(
                "Quotes in command arguments can confuse scripts like"
                " configure.",
                "The following arguments may cause problems when executed:",
                str("\n".join(["    " + arg for arg in quoted_args])),
                "Quotes aren't needed because spack doesn't use a shell.",
                "Consider removing them")
    
        cmd = self.exe + list(args)
    
        cmd_line = "'%s'" % "' '".join(
            map(lambda arg: arg.replace("'", "'\"'\"'"), cmd))
    
        tty.debug(cmd_line)
    
        try:
            proc = subprocess.Popen(
                cmd,
                stdin=istream,
                stderr=estream,
                stdout=ostream,
                env=env)
            out, err = proc.communicate()
    
            result = None
            if output is str or error is str:
                result = ''
                if output is str:
                    result += text_type(out.decode('utf-8'))
                if error is str:
                    result += text_type(err.decode('utf-8'))
    
            rc = self.returncode = proc.returncode
            if fail_on_error and rc != 0 and (rc not in ignore_errors):
                long_msg = cmd_line
                if result:
                    # If the output is not captured in the result, it will have
                    # been stored either in the specified files (e.g. if
                    # 'output' specifies a file) or written to the parent's
                    # stdout/stderr (e.g. if 'output' is not specified)
                    long_msg += '\n' + result
    
                raise ProcessError('Command exited with status %d:' %
>                                  proc.returncode, long_msg)
E                                  spack.util.executable.ProcessError: Command exited with status -6:
E                                      '/bin/bash' '-c' 'source /home/culpo/tmp/spack-issues/14496/spack/lib/spack/spack/test/data/sourceme_first.sh &> /dev/null && PYTHONHOME="/home/culpo/tmp/spack-issues/14496/py37" "/home/culpo/tmp/spack-issues/14496/py37/bin/python3" -c "import os; import json; print(json.dumps(dict(os.environ)))"'

../util/executable.py:189: ProcessError
-------------------------------------------------------------- Captured stderr call --------------------------------------------------------------
==> Warning: Quotes in command arguments can confuse scripts like configure.
  The following arguments may cause problems when executed:
      source /home/culpo/tmp/spack-issues/14496/spack/lib/spack/spack/test/data/sourceme_first.sh &> /dev/null && PYTHONHOME="/home/culpo/tmp/spack-issues/14496/py37" "/home/culpo/tmp/spack-issues/14496/py37/bin/python3" -c "import os; import json; print(json.dumps(dict(os.environ)))"
  Quotes aren't needed because spack doesn't use a shell.
  Consider removing them
Fatal Python error: initfsencoding: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fe745a54740 (most recent call first):
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Information on your system

I'm working in a virtual environment, with:

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

$ uname -a
Linux nuvolari 5.3.0-26-generic #28~18.04.1-Ubuntu SMP Wed Dec 18 16:40:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions