Skip to content

Tox 4.3.0 breaks substitution for values from other sections #2863

@benhoyt

Description

@benhoyt

Issue

Up till Tox 4.2.8 our tox config worked fine, however as of the just-released 4.3.0 it now breaks on the substitution of values from other sections using the {[sectionname]valuename} syntax (relevant Tox docs).

I looked at the 4.3.0 changelog, and it is almost certainly caused by the recent merge of #2861, "Rewrite substitution parser". CC: @masenf @gaborbernat

A cut-down version of our tox.ini is as follows (full version here):

[tox]
skipsdist=True

[vars]
src_path = {toxinidir}/ops/
tst_path = {toxinidir}/test/
all_path = {[vars]src_path} {[vars]tst_path}

[testenv]
basepython = python3

[testenv:foo]
allowlist_externals = echo
commands =
    echo ECHOING {[vars]all_path}

With 4.2.8 and earlier, tox -e foo looked like this (note the ECHOING line):

$ pip install tox==4.2.8
...
Successfully installed tox-4.2.8

$ tox --version
4.2.8 from /home/ben/.local/lib/python3.10/site-packages/tox/__init__.py

$ tox -e foo
foo: commands[0]> echo ECHOING /home/ben/w/operator/ops/ /home/ben/w/operator/test/
ECHOING /home/ben/w/operator/ops/ /home/ben/w/operator/test/
  foo: OK (0.02=setup[0.02]+cmd[0.00] seconds)
  congratulations :) (0.04 seconds)

However, with 4.3.0, tox -e foo is not interpolating those variables but echoing them literally:

$ pip install tox==4.3.0
...
Successfully installed tox-4.2.8

$ tox --version
4.3.0 from /home/ben/.local/lib/python3.10/site-packages/tox/__init__.py

0 ~/w/operator$ tox -e foo
foo: commands[0]> echo ECHOING '{[vars]src_path}' '{[vars]tst_path}'
ECHOING {[vars]src_path} {[vars]tst_path}
  foo: OK (0.02=setup[0.02]+cmd[0.00] seconds)
  congratulations :) (0.05 seconds)

Environment

Ubuntu 22.04 amd64

pip list
$ pip list
Package                Version
---------------------- ---------------
apturl                 0.5.2
awscli                 1.22.34
bcrypt                 3.2.0
bleach                 4.1.0
blinker                1.4
botocore               1.23.34
Brlapi                 0.8.3
build                  0.8.0
cachetools             5.2.1
certifi                2020.6.20
chardet                5.1.0
chrome-gnome-shell     0.0.0
click                  8.0.3
cmarkgfm               0.8.0
colorama               0.4.6
command-not-found      0.3
cryptography           3.4.8
cupshelpers            1.0
dbus-python            1.2.18
defer                  1.0.6
dhash                  1.4
distlib                0.3.6
distro                 1.7.0
distro-info            1.1build1
docutils               0.17.1
duplicity              0.8.21
fasteners              0.14.1
filelock               3.9.0
future                 0.18.2
html5lib               1.1
httplib2               0.20.2
idna                   3.3
importlib-metadata     4.6.4
jeepney                0.7.1
jmespath               0.10.0
keyring                23.5.0
language-selector      0.1
launchpadlib           1.10.16
lazr.restfulclient     0.14.4
lazr.uri               1.0.6
lockfile               0.12.2
louis                  3.20.0
macaroonbakery         1.3.1
Mako                   1.1.3
MarkupSafe             2.0.1
monotonic              1.6
more-itertools         8.10.0
netifaces              0.11.0
oauthlib               3.2.0
olefile                0.46
packaging              23.0
paramiko               2.9.3
pep517                 0.13.0
pexpect                4.8.0
Pillow                 9.0.1
pip                    22.0.2
pkginfo                1.8.2
platformdirs           2.6.2
pluggy                 1.0.0
protobuf               3.12.4
ptyprocess             0.7.0
py                     1.10.0
pyasn1                 0.4.8
pycairo                1.20.1
pycups                 2.0.1
Pygments               2.11.2
PyGObject              3.42.1
PyJWT                  2.3.0
pymacaroons            0.13.0
PyNaCl                 1.5.0
pyparsing              2.4.7
pyproject_api          1.4.0
pyRFC3339              1.1
python-apt             2.3.0+ubuntu2.1
python-dateutil        2.8.1
python-debian          0.1.43ubuntu1
pytz                   2022.1
pyxdg                  0.27
PyYAML                 5.4.1
readme-renderer        34.0
reportlab              3.6.8
requests               2.25.1
requests-toolbelt      0.9.1
rfc3986                1.5.0
roman                  3.3
rsa                    4.8
s3transfer             0.5.0
SecretStorage          3.3.1
setuptools             59.6.0
six                    1.16.0
systemd-python         234
toml                   0.10.2
tomli                  2.0.1
tox                    4.3.0
tqdm                   4.57.0
twine                  3.8.0
ubuntu-advantage-tools 27.12
ubuntu-drivers-common  0.0.0
ufw                    0.36.1
unattended-upgrades    0.1
urllib3                1.26.5
usb-creator            0.3.7
virtualenv             20.17.1
wadllib                1.3.6
Wand                   0.6.10
webencodings           0.5.1
wheel                  0.37.1
xdg                    5
xkit                   0.0.0
zipp                   1.0.0

Output of running tox

Output of tox -rvv (when 4.3.0 is installed):

tox -rvv
$ tox -rvv
py: 70 I find interpreter for spec PythonSpec(major=3) [virtualenv/discovery/builtin.py:56]
py: 70 I proposed PythonInfo(spec=CPython3.10.6.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:63]
py: 70 D accepted PythonInfo(spec=CPython3.10.6.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:65]
py: 71 D filesystem is case-sensitive [virtualenv/info.py:24]
py: 83 I create virtual environment via CPython3Posix(dest=/home/ben/w/operator/.tox/py, clear=False, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:48]
py: 83 D create folder /home/ben/w/operator/.tox/py/bin [virtualenv/util/path/_sync.py:9]
py: 83 D create folder /home/ben/w/operator/.tox/py/lib/python3.10/site-packages [virtualenv/util/path/_sync.py:9]
py: 83 D write /home/ben/w/operator/.tox/py/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
py: 83 D 	home = /usr/bin [virtualenv/create/pyenv_cfg.py:34]
py: 83 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:34]
py: 83 D 	version_info = 3.10.6.final.0 [virtualenv/create/pyenv_cfg.py:34]
py: 83 D 	virtualenv = 20.17.1 [virtualenv/create/pyenv_cfg.py:34]
py: 83 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
py: 83 D 	base-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
py: 83 D 	base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
py: 83 D 	base-executable = /usr/bin/python3 [virtualenv/create/pyenv_cfg.py:34]
py: 83 D symlink /usr/bin/python3 to /home/ben/w/operator/.tox/py/bin/python [virtualenv/util/path/_sync.py:28]
py: 83 D create virtualenv import hook file /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/_virtualenv.pth [virtualenv/create/via_global_ref/api.py:89]
py: 84 D create /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/_virtualenv.py [virtualenv/create/via_global_ref/api.py:92]
py: 84 D ============================== target debug ============================== [virtualenv/run/session.py:50]
py: 84 D debug via /home/ben/w/operator/.tox/py/bin/python /home/ben/.local/lib/python3.10/site-packages/virtualenv/create/debug.py [virtualenv/create/creator.py:197]
py: 84 D {
  "sys": {
    "executable": "/home/ben/w/operator/.tox/py/bin/python",
    "_base_executable": "/home/ben/w/operator/.tox/py/bin/python",
    "prefix": "/home/ben/w/operator/.tox/py",
    "base_prefix": "/usr",
    "real_prefix": null,
    "exec_prefix": "/home/ben/w/operator/.tox/py",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib/python310.zip",
      "/usr/lib/python3.10",
      "/usr/lib/python3.10/lib-dynload",
      "/home/ben/w/operator/.tox/py/lib/python3.10/site-packages"
    ],
    "meta_path": [
      "",
      "",
      "",
      ""
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]",
  "makefile_filename": "/usr/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile",
  "os": "",
  "site": "",
  "datetime": "",
  "math": "",
  "json": ""
} [virtualenv/run/session.py:51]
py: 97 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ben/.local/share/virtualenv) [virtualenv/run/session.py:55]
py: 98 D install pip from wheel /home/ben/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.3.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
py: 98 D install wheel from wheel /home/ben/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/wheel-0.38.4-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
py: 99 D install setuptools from wheel /home/ben/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/setuptools-65.6.3-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
py: 99 D copy /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.3.1-py3-none-any/pip-22.3.1.virtualenv to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/pip-22.3.1.virtualenv [virtualenv/util/path/_sync.py:36]
py: 100 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.3.1-py3-none-any/pip to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/pip [virtualenv/util/path/_sync.py:36]
py: 100 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.dist-info to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/wheel-0.38.4.dist-info [virtualenv/util/path/_sync.py:36]
py: 100 D copy /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/setuptools-65.6.3.virtualenv to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/setuptools-65.6.3.virtualenv [virtualenv/util/path/_sync.py:36]
py: 101 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/setuptools to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/setuptools [virtualenv/util/path/_sync.py:36]
py: 102 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/wheel [virtualenv/util/path/_sync.py:36]
py: 105 D copy /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.virtualenv to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/wheel-0.38.4.virtualenv [virtualenv/util/path/_sync.py:36]
py: 106 D generated console scripts wheel3 wheel3.10 wheel-3.10 wheel [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
py: 127 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/_distutils_hack to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/_distutils_hack [virtualenv/util/path/_sync.py:36]
py: 127 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/setuptools-65.6.3.dist-info to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/setuptools-65.6.3.dist-info [virtualenv/util/path/_sync.py:36]
py: 128 D copy /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/distutils-precedence.pth to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/distutils-precedence.pth [virtualenv/util/path/_sync.py:36]
py: 128 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/pkg_resources to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/pkg_resources [virtualenv/util/path/_sync.py:36]
py: 131 D generated console scripts  [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
py: 139 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.3.1-py3-none-any/pip-22.3.1.dist-info to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/pip-22.3.1.dist-info [virtualenv/util/path/_sync.py:36]
py: 140 D generated console scripts pip-3.10 pip pip3.10 pip3 [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
py: 140 I add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [virtualenv/run/session.py:61]
py: 141 D write /home/ben/w/operator/.tox/py/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
py: 141 D 	home = /usr/bin [virtualenv/create/pyenv_cfg.py:34]
py: 141 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:34]
py: 141 D 	version_info = 3.10.6.final.0 [virtualenv/create/pyenv_cfg.py:34]
py: 141 D 	virtualenv = 20.17.1 [virtualenv/create/pyenv_cfg.py:34]
py: 141 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
py: 141 D 	base-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
py: 141 D 	base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
py: 141 D 	base-executable = /usr/bin/python3 [virtualenv/create/pyenv_cfg.py:34]
  py: OK (0.07 seconds)
  congratulations :) (0.10 seconds)

Metadata

Metadata

Assignees

Labels

bug:normalaffects many people or has quite an impact

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions