Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--with-system-libmpdec / --with-system-expat still uses the vendored headers #98707

Closed
hroncok opened this issue Oct 26, 2022 · 5 comments · Fixed by #98711
Closed

--with-system-libmpdec / --with-system-expat still uses the vendored headers #98707

hroncok opened this issue Oct 26, 2022 · 5 comments · Fixed by #98711
Labels
3.11 bug and security fixes 3.12 new features, bug and security fixes build The build process and cross-build type-bug An unexpected behavior, bug, or error

Comments

@hroncok
Copy link
Contributor

hroncok commented Oct 26, 2022 •
edited by erlend-aasland

Bug report

Python 3.12.0a1 or the main branch. Run:

$ sudo dnf install mpdecimal-devel expat-devel  # or similar for your OS
...
$ git clean -fdx
$ rm -r Modules/_decimal/libmpdec
$ rm -r Modules/expat
$ ./configure --config-cache --enable-shared --with-system-libmpdec --with-system-expat
$ make
...
make: *** No rule to make target 'Modules/_decimal/libmpdec/basearith.h', needed by 'Modules/_decimal/_decimal.o'.  Stop.

$ git restore Modules/_decimal/libmpdec
$ make
...
make: *** No rule to make target 'Modules/expat/ascii.h', needed by 'Modules/pyexpat.o'.  Stop.

This works fine with Python 3.11.0. I suspect somewhere along #94474 or a similar change, this has changed behavior, but I have not tried bisecting yet.

Your environment

  • CPython versions tested on: 3.12.0a1 or the main branch
  • Operating system and architecture: Fedora Linux x86_64
@hroncok hroncok added the type-bug An unexpected behavior, bug, or error label Oct 26, 2022
@hroncok
Copy link
Contributor Author

hroncok commented Oct 26, 2022 •
edited

Bisecting with only Modules/_decimal/libmpdec removed, because I got some other failures with removed Modules/expat on the way.

EDIT: Apparently, that's what you get when you spell expat as expact.

@hroncok
Copy link
Contributor Author

hroncok commented Oct 26, 2022 •
edited

Indeed, my gut feeling was correct:

81dca70 is the first new commit
commit 81dca70
Author: Christian Heimes @tiran
Date: Thu Jul 14 09:51:49 2022 +0200

gh-93939: Build C extensions without setup.py (GH-94474)

@hroncok hroncok mentioned this issue Oct 26, 2022
Merged
@hroncok
Copy link
Contributor Author

hroncok commented Oct 26, 2022

I am really really bad at autotools, but my best guess is that this is caused by:

cpython/Makefile.pre.in

Line 2584 in de69816

MODULE_PYEXPAT_DEPS=$(LIBEXPAT_HEADERS) @LIBEXPAT_INTERNAL@

As LIBEXPAT_HEADERS contains a list that starts with Modules/expat/ascii.h.

cpython/Makefile.pre.in

Line 2589 in de69816

MODULE__DECIMAL_DEPS=$(srcdir)/Modules/_decimal/docstrings.h $(LIBMPDEC_HEADERS) @LIBMPDEC_INTERNAL@

As LIBMPDEC_HEADERS contains a list that starts with Modules/_decimal/libmpdec/basearith.h.

In both cases, the dependency needs to be made conditional on the relevant --with-system-xxx option.

@hroncok
Copy link
Contributor Author

hroncok commented Oct 26, 2022 •
edited

Testing a possible fix.

@hroncok hroncok changed the title --with-system-libmpdec / --with-system-expact still uses the vendored headers --with-system-libmpdec / --with-system-expat still uses the vendored headers Oct 26, 2022
hroncok added a commit to hroncok/cpython that referenced this issue Oct 26, 2022
@hroncok
Don't let --with-system-libmpdec / --with-system-expat use the vendor…
0017bdc 
…ed headers

This was a regression in Python 3.12.02 that prevented Fedora doing this:

    $ rm -r Modules/_decimal/libmpdec
    $ rm -r Modules/expat

Before building Python with --with-system-libmpdec --with-system-expat.

The errors were:

    make: *** No rule to make target 'Modules/_decimal/libmpdec/basearith.h', needed by 'Modules/_decimal/_decimal.o'.  Stop.
    make: *** No rule to make target 'Modules/expat/ascii.h', needed by 'Modules/pyexpat.o'.  Stop.

Now the make-dependency on the headers only exists
when --with-system-libmpdec / --with-system-expat is **not** used.

Fixes python#98707
hroncok added a commit to hroncok/cpython that referenced this issue Oct 26, 2022
@hroncok
pythongh-98707: Don't let --with-system-libmpdec / --with-system-expa…
803285d 
…t use the vendored headers

This was a regression in Python 3.12.02 that prevented Fedora doing this:

    $ rm -r Modules/_decimal/libmpdec
    $ rm -r Modules/expat

Before building Python with --with-system-libmpdec --with-system-expat.

The errors were:

    make: *** No rule to make target 'Modules/_decimal/libmpdec/basearith.h', needed by 'Modules/_decimal/_decimal.o'.  Stop.
    make: *** No rule to make target 'Modules/expat/ascii.h', needed by 'Modules/pyexpat.o'.  Stop.

Now the make-dependency on the headers only exists
when --with-system-libmpdec / --with-system-expat is **not** used.

Fixes python#98707
@hroncok hroncok mentioned this issue Oct 26, 2022
Merged
@hroncok
Copy link
Contributor Author

hroncok commented Oct 26, 2022 •
edited

FTR this was a problem on older Pythons as well, but it was not exposed. #94474 did not introduce it, but it exposed it. Another problem this will most likely fix is:

--- STDERR ---
/tmp/tmpt5xmo3n_/cpython/Objects/obmalloc.c:1449:1: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
 1449 | arena_map_get(block *p, int create)
      | ^~~~~~~~~~~~~
make: *** No rule to make target '/tmp/tmpt5xmo3n_/cpython/Modules/_decimal/libmpdec/basearith.c', needed by 'Modules/_decimal/libmpdec/basearith.o'.  Stop.
make: *** Waiting for unfinished jobs....
---- END ----
ERROR
======================================================================
ERROR: test_freeze_simple_script (test.test_tools.test_freeze.TestFreeze.test_freeze_simple_script)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.11.0/Lib/test/test_tools/test_freeze.py", line 27, in test_freeze_simple_script
    outdir, scriptfile, python = helper.prepare(script, outdir)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/Python-3.11.0/Tools/freeze/test/freeze.py", line 159, in prepare
    _run_quiet([MAKE, '-C', builddir, '-j8'])
  File "/builddir/build/BUILD/Python-3.11.0/Tools/freeze/test/freeze.py", line 25, in _run_quiet
    return subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/Python-3.11.0/Lib/subprocess.py", line 569, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/make', '-C', '/tmp/tmpt5xmo3n_/python-build', '-j8']' returned non-zero exit status 2.
----------------------------------------------------------------------
Ran 1 test in 70.677s
FAILED (errors=1)

EDIT: It does not fix it.

@erlend-aasland erlend-aasland added the build The build process and cross-build label Oct 26, 2022
hroncok added a commit to hroncok/cpython that referenced this issue Oct 27, 2022
@hroncok
pythongh-98707: Don't let --with-system-libmpdec / --with-system-expa…
a1e927c 
…t use the vendored headers

This was a regression in Python 3.12.0a2 that prevented Fedora doing this:

    $ rm -r Modules/_decimal/libmpdec
    $ rm -r Modules/expat

Before building Python with --with-system-libmpdec --with-system-expat.

The errors were:

    make: *** No rule to make target 'Modules/_decimal/libmpdec/basearith.h', needed by 'Modules/_decimal/_decimal.o'.  Stop.
    make: *** No rule to make target 'Modules/expat/ascii.h', needed by 'Modules/pyexpat.o'.  Stop.

Now the make-dependency on the headers only exists
when --with-system-libmpdec / --with-system-expat is **not** used.

Fixes python#98707
hrnciar pushed a commit to fedora-python/cpython that referenced this issue Oct 27, 2022
@hroncok @erlend-aasland @hrnciar
00389: Don't let --with-system-libmpdec / --with-system-expat use the…
3f3d7c1 
… vendored headers

This was a regression in Python 3.12.0a2 that prevented Fedora doing
this:

    $ rm -r Modules/_decimal/libmpdec
    $ rm -r Modules/expat

Before building Python with --with-system-libmpdec --with-system-expat.

The errors were:

    make: *** No rule to make target
'Modules/_decimal/libmpdec/basearith.h', needed by
'Modules/_decimal/_decimal.o'.  Stop.
    make: *** No rule to make target 'Modules/expat/ascii.h', needed by
'Modules/pyexpat.o'.  Stop.

Now the make-dependency on the headers only exists
when --with-system-libmpdec / --with-system-expat is **not** used.

Fixes python#98707

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
hrnciar pushed a commit to fedora-python/cpython that referenced this issue Oct 27, 2022
@hroncok @erlend-aasland @hrnciar
00389: Don't let --with-system-libmpdec / --with-system-expat use the…
b031b7b 
… vendored headers

This was a regression in Python 3.12.0a2 that prevented Fedora doing
this:

    $ rm -r Modules/_decimal/libmpdec
    $ rm -r Modules/expat

Before building Python with --with-system-libmpdec --with-system-expat.

The errors were:

    make: *** No rule to make target
'Modules/_decimal/libmpdec/basearith.h', needed by
'Modules/_decimal/_decimal.o'.  Stop.
    make: *** No rule to make target 'Modules/expat/ascii.h', needed by
'Modules/pyexpat.o'.  Stop.

Now the make-dependency on the headers only exists
when --with-system-libmpdec / --with-system-expat is **not** used.

Fixes python#98707

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
@gpshead gpshead added the 3.12 new features, bug and security fixes label Nov 7, 2022
erlend-aasland pushed a commit that referenced this issue Nov 11, 2022
@hroncok
gh-98707: configure --with-system-libmpdec and --with-system-expat no…
6abec1c 
… longer include vendored headers (#98711)
@bedevere-bot bedevere-bot mentioned this issue Nov 11, 2022
Merged
erlend-aasland pushed a commit to erlend-aasland/cpython that referenced this issue Nov 11, 2022
@hroncok @erlend-aasland
[3.11] pythongh-98707: configure --with-system-libmpdec and --with-sy…
d97137c 
…stem-expat no longer include vendored headers (pythonGH-98711).

(cherry picked from commit 6abec1c)

Co-authored-by: Miro Hrončok <miro@hroncok.cz>
@erlend-aasland erlend-aasland added the 3.11 bug and security fixes label Nov 11, 2022
ethanfurman pushed a commit to ethanfurman/cpython that referenced this issue Nov 12, 2022
@hroncok @ethanfurman
pythongh-98707: configure --with-system-libmpdec and --with-system-ex…
16bd877 
…pat no longer include vendored headers (python#98711)
erlend-aasland added a commit that referenced this issue Nov 13, 2022
@erlend-aasland @hroncok
[3.11] gh-98707: configure --with-system-libmpdec and --with-system-e…
eac1a63 
…xpat no longer include vendored headers (GH-98711) (#99391)

(cherry picked from commit 6abec1c)

Co-authored-by: Miro Hrončok <miro@hroncok.cz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.11 bug and security fixes 3.12 new features, bug and security fixes build The build process and cross-build type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
4 participants
@gpshead @tiran @hroncok @erlend-aasland