Skip to content

EOFError during parallel read but not sequential #11449

@tupui

Description

@tupui

Describe the bug

When building SciPy, I am now getting an EOFError on macOS (could reproduce on M1 and M2) with -j>1.

I am not sure what happened, I tried a few Python versions and a few versions of the doc dependencies and I always get the parallelism error. I am thinking now that maybe an update of macOS itself could have break something??

The issue is not present on Ubuntu.

Traceback
❯ python dev.py doc -j 4
💻  ninja -C /Users/tupui/Documents/Code/scipy/build -j8
ninja: Entering directory `/Users/tupui/Documents/Code/scipy/build'
[4/4] Generating scipy/generate-version with a custom command
Build OK
💻  meson install -C build --only-changed
Installing, see meson-install.log...
Installation OK
# for testing
# @echo installed scipy 2869717 matches git version 2869717; exit 1
mkdir -p build/html build/doctrees
LANG=C /opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/bin/python -msphinx -WT --keep-going  -b html -d build/doctrees -j4  source build/html 
Running Sphinx v7.0.1
SciPy (VERSION 1.12.0.dev)
[autosummary] generating autosummary for: building/blas_lapack.rst, building/compilers_and_options.rst, building/cross_compilation.rst, building/distutils_equivalents.rst, building/index.rst, building/introspecting_a_build.rst, building/redistributable_binaries.rst, building/understanding_meson.rst, dev/api-dev/api-dev-toc.rst, dev/api-dev/nan_policy.rst, ..., tutorial/stats/discrete_zipf.rst, tutorial/stats/discrete_zipfian.rst, tutorial/stats/resampling.rst, tutorial/stats/sampling.rst, tutorial/stats/sampling_dau.rst, tutorial/stats/sampling_dgt.rst, tutorial/stats/sampling_hinv.rst, tutorial/stats/sampling_pinv.rst, tutorial/stats/sampling_srou.rst, tutorial/stats/sampling_tdr.rst
[autosummary] generating autosummary for: /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.optimize.OptimizeResult.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.optimize.RootResults.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.bsr_array.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.bsr_matrix.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.coo_array.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.coo_matrix.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.csc_array.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.csc_matrix.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.csr_array.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.csr_matrix.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.dia_array.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.dia_matrix.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.dok_array.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.dok_matrix.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.lil_array.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.sparse.lil_matrix.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.stats._result_classes.PearsonRResult.rst, /Users/tupui/Documents/Code/scipy/doc/source/reference/generated/scipy.stats._result_classes.TtestResult.rst
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from https://numpy.org/devdocs/objects.inv...
loading intersphinx inventory from https://numpy.org/neps/objects.inv...
loading intersphinx inventory from https://matplotlib.org/stable/objects.inv...
loading intersphinx inventory from https://asv.readthedocs.io/en/stable/objects.inv...
loading intersphinx inventory from https://www.statsmodels.org/stable/objects.inv...
myst v0.18.1: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions=[], disable_syntax=[], all_links_external=False, url_schemes=('http', 'https', 'mailto', 'ftp'), ref_domains=None, highlight_code_blocks=True, number_code_blocks=[], title_to_header=False, heading_anchors=None, heading_slug_func=None, footnote_transition=True, words_per_minute=200, sub_delimiters=('{', '}'), linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area')
myst-nb v0.17.2: NbParserConfig(custom_formats={}, metadata_key='mystnb', cell_metadata_key='mystnb', kernel_rgx_aliases={}, execution_mode='auto', execution_cache_path='', execution_excludepatterns=(), execution_timeout=30, execution_in_temp=False, execution_allow_errors=False, execution_raise_on_error=False, execution_show_tb=False, merge_streams=False, render_plugin='default', remove_code_source=False, remove_code_outputs=False, code_prompt_show='Show code cell {type}', code_prompt_hide='Hide code cell {type}', number_source_lines=False, output_stderr='show', render_text_lexer='myst-ansi', render_error_lexer='ipythontb', render_image_options={}, render_figure_options={}, render_markdown_format='commonmark', output_folder='build', append_css=True, metadata_to_fm=False)
Using jupyter-cache at: /Users/tupui/Documents/Code/scipy/doc/build/.jupyter_cache
building [mo]: targets for 0 po files that are out of date
writing output... 
building [html]: targets for 4430 source files that are out of date
updating environment: [new config] 4430 added, 0 changed, 0 removed
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
<string>:46: DeprecationWarning: invalid escape sequence '\,'
<string>:46: DeprecationWarning: invalid escape sequence '\,'
<string>:50: DeprecationWarning: invalid escape sequence '\d'
<string>:46: DeprecationWarning: invalid escape sequence '\l'

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/site-packages/sphinx/cmd/build.py", line 285, in build_main
    app.build(args.force_all, args.filenames)
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/site-packages/sphinx/application.py", line 351, in build
    self.builder.build_update()
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 294, in build_update
    self.build(to_build,
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 311, in build
    updated_docnames = set(self.read())
                           ^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 416, in read
    self._read_parallel(docnames, nproc=self.app.parallel)
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 472, in _read_parallel
    tasks.join()
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/site-packages/sphinx/util/parallel.py", line 99, in join
    if not self._join_one():
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/site-packages/sphinx/util/parallel.py", line 117, in _join_one
    exc, logs, result = pipe.recv()
                        ^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/multiprocessing/connection.py", line 249, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/multiprocessing/connection.py", line 413, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/multiprocessing/connection.py", line 382, in _recv
    raise EOFError
EOFError

Exception occurred:
  File "/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev-311/lib/python3.11/multiprocessing/connection.py", line 382, in _recv
    raise EOFError
EOFError
The full traceback has been saved in /var/folders/5x/5blbwvrs6l5_wct5ncsp5f_m0000gn/T/sphinx-err-cfecful5.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [html-build] Error 2

How to Reproduce

This can be hard to do... We have a few guides to build SciPy and the doc itself https://scipy.github.io/devdocs/building/index.html#building-from-source

If you do that, use conda/mamba as it's almost straightforward. And don't even try on Windows as it can be very hard to do. It "should" just be:

brew install gfortran openblas pkg-config
git clone https://github.com/scipy/scipy.git
git submodule update --init
mamba env create -f environment.yml
mamba activate scipy-dev
python dev.py build

Once you have a build, then to build the doc: python dev.py doc -j x

Environment Information

Platform:              darwin; (macOS-13.3.1-arm64-arm-64bit)
Python version:        3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:26:40) [Clang 14.0.6 ])
Python implementation: CPython
Sphinx version:        7.0.1
Docutils version:      0.19
Jinja2 version:        3.1.2
Pygments version:      2.14.0

Sphinx extensions

extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.autosummary',
    'sphinx.ext.coverage',
    'sphinx.ext.mathjax',
    'sphinx.ext.intersphinx',
    'numpydoc',
    'sphinx_design',
    'scipyoptdoc',
    'doi_role',
    'matplotlib.sphinxext.plot_directive',
    'myst_nb',
]

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions