ENH: Parallel example execution in doc build#12708
Conversation
|
The rendered doc looks okay and the Pretty good considering the first 20+ minutes of the build was spent spinning up the CI and downloading data, then the last 10+ minutes was Sphinx actually rendering the docs, so now all the examples run in under 30 minutes!
And our RAM usage was never close to the 16GB limit, either:
|
|
Amazing! Shall we cap the RAM to 8 GB, then, to alert us in case some highly memory-inefficient change to the code has been made? |
Maybe at some point. But we also could consider making some of the existing examples use a bit more memory on purpose to make them better -- we've had to cut corners in a number of places (cropping raw, etc.) to keep memory usage below 1.5GB-ish for each example. Allowing some of those to be 3GB-ish might make them look better or cut down on |
* upstream/main: (252 commits) Disable the "Back to top" button in the documentation (mne-tools#12688) DOC: match_channel_orders works on Epochs and Evoked, too (mne-tools#12699) Scale points and labels in montage plot (mne-tools#12703) Add license header to mne.stats.erp (mne-tools#12712) Update license year to 2024 (mne-tools#12713) Add standardized measurement error (SME) (mne-tools#12707) ENH: Parallel example execution in doc build (mne-tools#12708) MAINT: Update PR template (mne-tools#12692) MAINT: Fix doc build (mne-tools#12706) [pre-commit.ci] pre-commit autoupdate (mne-tools#12702) Improve documentation of ylim argument through Evoked plotting function (mne-tools#12697) [pre-commit.ci] pre-commit autoupdate (mne-tools#12696) BUG: Fix bug with CSP rank="full" (mne-tools#12694) MRG: Add epochs metadata summary to HTML representation (mne-tools#12686) Correct `Epochs.apply_function` docstring (mne-tools#12691) FIX: Gracefully handle missing datetime in Eyelink File (mne-tools#12687) MAINT: Restore SciPy pre (mne-tools#12689) Enh single channel annotation (mne-tools#12669) [pre-commit.ci] pre-commit autoupdate (mne-tools#12682) Bump autofix-ci/action from 1.2 to 1.3 in the actions group (mne-tools#12681) ...
In theory this should make our doc build quite a bit faster.
Switch to a machine executor which should give us ~15GB memory instead of 8GB. This will also move to ubuntu-24.04 so might need a bit of tweaking to get packages etc. correct, we'll see!
Use the new Sphinx-gallery parallel building with 4 jobs (when running a full build) to hopefully speed up doc builds. Last build time was ~2.5h so hopefully this gets us closer to 1h. I'm optimistic based on looking at the CPU usage of the last full
mainbuild was mostly at 25% (and the peaks we can squash by settingOPENBLAS_NUM_THREADS=1from where it was at4):Directly copy output report files to the output folder rather than use
copyfiles-- not sure why we did it the other way before