CI: Fix Makefiles for parallel builds #2825
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
❌ Build nut 2.8.2.2778-master failed (commit 693f88eb04 by @jimklimov) |
…s and renders once (for unmodified sources) Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
… once (for unmodified sources) Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
… recipes were hopefully rectified Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…en do "all-recursive" and no longer retry "doc" just-in-case Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
On the one hand, code builds are well-parallelized, easy to avoid extra work when re-making, and are more useful in typical development cycles. The ChangeLog HTML/PDF generation is a time-consuming hog when enabled, and with automake's "all-recursive" implementation handling SUBDIRS only sequentially, it blocks the useful part of build for a long time when we e.g. `git commit` the code changes during development. Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…fter "all" Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
… of automake SUBDIRS and how it hurts parallel builds of NUT Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…), and detail the dependencies Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…CS_NO_MAN option to simplify parallel work from top-level Makefile while allowing legacy "cd docs && make" to still do the right thing Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…es more visibly Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…ual libs Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…Windows (target-dependent) Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…)built Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…common.la and nut_version.h Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…itionals in scripts/Windows/Makefile.am (note HAVE_MINGW_RESGEN too) Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
…r quicker SUBDIRS_ALL_LIBS_LOCAL Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
… not just "all" per se In automake, "all: all-recursive" seems to be always defined, so we must be a dependency of that, otherwise we ended up building BOTH the default "all-recursive" and the optimized parallel sub-make, so they wrote into same files and crashed. Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
… if transitive Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 1, 2025
…e -j` [networkupstools#2825] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 1, 2025
…e -j` [networkupstools#2825] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 1, 2025
…e -j` [networkupstools#2825] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
This was referenced Mar 2, 2025
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 12, 2025
…otools [networkupstools#2842, networkupstools#2825] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 12, 2025
…etworkupstools#2825, networkupstools#2842] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 12, 2025
…etworkupstools#2825, networkupstools#2842] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to sshambar/nut
that referenced
this pull request
Mar 12, 2025
…kupstools#2825, networkupstools#2842] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to sshambar/nut
that referenced
this pull request
Mar 12, 2025
…hain [networkupstools#2825, networkupstools#2842] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to sshambar/nut
that referenced
this pull request
Mar 12, 2025
…ILT_SOURCES so it is made early [networkupstools#2825, networkupstools#2842] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 15, 2025
… SUBDIRS of main Makefile; avoid two definitions of "all" goal there [networkupstools#2842, networkupstools#2825] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
nut-travis
pushed a commit
that referenced
this pull request
Mar 15, 2025
Signed-off-by: Jim Klimov <jimklimov@gmail.com>
nut-travis
pushed a commit
that referenced
this pull request
Mar 15, 2025
Signed-off-by: Jim Klimov <jimklimov@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 16, 2025
Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to sshambar/nut
that referenced
this pull request
Mar 16, 2025
…orkupstools#2825] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 17, 2025
…ols#2825] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Mar 17, 2025
…ols#2825] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Apr 18, 2025
…(generate sources via nut-scanner-deps/tools only once) [networkupstools#2825] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Apr 18, 2025
…(generate sources via nut-scanner-deps/tools only once) [networkupstools#2825] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Jul 28, 2025
…nd it has work to do only once [follows up from PR networkupstools#2825] In practice we do call it as many times as there are listed dependency files, but it only generates them on the first run and quicky bails out later on. This should help avoid writing into same files in parallel, ending up with garbled or incomplete files when we want to use them. Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Jul 28, 2025
…ipe [follows up from PR networkupstools#2825] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Jul 28, 2025
…ncies [follows up from PR networkupstools#2825, highlights why issue networkupstools#2584 better be solved] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Jul 28, 2025
…nd it has work to do only once [follows up from PR networkupstools#2825] In practice we do call it as many times as there are listed dependency files, but it only generates them on the first run and quicky bails out later on. This should help avoid writing into same files in parallel, ending up with garbled or incomplete files when we want to use them. Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Jan 11, 2026
…refer to $(top_builddir)/include so partial builds also know about config.h [networkupstools#2825 etc] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
jimklimov
added a commit
to jimklimov/nut
that referenced
this pull request
Jan 11, 2026
…refer to $(top_builddir)/include so partial builds also know about config.h [networkupstools#2825 etc] Signed-off-by: Jim Klimov <jimklimov+nut@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Standard automake
SUBDIRSallow for nested projects with every directory being buildable alone as well, however the recursive builds are effectively sequential (iterating over the listed directories one by one). This is nicely deterministic for sequential builds, but not so nice for parallel builds with unbalanced workload in the directories (noted whenChangeLog.(pdf|html)takes minutes to build while the other cores are quiet).A commonly recommended solution is to collapse everything to one Makefile which is not too feasible for us.
So this PR goes another way, to explicitly define the web of known dependencies between directories (or specifically the header files and our numerous helper, mock and client libraries that can be used in other directories too). This ends up with a highly parallelisable build routine, where docs and code can consume CPU simultaneously and less wall-clock time gets spent overall. However sequential builds, as well as a dozen of less time-critical recursive goals like
checkorinstallof built artifacts, are handled by exactly same code as before (whatever automake generates).This PR also revises interaction between root, docs and docs/man Makefiles, to minimize duplicate work that might be done to build or check documentation - this is another notable time hog, so sad to do it e.g. every time we pass a
make check(sometimes several times per build).Hopefully NUT CI farm turnaround will be a bit faster now.
UPDATE: Hopefully the later commits also definitively resolve the mix of our desire to build a new set of
ChangeLog*products when git metadata changes, but to not roll rebuilding them in ordinary cycles that visit these files several times (make distchecketc.)