Skip to content

Conversation

@hoffie
Copy link
Member

@hoffie hoffie commented Mar 1, 2022

Short description of changes

This PR installs Qt jom via choco and uses it in place of nmake. jom supports parallel builds and this PR enables them.

CHANGELOG: Internal: Speed up Windows build process by using jom with parallelization instead of nmake in autobuilds.

Context: Fixes an issue?

Performance

Does this change need documentation? What needs to be documented and how?

No.

Status of this Pull Request

Note: Build times do vary so take these numbers with a grain of salt.

What is missing until this pull request can be merged?

CI/Reviews

Checklist

  • I've verified that this Pull Request follows the general code principles
  • I tested my code and it does what I want
  • My code follows the style guide
  • I waited some time after this Pull Request was opened and all GitHub checks completed without errors.
  • I've filled all the content above

@hoffie hoffie added this to the Release 3.9.0 milestone Mar 1, 2022
@hoffie
Copy link
Member Author

hoffie commented Mar 2, 2022

Quick test from a branch which has the deploy_windows.ps1 changes from this PR, but does not install jom in the prepare script: https://github.com/hoffie/jamulus/runs/5387646863?check_suite_focus=true#step:9:126

(This was before @ann0see's suggestion to add an explicit echo for this case, so don't search for it ;))

  • Build logic still works even without jom installed

Copy link
Member

@ann0see ann0see left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we test this locally too?

@hoffie
Copy link
Member Author

hoffie commented Mar 2, 2022

Should we test this locally too?

I can't. I think I would be OK without explicit testing:

  • I've tried a run without jom installed (posted a comment in the same minute as yours)

  • I'd argue that only developers manually build on Windows anyway (contrary to Linux). So if there's really an issue, it'll hit someone qualified. :)

  • We're still in main 3.9.0 development phase

  • There are hints that this build logic does not kick in when building from Qt creator or the MSVC GUI, so it'll only affect people who build manually (similar to autobuild). Source: https://developex.com/blog/how-to-make-your-c-qt-project-build-10x-faster-with-4-optimizations/

    If you build with MSVC – you will internally use ‘jom’ or ‘nmake’ for which you don’t need to specify any extra parameters.
    [...]
    In cases where you work with Qt Creator – you’ll likely use ‘jom’ internally, you can see it in “Projects” -> “Build Settings” -> “Build Steps” : “Make: jom.exe …”, if there’s ‘nmake.exe’ – you can click “Details” and override it with ‘jom’, which is a free addition to Qt.
    [...]
    The Microsoft compiler handles parallelism by itself, while ‘nmake’ – doesn’t, so custom build steps won’t run in parallel unless you use ‘make’ or ‘jom’.

I'm going to add the other suggestion (/MP) as well, maybe it provides even more benefits.

@hoffie
Copy link
Member Author

hoffie commented Mar 2, 2022

I'm going to add the other suggestion (/MP) as well, maybe it provides even more benefits.

Well, I won't -- it'll only help if multiple files are compiled at once in the Makefile. This doesn't seem to be the case, at least I don't see any cl calls with multiple .cpp targets.

Copy link
Member

@ann0see ann0see left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. So approving in these terms.

@hoffie
Copy link
Member Author

hoffie commented Mar 2, 2022

In Linux I thought multiple files were compiled in parallel (at least locally) and I don’t know if you can judge via CI log.

There are two places where "parallelization" can happen:

  • By running compiler invocations in parallel (that's what make -j and jom /N do. That's not necessarily visible in build logs, yep.
  • By compiling multiple source files in a single compiler invocation. This isn't real parallelization, but it'll likely reduce overall compiler warm up times, .h parsing, etc. This should be visible in the build logs as the invocation would have to change for that. That's where /MP would be relevant. But if we don't expect benefits, I'm a bit hesistant to add it.

$JackVersion = "1.9.17"
$Msvc32Version = "win32_msvc2019"
$Msvc64Version = "win64_msvc2019_64"
$JomVersion = "1.1.2"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a documented process in place to review pinned versions from time to time?

(Don't I love coming up with unconnected issues...)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a documented process in place to review pinned versions from time to time?

Not yet, but there's at least one reminder: #2346

@hoffie hoffie merged commit 226c444 into jamulussoftware:master Mar 2, 2022
@hoffie hoffie deleted the jom branch March 2, 2022 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants