perftest: Add --with-ccache and --only-ccache options#59
perftest: Add --with-ccache and --only-ccache options#59egmontkob-work merged 1 commit intomasterfrom
Conversation
|
Firebuild seems to be fasten than ccache with the default configuration on Ubuntu 20.04: The |
|
Cache size of firebuild is 14x of ccache's, though. |
| debug("Building «" + name + "» without firebuild") | ||
| (status, times0) = run_build_cmd("env -C " + srcdir + " " + cmd, timeout_minutes) | ||
| debug("Building «{}» without {}".format(name, build_tool)) | ||
| (status, times0) = run_build_cmd("env -C " + srcdir + " CCACHE_DISABLE=1 " + cmd, timeout_minutes) |
There was a problem hiding this comment.
CCACHE_DISABLE=1 has the performance penalty that the executable (gcc or friend) is still launched via a hop to the ccache binary.
When combined with firebuild (bear with me), the cost is not just an execve() hop; it's an execve() hop that we do intercept, a new scproc_query, a new firebuild cache lookup etc., it could be significant.
It would be much more accurate (and more beneficial for us) to completely avoid ccache from the game in these cases, e.g. make sure it's not found in PATH... but it's more complicated than that.
Some autoconf/cmake/meson-like systems (can't recall which) explicitly check for the presence of the ccache executable, and if found, they invoke ccache gcc instead of gcc. Therefore, omitting /usr/lib/ccache from $PATH is not enough.
(As a result, when we test firebuild, we might accidentally test the firebuild + ccache combo which doesn't make sense.)
Alas the ccache binary is installed in the global /usr/bin directory, so we cannot selectively decide to "hide" it from arbitrary build systems (at least not without root permission, which we don't want to have in inner).
I think outer should build up a container where /usr/bin/ccache is moved away to some other, hidden location (maybe /usr/lib/ccache could be used for this, assuming that update-ccache-symlinks doesn't wipe it out; if it does then another brand new directory), which we may selectively add to / remove from PATH.
(While at it, we might even want to manually compile ccache instead of using the stock Ubuntu package, to use the newest version.)
| build_tool_timestamp = "" | ||
| if not args.only_ccache: | ||
| build_tool_version = os.popen("firebuild --version | awk '/Firebuild Git / {print $3}'").read().strip() | ||
| build_tool_timestamp = os.popen("date -d \"$(dpkg-parsechangelog -l /usr/share/doc/firebuild/changelog.gz -S Date)\" +%s").read().strip() |
There was a problem hiding this comment.
If I'm not mistaken, you generate a buildtimes.csv where the only way to tell if it belongs to firebuild or ccache is to parse the version number and guess which binary it belongs to?
Also, it is not possible to build up a container and then perform both firebuild and ccache measurements? Or I don't completely understand everything.
I'd prefer to address #58 first, and on top of that this whole situation could be clarified a lot.
There could be independent cmdline flags whether to run a vanilla test or not, whether to run 0, 1, or 2 ccache tests, and whether to run 0, 1, 2, 3, 4, 5, ... firebuild tests (all within the same container).
The generated CSV could easily and clearly identify the build type, e.g. a column containing v, f or c for vanilla / firebuild / ccache, and another column containing the build sequence number.
I don't insist to do #58 first, but this change could become so much nicer on top of that.
perftest/inner
Outdated
| help="Generate a report for each firebuild-intercepted run and save them") | ||
| parser.add_argument("--with-ccache", | ||
| action="store_true", | ||
| help="Install and use ccache along with firebuild") |
There was a problem hiding this comment.
"Install and" isn't valid for "inner". (x2)
perftest/inner
Outdated
| help="Generate a report for each firebuild-intercepted run and save them") | ||
| parser.add_argument("--with-ccache", | ||
| action="store_true", | ||
| help="Install and use ccache along with firebuild") |
There was a problem hiding this comment.
"ccache along with firebuild" - I don't get it. Using or measuring the combination of the two doesn't make any sense to me. And it's not that you measure one first, and the other one next (it would require more phases than we have below). Or am I missing something?
There was a problem hiding this comment.
I see you got the idea later. :-)
|
Ok, I think I understand now: you do want to measure the firebuild+ccache combo, where (due to Still, instead of the It's not the particular syntax (that's subject to improvement), my main two points are:
I'd like to address #58 in the near future anyway (maybe right after the stat stuff if I get to do that); what do you think of doing that first? |
to run firebuild with ccache in use or just compare it to ccache-accelerated builds.
|
The main reason for the change in this PR is getting a rough estimate about how firebuild compares to ccache and seeing if it is beneficial to combine them. I don't think we will frequently use the these combinations, since firebuild's performance is more interesting for us than the comparison with ccache. As you pointed out it is not enough to drop ccache from the The complexity of running many different (potentially repeated) tests in the same container is not worth it for now IMO. BTW firebuild+ccache is much worse (in the 2nd build), than firebuild on its own, which means that we can safely recommend using firebuild instead of ccache :-): |
|
Thanks for the explanations given here and during our meeting. I'm still planning to address #58 at some point in the near future, which could provide a cleaner basis for this functionaly. But until then it's great this way, thanks!
Yay! 🎉 |
to run firebuild with ccache in use or just compare it to ccache-accelerated builds.