Skip to content

perftest: Add --with-ccache and --only-ccache options#59

Merged
egmontkob-work merged 1 commit intomasterfrom
ccache-support
Mar 18, 2022
Merged

perftest: Add --with-ccache and --only-ccache options#59
egmontkob-work merged 1 commit intomasterfrom
ccache-support

Conversation

@rbalint
Copy link
Collaborator

@rbalint rbalint commented Mar 16, 2022

to run firebuild with ccache in use or just compare it to ccache-accelerated builds.

@rbalint rbalint requested a review from egmontkob-work March 16, 2022 21:24
@rbalint
Copy link
Collaborator Author

rbalint commented Mar 17, 2022

Firebuild seems to be fasten than ccache with the default configuration on Ubuntu 20.04:

rbalint@zen:~$ projects/firebuild-infra/perftest/compare_builds.R ~/buildtimes.csv ccache-3.7.7 v0.1-256-g557104d
Time % increase from ccache-3.7.7 to v0.1-256-g557104d 
comparing 186 run pairs.

     real1              user1                 sys1         
 Min.   :-6.25558   Min.   :-10.541210   Min.   :-30.8772  
 1st Qu.:-0.81061   1st Qu.: -0.535208   1st Qu.: -3.1100  
 Median :-0.05481   Median :  0.031785   Median :  0.2188  
 Mean   : 0.04742   Mean   :  0.001764   Mean   :  0.1828  
 3rd Qu.: 0.99834   3rd Qu.:  0.624000   3rd Qu.:  3.4453  
 Max.   : 7.97939   Max.   :  9.590131   Max.   : 47.0968  
 NA's   :1          NA's   :1            NA's   :1         

                  real1     user1      sys1
 Sum. incr.: -0.2103813 -2.404746 -2.047558

     real2              user2                sys2        
 Min.   :-10.4641   Min.   :-24.15118   Min.   :-38.117  
 1st Qu.: -0.8239   1st Qu.:-10.53831   1st Qu.:-18.317  
 Median :  3.2609   Median : -6.72634   Median : -9.342  
 Mean   :  3.9054   Mean   : -5.81211   Mean   : -2.681  
 3rd Qu.:  8.2883   3rd Qu.: -0.06305   3rd Qu.:  8.400  
 Max.   : 46.0417   Max.   :  8.74103   Max.   :163.719  
 NA's   :1          NA's   :1           NA's   :1        

                real2     user2      sys2
 Sum. incr.: 4.573544 -10.18548 -1.885447

     real3            user3             sys3       
 Min.   :-98.69   Min.   :-99.52   Min.   :-95.71  
 1st Qu.:-61.47   1st Qu.:-89.09   1st Qu.:-83.31  
 Median :-43.75   Median :-82.58   Median :-74.63  
 Mean   :-40.66   Mean   :-68.35   Mean   :-56.57  
 3rd Qu.:-21.22   3rd Qu.:-58.31   3rd Qu.:-46.42  
 Max.   :214.24   Max.   : 10.33   Max.   :239.30  
 NA's   :1        NA's   :1        NA's   :1       

                 real3     user3      sys3
 Sum. incr.: -43.06206 -72.38203 -64.72306

Cache size % increase from ccache-3.7.7 to v0.1-256-g557104d
  cache.size.1     cache.size.2   
 Min.   : 361.4   Min.   : 336.7  
 1st Qu.: 499.6   1st Qu.: 499.7  
 Median : 628.3   Median : 696.8  
 Mean   :   Inf   Mean   :   Inf  
 3rd Qu.:2183.3   3rd Qu.:2193.3  
 Max.   :   Inf   Max.   :   Inf  
 NA's   :1        NA's   :1       

             cache.size.1 cache.size.2
 Sum. incr.:     1404.111     1443.635

Total time with firebuild (%) in v0.1-256-g557104d :
         first run second run
real      113.3704   38.91716
user      100.7620   17.71862
sys       131.5636   28.47504
user+sys  104.2819   18.94784

The NA is apt failing.

@rbalint
Copy link
Collaborator Author

rbalint commented Mar 17, 2022

Cache size of firebuild is 14x of ccache's, though.

debug("Building «" + name + "» without firebuild")
(status, times0) = run_build_cmd("env -C " + srcdir + " " + cmd, timeout_minutes)
debug("Building «{}» without {}".format(name, build_tool))
(status, times0) = run_build_cmd("env -C " + srcdir + " CCACHE_DISABLE=1 " + cmd, timeout_minutes)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CCACHE_DISABLE=1 has the performance penalty that the executable (gcc or friend) is still launched via a hop to the ccache binary.

When combined with firebuild (bear with me), the cost is not just an execve() hop; it's an execve() hop that we do intercept, a new scproc_query, a new firebuild cache lookup etc., it could be significant.

It would be much more accurate (and more beneficial for us) to completely avoid ccache from the game in these cases, e.g. make sure it's not found in PATH... but it's more complicated than that.


Some autoconf/cmake/meson-like systems (can't recall which) explicitly check for the presence of the ccache executable, and if found, they invoke ccache gcc instead of gcc. Therefore, omitting /usr/lib/ccache from $PATH is not enough.

(As a result, when we test firebuild, we might accidentally test the firebuild + ccache combo which doesn't make sense.)

Alas the ccache binary is installed in the global /usr/bin directory, so we cannot selectively decide to "hide" it from arbitrary build systems (at least not without root permission, which we don't want to have in inner).

I think outer should build up a container where /usr/bin/ccache is moved away to some other, hidden location (maybe /usr/lib/ccache could be used for this, assuming that update-ccache-symlinks doesn't wipe it out; if it does then another brand new directory), which we may selectively add to / remove from PATH.

(While at it, we might even want to manually compile ccache instead of using the stock Ubuntu package, to use the newest version.)

build_tool_timestamp = ""
if not args.only_ccache:
build_tool_version = os.popen("firebuild --version | awk '/Firebuild Git / {print $3}'").read().strip()
build_tool_timestamp = os.popen("date -d \"$(dpkg-parsechangelog -l /usr/share/doc/firebuild/changelog.gz -S Date)\" +%s").read().strip()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm not mistaken, you generate a buildtimes.csv where the only way to tell if it belongs to firebuild or ccache is to parse the version number and guess which binary it belongs to?

Also, it is not possible to build up a container and then perform both firebuild and ccache measurements? Or I don't completely understand everything.

I'd prefer to address #58 first, and on top of that this whole situation could be clarified a lot.

There could be independent cmdline flags whether to run a vanilla test or not, whether to run 0, 1, or 2 ccache tests, and whether to run 0, 1, 2, 3, 4, 5, ... firebuild tests (all within the same container).

The generated CSV could easily and clearly identify the build type, e.g. a column containing v, f or c for vanilla / firebuild / ccache, and another column containing the build sequence number.

I don't insist to do #58 first, but this change could become so much nicer on top of that.

perftest/inner Outdated
help="Generate a report for each firebuild-intercepted run and save them")
parser.add_argument("--with-ccache",
action="store_true",
help="Install and use ccache along with firebuild")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Install and" isn't valid for "inner". (x2)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed.

perftest/inner Outdated
help="Generate a report for each firebuild-intercepted run and save them")
parser.add_argument("--with-ccache",
action="store_true",
help="Install and use ccache along with firebuild")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"ccache along with firebuild" - I don't get it. Using or measuring the combination of the two doesn't make any sense to me. And it's not that you measure one first, and the other one next (it would require more phases than we have below). Or am I missing something?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you got the idea later. :-)

@egmontkob-work
Copy link
Contributor

egmontkob-work commented Mar 17, 2022

Ok, I think I understand now: you do want to measure the firebuild+ccache combo, where (due to ccache being on dont_intercept) we let ccache handle the caching of c/c++ code, whereas we ourselves handle the non-c/c++ steps.

Still, instead of the --with-ccache and --only-ccache options, I'd find it nicer to specify exactly what we want to test, allowing multiple combinations of them. What I'm thinking of is something like v c=2 fc=2 f=5 to run vanilla, then ccache-only twice, then firebuild+ccache twice, then firebuild-only 5 times.

It's not the particular syntax (that's subject to improvement), my main two points are:

  • specify in an absolute manner what to test, rather than in a relative manner compared to the default being "firebuild only", i.e. "with-ccache" adds ccache to it and it becomes "firebuild+ccache", or "only-ccache" means to remove the default "firebuild" and add "ccache", it's just too complicated logic for me :)
  • improved functionality (allow to test multiple setups in the same container)

I'd like to address #58 in the near future anyway (maybe right after the stat stuff if I get to do that); what do you think of doing that first?

to run firebuild with ccache in use or just compare it to ccache-accelerated builds.
@rbalint
Copy link
Collaborator Author

rbalint commented Mar 17, 2022

The main reason for the change in this PR is getting a rough estimate about how firebuild compares to ccache and seeing if it is beneficial to combine them. I don't think we will frequently use the these combinations, since firebuild's performance is more interesting for us than the comparison with ccache.

As you pointed out it is not enough to drop ccache from the PATH to disable using ccache. This is why CCACHE_DISABLE=1 is passed to the vanilla build. Firebuild does not pass the variable to the build process, thus it is not set in build 1 and 2.

The complexity of running many different (potentially repeated) tests in the same container is not worth it for now IMO.

BTW firebuild+ccache is much worse (in the 2nd build), than firebuild on its own, which means that we can safely recommend using firebuild instead of ccache :-):

rbalint@zen:~$ projects/firebuild-infra/perftest/compare_builds.R ~/buildtimes-tmp.csv v0.1-256-g557104d v0.1-256-g557104d+ccache-3.7.7
Time % increase from v0.1-256-g557104d to v0.1-256-g557104d+ccache-3.7.7 
comparing 183 run pairs.

     real1              user1                sys1         
 Min.   :-9.50135   Min.   :-11.46091   Min.   :-36.3248  
 1st Qu.:-0.75246   1st Qu.: -0.78276   1st Qu.: -3.3609  
 Median :-0.05012   Median : -0.13169   Median :  0.2639  
 Mean   : 0.15026   Mean   : -0.06664   Mean   :  0.6425  
 3rd Qu.: 0.91409   3rd Qu.:  0.67203   3rd Qu.:  4.1093  
 Max.   : 6.45161   Max.   : 11.17230   Max.   : 32.2581  

                   real1      user1       sys1
 Sum. incr.: -0.08422211 -0.1101106 -0.1571603

     real2              user2             sys2        
 Min.   :-31.3837   Min.   :-5.728   Min.   :-17.665  
 1st Qu.: -0.6391   1st Qu.: 1.817   1st Qu.:  4.181  
 Median :  1.4108   Median : 8.810   Median : 18.440  
 Mean   :  1.4636   Mean   : 8.648   Mean   : 16.611  
 3rd Qu.:  3.5039   3rd Qu.:12.176   3rd Qu.: 27.872  
 Max.   : 26.0290   Max.   :33.360   Max.   : 68.037  

                real2    user2    sys2
 Sum. incr.: 2.355445 10.10694 12.6272

     real3              user3                sys3        
 Min.   : -58.451   Min.   :   -7.434   Min.   : -20.71  
 1st Qu.:   8.538   1st Qu.:   36.489   1st Qu.:  31.16  
 Median :  27.086   Median :  166.790   Median : 186.27  
 Mean   : 175.087   Mean   :  675.185   Mean   : 256.42  
 3rd Qu.:  66.706   3rd Qu.:  540.960   3rd Qu.: 375.41  
 Max.   :7442.553   Max.   :21078.125   Max.   :1366.67  

                real3    user3     sys3
 Sum. incr.: 39.04664 103.2177 103.6825

Cache size % increase from v0.1-256-g557104d to v0.1-256-g557104d+ccache-3.7.7
  cache.size.1        cache.size.2     
 Min.   :-96.37185   Min.   :-96.5243  
 1st Qu.:-83.96475   1st Qu.:-85.0000  
 Median :-76.83138   Median :-76.8110  
 Mean   :-70.38012   Mean   :-70.3742  
 3rd Qu.:-68.79007   3rd Qu.:-68.6802  
 Max.   :  0.05168   Max.   :  0.6786  

             cache.size.1 cache.size.2
 Sum. incr.:    -77.10045    -76.25539

Total time with firebuild (%) in v0.1-256-g557104d+ccache-3.7.7 :
         first run second run
real      116.2418   54.53774
user      111.2299   37.51947
sys       149.1526   59.02466
user+sys  115.5903   39.99218

@egmontkob-work
Copy link
Contributor

Thanks for the explanations given here and during our meeting.

I'm still planning to address #58 at some point in the near future, which could provide a cleaner basis for this functionaly. But until then it's great this way, thanks!

which means that we can safely recommend using firebuild instead of ccache :-):

Yay! 🎉

@egmontkob-work egmontkob-work merged commit 5df476f into master Mar 18, 2022
@rbalint rbalint deleted the ccache-support branch March 18, 2022 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants