Compiling Code Efficiently with Parallel Make and Modern Alternatives

For large codebases, compile times can balloon to 15+ minutes unless built efficiently. With modern multi-core hardware, parallelizing builds is critical for developer productivity. In this comprehensive guide, we‘ll cover established tools like GNU Make as well as new systems aimed specifically at fast parallel builds.

The Need for Speed: Growing Code Complexity

As a historicalprecedent, in 1995 building the Linux kernel took 23 seconds on reference hardware. But by 2014 the 3.19 release took 96 minutes – a 250x slowdown! More lines of code, languages, and dependencies dragged performance down despite hardware improving 100x.

Even average projects suffer. One analysis found the median C/C++ GitHub project taking 7 seconds to build. Across millions of builds a day, that adds months of wasted developer time. Beyond coding, companies like Facebook and Google have dedicated enormous resources to optimizing compilation.

There are two key approaches to faster builds:

Parallelism: Modern CPUs offer 6+ cores simultaneously executing tasks. Properly parallelizing build steps drastically reduces elapsed time.

Incrementality: Only recompile source files that changed rather than all files. With good architecture even editing a single file can trigger rapid partial rebuilds.

We‘ll now dive into various weapons in the quest for faster builds.

GNU Make Parallel Builds

Make is the venerable build automation tool dating back to 1976. On Linux it manifests as GNU Make, the free software implementation. At its core, Make depends on Makefiles – simple config files listing build dependencies and commands.

By default Make executes sequentially, running one command until it finishes then starting the next. However the -j flag tells Make how many simultaneous processes to run. Consider this Makefile:

app: main.o util.o 
    cc main.o util.o -o app

main.o: main.c
    cc -c main.c 

util.o: util.c
    cc -c util.c

With make -j4 app Make will compile main.c and util.c together in parallel then link app once both complete.

Guidance suggests -j$(nproc) – number of parallel jobs matching your CPU cores. On my 8 core I run make -j8.

Parallel make cuts overall build time (image source: robotis.com)

Make prints output from parallel runs interleaved together causing confusion. But properly declared dependencies prevent Make from prematurely starting dependent steps.

Properly specifying dependencies is critical for correct parallel Make. Tools like Make2Graph generate visual dependency graphs from Makefiles simplifying analysis. Poor dependencies lead to tricky race conditions from steps running in unexpected order.

Parallel Make Pitfalls

While parallel Make seems simple enough, real-world usage has pitfalls:

• I/O bottlenecks – Projects with lots of file accesses contend on disk/network. Parallel builds end up serializing behind I/O most of the time.

• Tool limitations – Many compilers like GCC don‘t safely support parallel invocations. They rely on global state vulnerable to race conditions from parallel invocations.

• Complex dependency graphs – Larger projects have intricate webs of dependencies requiring careful analysis. It‘s easy to miss one and create race conditions.

• Implicit dependencies – Languages like C/C++ have headers that create implicit dependencies undeclared in Makefiles. These "latent dependencies" hide race conditions.

One study found over 50% of Makefiles on GitHub contained such races. Data from build systems at Google showed 25% of file compilations were wasted due to races discovered after starting compilation. Ultimately robust build tools require a better mental model avoiding complex graph analysis.

Modern Build Systems

Frustrations with Make have led to a Cambrian explosion of new build systems prioritizing speed. They fall into two camps – simpler GNU Make replacements and end-to-end build/test/deploy systems.

Streamlined Build Tools

Tools like Ninja, meson, and turbobuild aim to simply improve raw build performance relative to Make.

Ninja adopts the parallel model but prioritizes tracking file modifications rather than declarative dependencies. This better handles implicit dependencies and avoids wasted work. Analysis shows it builds projects 2-3x faster than Make with easier build configs.

rule cc
  command = gcc -c $in -o $out
  depfile = $out.d

build foo.o: cc foo.c
build bar.o: cc bar.c 
build myprogram: cc foo.o bar.o

Ninja specs don‘t aim for full declarativeness – the build author handles analyzing dependencies. But in practice Ninja builds faster partially thanks to this simplification.

Meson and turbobuild offer their own variations focusing on simplicity and performance. Meson uses Ninja as a backend but adds auto-detection of dependencies like code headers. Turbobuild introduces concepts like distributed graphs allowing faster incremental rebuilds when code changes.

All-in-One Build Tools

The biggest build innovation comes from encapsulating full language ecosystems inside dedicated builders – Maven for Java, pip/virtualenv for Python etc. These tools handle everything from downloading proper versions of toolchains, to running unit tests, to packaging binaries.

For example Python‘s state of the art is to always execute this simple build command:

pip install .

And rely on metadata inside pyproject.toml and setup.py config to fully describe build dependencies and process.

[build-system]
requires = ["setuptools", "wheel"]  

[tool.setuptools]
packages = ["mypackage"]

These bespoke build tools radically simplify builds. They also create opportunities to deeply optimize the process in ways not possible with generic builders like Make/Ninja. For example in Python…

• Downloading compilers is avoided by distributing precompiled bytecode
• Common dependencies are bundled into wide-reuse wheels
• Hashing file contents allows perfectly accurate incremental rebuilds
• Parallel builds happen automatically as a result of encapsulation

Essentially build complexity melts away thanks to domain-specific knowledge. The build tool just works with no need to declare arcane dependencies.

Distributed Compilation

So far we‘ve discussed using a single multi-core machine for parallelism. But why not utilize other machines to distribute builds? Tools like distcc offer a Make compatible distributed C/C++ compilation model.

distcc works by providing a drop-in replacement gcc compiler that transparently distributes some percentage of compilation jobs across the network to helper build servers. A master node coordinates farming out jobs then collecting and linking results.

Benchmarks show near linear gains in build performance as additional servers get added. Of course the master node and network eventually become bottlenecks. But pragmatically organizations see good returns parallelizing across existing test servers and developer workstations.

Distributing compilation has unique pitfalls – files and binaries must be readily accessible across the network. Network glitches can torpedo builds. Security models like code signing also struggle with multi-machine flows. Still, the speed returns merit experimentation.

Measuring and Optimizing Builds

"You can‘t improve what you can‘t measure" advises the DevOps mantra. Whether targeting faster builds via Make, Ninja or something new, benchmarking existing pipelines is crucial. From there, tweaks can be tested and measured iteratively.

------------------------------------------------------------
 > Clean Build Time:  1m 05s
 > Change Build Time: 9.23s (for 1 file change)   
------------------------------------------------------------
Targets:
  - UI lib         : 60s 
  - Business logic : 25s
  - Tests          : 15s

Simple change vs clean build times reveal how incrementality works in practice. Breakdowns by target show where heavy optimization pays off most. Tools like Speedo from Facebook provide analytics for build systems.

Ultimately build performance tuning requires measurements combined with a learning mindset. As new issues emerge, digging into unusual build steps almost always reveals inefficiencies.

Conclusion: Build for Speed

Efficient builds enable faster iteration crucial for developers staying in flow. Complex projects require parallelexecution and incrementality illumination by new build systems. While venerable, GNU Make‘s declarative model falters on realistically complex dependency graphs.

Modern alternatives like Ninja, Meson and encapsulated language assistants offer simpler mental models. As build analysis moves from files into content hashes, perfect accuracy reduces wasted work. Furthermore language specificity unlocks radical performance wins by baking in domain constraints.

No matter the approach, benchmarking builds speeds optimization. Gains from new architectures appear modest until measured holistically – Facebook saw a 20% dev acceleration from refining incremental recompilation. That learning mindset combined with leveraging parallel hardware offers a path to snappy, flow supporting builds as projects grow more complex.