Support consuming autoconf/make external dependencies via a Bazel rule #2792
Description
Having recently converted a mid-sized project from cmake to Bazel (https://github.com/lyft/envoy, envoyproxy/envoy#415), one of the very significant challenges was dealing with more than a dozen external dependencies (https://lyft.github.io/envoy/docs/install/requirements.html), almost all of which had no native Bazel support and were mostly autoconf/make based.
We had a requirement to consume these external dependencies in multiple ways. Some organizations (notably Lyft, the project owner) required the dependencies as prebuilts, essentially .a and .h files in a standard --prefix layout. This was also required by our CI system for performance reasons. Other developers wanted a way to have Bazel perform the build of the dependencies. To compose with other projects, allowing Envoy to be consumed as a git_repository, we wanted to avoid having the consuming project figure out its own dependencies, so we needed a self-contained Bazel build of the dependencies as well for this reason.
We settled on building the dependencies under a common recursive make both when creating the prebuilts and also ran the recursive make under a Bazel genrule for the Bazel build. Unfortunately, AFAICT, genrules need to know ahead of time all the headers and libraries in its outs, we can't just glob them. Explicitly declaring the libraries is fine, each dependency only had 1 or 2. The headers would be a nightmare though by hand, there are O(1k) of these.
We settled on performing an offline build outside of Bazel, then analyzing the build outputs to generate a .bzl with the smarts to provide a genrule for the dependencies. This .bzl then gets checked into the repository. See envoyproxy/envoy#716. This seems like a huge amount of engineering effort to allow for easy consumption of external deps that are non-Bazel based (Please let us know if there is a better way!).
I think Bazel should have an option in its git_repository for taking an external repository, performing a native build under autoconf/make (or whatever native build system it has) and discovering its generated files (perhaps with globs), that runs prior to building the dependency graph.