Which Build Tool For A Bootstrappable Project?

I have a nascent side project which is intended to participate in a bootstrap chain. This means it shouldn’t depend on too many things, that the transitive closure of its build dependencies must also be small, and at no point in the process should any build depend on an opaque binary blob.

Choices on the language side are pretty constrained. Zig is currently not a candidate (despite the language itself being rather promising), because it has removed its C++-based bootstrap in favour of keeping a WASM-based build of a previous compiler version. It’s great that their compiler output is reproducible — Zig-built-by-Zig is byte-for-byte identical with Zig-built-via-WASM — but for now, it’s not truly bootstrappable. (Andrew Kelley says he hopes someone writes a Zig compiler in C when Zig stabilises. I sincerely hope this happens.)

Rust is right out, for reasons described in the Zig article:

Use a prior build of the compiler - This is the approach taken by Rust as well as many other languages.

One big downside is losing the ability to build any commit from source without meta-complexity creeping in. For example, let’s say that you are trying to do git bisect. At some point, git checks out an older commit, but the script fails to build from source because the binary that is being used to build the compiler is now the wrong version. Sure, this can be addressed, but this introduces unwanted complexity that contributors would rather not deal with.

Additionally, building the compiler is limited by what targets prior binaries are available for. For example, if there is not a riscv64 build of the compiler available, then you can’t build from source on riscv64 hardware.

The bottom line here is that it does not adequately support the use case of being able to build any commit on any system.

As far as I can see, the best choice for writing bootstrap-related software in 2024 is still C99, with as few dependencies as possible. Any (hopefully few) necessary dependencies should also be bootstrappable, written in C99 and ideally provide pkg-config-style .pc files to describe the necessary compiler/linker flags. But at least there are several C compilers as well as several implementations of pkg-config (the FreeDesktop one, pkgconf, u-config, etc.).

Since we are compiling C, what should we use for the build system? Autotools is under scrutiny again in the wake of the xz-utils compromise, as code to trigger the payload was smuggled into the dist tarball as “autotools junk” that nobody looks at. Should bootstrappable projects still use autotools, or is there something better in 2024?

Autotools Is A Pain, But Not As Much As You Think

I have historically used automake (and the other autotools) as my go-to build environment for C projects. For simple projects (example: MudCore), it’s really not that much effort to set up, and you get a lot for free:

• The generated Makefiles support the GNU Makefile Conventions;

• The user can set a custom --prefix at configure time;

• Dependencies are tracked correctly as a side-effect of compilation (guaranteeing correct rebuilds during incremental development, while avoiding a wasteful dependency-generation pass for one-off builds); and

• The DESTDIR variable is respected at install time (important for package managers and tools like GNU Stow).

Many handwritten Makefiles fail at least one of these criteria.

That said, the autotools do have their warts, both for developers and users:

• At some point you need more than the common macros provide, and if you can’t get a useful macro from the Autoconf Archive, you’ll be writing in at least one ancient tongue: portable sh, portable Makefile, and the autoconf-specific dialect of GNU m4.

• Many of these tools are barely maintained in 2024. libtool went a long time with a new release on GNU mirrors (and in distro packages) but with the old version number on the webpage.

• The autotools are slow, particularly when setting up new projects. Adding a new source file to Makefile.am is a momentary hitch as Makefile regenerates. Adding a new dependency to configure.ac causes a re-building of the entire build system and a full ./configure run, which is a flow-breaking wait of several seconds. The -C flag to configure enables caching; this helps but then nags you to make distclean if ever you need to change flags or other config settings.

• automake hasn’t really moved past the recursive-make idiom (correctly considered harmful for around 20 years). This means you have to carefully order the SUBDIRS variable to ensure correct from-scratch builds, and running make in a subdirectory might not do the right thing. You can write non-recursive Makefile.am, and even break large Makefile.ams apart using include and the little-known %reldir% and %canon_reldir% substitutions, but it doesn’t work for all of automake’s features (e.g., texinfo support). So you have to write a half-recursive set of Makefile.am files.

• If you use libtool, sometimes your object files are actual objects, and sometimes they are little magic shell scripts. And then you need to run libtool --mode=execute gdb ./foo to get into a debugger.

• If you turn on per-target flags, automake will rename all the objects that are built for that target, and gives them pretty ugly names. This gives me an uneasy feeling that I’m not quite doing things the way it wants.

• Many “modern” features (e.g., subdir-objects, color-tests, silent-rules, parallel-tests) are off by default, for backwards-compatibility reasons. You just have to know to turn them on.

The Best Alternative Is meson And I Don’t Like It

I asked online for recommendations for a modern, portable build system for C. CMake is absolutely out: I still haven’t forgiven it for deciding that the space-separated output of pkg-config meant it was a “list”, then emitting that list as a semicolon-separated string into my build commands. CMake was a good intermediate step and brought some good ideas (interactive configuration, enforced out-of-tree builds), but I think we’ve surpassed it now.

Meson seems to be the lead recommendation, and a lot of the GTK/Gnome-verse packages seem to have switched over to it. I think it has a lot of shortcomings with regards to writing bootstrappable software, and I think these are serious concerns for software projects in general:

• meson depends on Python, which by default wants to depend on all sorts of things like Tk, sqlite, zlib, libbz2, liblzma, etc. If any of these packages develop a complicated bootstrap path, then bootstrapping any meson-based project becomes fraught.

• meson also depends on ninja, which means you have to bootstrap ninja too, using either CMake or its custom Python-based bootstrap script (props to the Ninja developers for including this). That means our C99 project needs a C++ compiler.

• meson’s design is built around a finite list of known compilers. If I want to produce software that participates in a bootstrap chain, what happens if meson falls out of fashion and new C compilers come onto the scene? meson is already unable to build with TinyCC, and there’s a three-year-old open issue to support generic compilers. “Test for features, not compilers” has been build system wisdom since the autoconf days, and was such a good idea that webdev rediscovered it after User-Agent sniffing became too much. I can’t see why meson has chosen this path.

• meson dist builds .tar.xz archives by default, and while it does support a --formats gztar option, I can’t see any way to make it the default. The GNU Guix project notes that xz is “notoriously difficult to bootstrap”.

  EDIT 2024-04-07: I’ve since learned that mescc-tools-extra has an xz decompressor, so we need not worry as much about the official lzma-utils bootstrap chain.

• subdir() calls aren’t isolated: the expected way to pass information between subdirectories seems to involve setting global variables in each subdir(). This seems to me a serious design flaw; it would have been much better to allow meson.build in a subdirectory to return a record or dict, which the parent can then inspect and share with other subdirectories. This is similar to the subdir-ordering problem that automake has with its SUBDIRS variable, but because meson considers the project as a whole, it’s not quite as bad.

• It’s awkward to avoid subdir() and write a single top-level meson.build because some functions like install_headers() will need you to manually specify deeper paths and explicit subdirectories.

• It’s not clear whether I should be running meson or ninja to achieve certain tasks. A correctly written automake project has two phases: you run ./configure to set up the build, and then you use make to drive the build, and let the generated Makefile rules rebuild the necessary bits of the build system.

• meson tries to alleviate this by detecting ninja and recommending that you run meson foo for everything. But this makes the problem worse: as with compilers, meson now has a set of hard-coded .ninja build runners that it knows about, and that has caused problems with Samurai in the past. (More on Samurai below.)

I don’t want this to just be a rant. There is a lot I genuinely like about meson’s design:

• It has only one language, and it’s a decent-enough one. Despite being imperative, many data structures are immutable or become immutable when used as an input. Example: freezing configuration_data() objects after they’ve been passed to configure_file() is a smart design decision and avoids a lot of potential “why isn’t my change showing up?” bugs.

• It sees the entire project at once, even if you use subdir(). This alone is huge, because it’s the only way to get cross-directory dependencies right in a nontrivial project.

• There are no generated files in source distributions.

• meson dist uses what’s in source control as the basis for the distribution archive, not what’s currently in the worktree. This makes it possible to automatically check for divergence between a source archive and the release tag.

• It’s easy to ask for a specific C standard and warning level. I can say, by default, “I want a pedantic C99 compiler”, which beats remembering to run ./configure -C CC='gcc -std=c99' CFLAGS='-Og -g -Warn -Wextra -pedantic' whenever I work on a fresh checkout. (That there seems to be no similar option for configuring the output of meson dist, and I don’t know why.)

• Similarly, there’s baked-in support for asan and ubsan.

• Compilers are objects in the Meson language, and so you run your configure-time checks against specific compilers. This is much cleaner than AC_LANG_PUSH/AC_LANG_POP.

• The “host” and “build” compilers can be distinguished in a cross-compilation scenario, making it possible for a single project to build tools to help the build. In automake, I’d probably put such tools in a separate package or write them in bash or perl.

• Depending on other packages is a first-class feature.

• Cross-compiling is a first-class feature. (automake also supports this.)

• Windows support is a first-class feature.

• It has a much nicer “aesthetic” than automake. This isn’t just about pretty colours in configure messages, but the fact only out-of-tree builds are supported means they can create much cleaner directory structures for per-target artefacts.

So, that’s it, then? Meson seems to have a similar number of warts, and a much more attractive feature set? Well, it’s complicated. I consider its finite list of compilers, its Python dependency, and the transitive closure of its bootstrap footprint to be pretty severe drawbacks. There is a possible answer in the muon and Samurai projects, which are C99 reimplementations of meson and ninja, respectively. Do they get us out of the woods? Yes and no:

• The authors of muon and Samurai have thought about bootstrapping, which is great. Samurai has a simple Makefile so it’ll build on nearly anything, and muon has a bootstrap.sh which builds itself and (optionally) a vendored copy of Samurai, after which you can configure and build muon into its final form.

• muon does not support everything meson does. It didn’t take me long to discover that meson supports passing a dict to the default_options kwarg of the project() function, but muon does not.

• In particular, muon dist is not a supported command.

• muon does not support cross-compilation.

• muon seems to support the idea of a “generic compiler”, which is great but means that neither meson nor muon is a strict subset of the other’s functionality.

• muon does not support hotdoc, the only documentation tool supported by meson’s standard list of modules.

Where Does That Leave Us?

I think there are two decent paths forward, if one cares about writing bootstrappable software:

1. Use Meson as the build language, but do all regular development using muon and Samurai. This ensures that the project is at least buildable and installable as part of a lean boostrap chain. Use meson to run “project administration”-type tasks, like generating documentation and dist tarballs. Remember to generate .tar.gz tarballs, so that bootstrappers don’t have to get all the way to xz-utils just to start the build.

2. Go back to automake and eat the additional warts in exchange for a guaranteed small set of dependencies. Despite everything, it still works well. Use non-recursive Makefile.am where you can (with include, %reldir%, and %canon_reldir%) and SUBDIRS where you must.

   I can think of a couple of ways to harden the build system against the sort of subversion that Jia Tan pulled:

   • For developers: stop passing --install to aclocal (i.e. in Makefile.am’s ACLOCAL_AMFLAGS). This prevents aclocal from copying the m4 files it uses into your build tree, which stops make dist from bringing them into your release tarballs. End users will still be able to build the package, but developers changing the build system will need to install autoconf-archive. That’s no problem; it’s available in most distros.

   • For distro packagers: unconditionally re-bootstrap the package (using autoreconf or similar) before building. People working on bootstrappable builds routinely do this, but I think it’s now something everyone should be doing. If you can’t bootstrap the build system, the package is nearly unmaintainable, and it reduces the likelihood of surprises sneaking into the configure script or other generated files. I don’t know what’s going on in Pythonland, but they seriously tell people to regenerate configure using a Makefile target which runs something in Docker, which sounds utterly bonkers to my ears. Requiring simple, single-command bootstrap of packages’ build systems should stop things from getting too wild.

     autoreconf-ing everything unfortunately puts more work onto distro build farms, but I see many packages in nixpkgs doing it anyway, because of the patches they apply to their packages. It’s probably a price worth paying just for the peace of mind.

Whether using Meson or autotools, it’s probably also worth thinking about building distro packages against VCS release tags instead of tarballs. xz-utils, being an autotooled package, had a lot more dark corners for Jia to hide his payload, but there’s really nothing autotools-specific about this attack. Building against VCS tags also makes it possible to detect when upstream force-pushes over a release tag, which is a thing some maintainers do.