Fixpoint

Background

The Real Bitcoin's build system for some years has consisted at the top level of a number of GNU Makefiles and a thing called "Rotor", building on an earlier "Stator". According to its 2015 introduction by Stanislav Datskovskiy, it served to compile the "bitcoind" executable deterministically and with full static linking, given a reasonable starting environment. The key to this magic was "buildroot", essentially a miniature, non-self-hosting Linux distribution designed for cross-compiling embedded systems.

The determinism came from capturing a full set of dependencies all the way down to the compiler. This came at the cost of adding considerable complexity to the process, as what was formerly a mere application took on all the potential problems involved in bootstrapping an operating system from an unpredictable environment, in addition to the already intricate build systems of the required libraries Boost, Berkeley DB (BDB) and OpenSSL. In an early sign of trouble, Michael Trinque found it wouldn't build BDB on his system without some CPU architecture specific configuration. In my own experience, I got it to work once, but when demonstrating to some friends on fairly similar Gentoo systems I'd built for them, it failed in multiple different ways. Ultimately I couldn't be bothered to track them all down, partly because of how unbearably slow it was: to try any change you would have to repeat the whole toolchain bootstrap.

The V cryptographic source code management system was introduced, with Bitcoin as its first user, shortly after Rotor; somehow the Rotor scripts and patches didn't end up in the V-tree proper, meaning that they in addition to the library and toolchain sources had to be rounded up in order to do offline (i.e. reliable) builds or study the code.

Finally, having already taken on the publication of a Linux distribution with similar static linking and deterministic bootstrapping goals, but going further in providing a self-sufficient system with native compilers, I had little desire to be stuck maintaining two different such beasts.

The vpatch

Thus I now present bitcoin_system_compiler.vpatch (with seals on the shelf). Building on my previous raw transactions patch, it:

• Rewrites the Makefiles almost entirely. This includes the "makefile.unix" inherited from earlier developers, greatly simplifying it and eliminating historical baggage such as dynamic linking tweaks, Ubuntu bug workarounds, linking of "libssl" and way too much "sed" magic. Additions include compiler warning flags (resulting in quite a bit of warning spew, some of which might be interesting) and building "test_bitcoin" by default. Automatic header dependency analysis is preserved.
• Removes some minor GNUisms like "tar xvfz" in order to work on BusyBox systems.
• Brings "openssl-004-musl-termios.patch" formerly found in the external rotor sources into the tree.
• Adds "boost-no-demangler.patch", discussed below.
• Removes the various "src/obj" directories and moves all compiler output to the "build" directory. (For instance, this makes it easier to "grep" or "diff" the code without tripping on binary files.)
• Avoids copying the "bitcoind" binary all around and removes the "bin" subdirectory: one place is enough.
• Corrects the oversight that a library build failure would be ignored on a second "make" invocation because the mere extracted directories were used as the targets for dependency calculation.
• Fixes parallel "make" by forcibly serializing the recursion into OpenSSL's ever-so-special custom build system.
• Avoids recursing into "deps" on a top-level "make clean" so that dependency tarballs won't need to be re-downloaded. (Ultimately these need to get cleaned up and imported directly to the tree.)
• Tweaks the BDB configuration to prevent "libtool" from attempting to build unwanted shared librarires.
• Tweaks the Boost "bjam" invocation (the "compression" module gets built at install time if suppressed only at build time) and removes some "|| true" constructs that caused failures to be ignored.

It still supports the "make ONLINE=1" mode to download out-of-tree dependencies into the "deps" subdirectory from deedbot; these are reduced to the three essentials (Boost, BDB, OpenSSL).

In short, it makes both development and deployment much less painful, with a sane starting system as the price of admission.

The undemangling

Special mention is in order for the new boost-no-demangler.patch. Gales Linux uses an older branch of GCC that didn't receive fixes when the long-existing "stack clashing" (archived) family of attacks was stirred up in 2017, meaning some applications could end up vulnerable, especially those using the hazardous yet popular "alloca" or variable-length array features.

As a first step in investigating this, I enabled a number of warnings by default in the GCC configuration relating to excessive or dynamic stack frame size. While these warnings produce many false positives, they've done nicely to illuminate some suspicious code, such as binutils/libiberty/cp-demangle.c. Got that all read? Me neither... but so what, that "libiberty" is just an internal part of the toolchain, or so say the docs, right? And it's "well-known" that you don't want to feed untrusted input to the linker and friends. But wait: the GCC build system copies that code into libstdc++; from there it gets linked into C++ programs. This happens even if the program doesn't use the nonstandard "__cxa_demangle" extension it provides, by way of the default exception handler ("terminate called after throwing an instance of std::whatever").

So my gcc-4.7.4-demangler-amputation.patch, included in the Gales toolchain, removes __cxa_demangle along with the copying of cp-demangle.c, and simplifies the termination function to print raw symbol name, as it previously did anyway for the case of demangler failure (hah! - we learn that they knew their code doesn't always work). These names can be fed to "c++filt" to decode them manually if need be. It then turns out that Boost contains a couple uses of __cxa_demangle - none of them in components actually needed by bitcoind, harrumph - so the Boost patch simply cuts out the assumption that GNU compilers support it.

Stats

$ diffstat bitcoin_system_compiler.vpatch
 .gitignore                          |   21 ----
 Makefile                            |   21 ----
 bin/Makefile                        |   13 --
 bin/Manifest.sha512                 |    1
 build/Makefile                      |   97 +++++++++++++--------
 build/Makefile.rotor                |   56 ------------
 deps/Makefile                       |  166 +-----------------------------------
 deps/Manifest.sha512                |   17 ---
 deps/boost-no-demangler.patch       |   49 ++++++++++
 deps/openssl-004-musl-termios.patch |   46 +++++++++
 manifest                            |    1
 src/makefile.unix                   |  145 -------------------------------
 src/obj-test/.gitignore             |    2
 src/obj/.gitignore                  |    2
 src/obj/nogui/.gitignore            |    2
 src/obj/test/.gitignore             |    2
 verify.mk                           |    5 -
 17 files changed, 168 insertions(+), 478 deletions(-)

Looking only at the "make" code, 471 lines across seven files is reduced to 112 lines across three: quite the improvement I should think!

Future directions

Some work that sorely needs doing, as suggested earlier, is getting those external libraries under control, through some combination of pruning their code to just the necessary parts, replacing their build systems, and importing to the tree, or changing bitcoind code to eliminate them.