Fixpoint

Meet fetch-bitcoind, version 1: a small(ish) shell script which streamlines the process of gathering a complete collection of source code - V patches and other artifacts - from a web server, as required for building a Bitcoin node and Gales Bitcoin Wallet. It provides cryptographic assurance of the validity and provenance of all downloads and assembles them into a source tree ready to compile and install. Due consideration has been given both to detecting errors and to reporting them in a concise and intelligible way, while preserving full diagnostics in a log for detailed inspection when necessary.

In effect it's implemented as a kind of lightweight V presser, reducing system requirements and foregoing the generality attempted unsuccessfully by its predecessors in favor of a dead-simple interface and dependable operation on a variety of platforms, with JWRD's own Gales Linux as the principal target.

Specifically, it keeps the patch signature and hash verification parts that more-or-less worked, while dropping the dependency solver; the full sequence of patch names is encoded directly in the script. The expectation, then, is that it will be re-released many times, with each simply-numbered version acting as a single point of reference for a given set of components that work together. This serves something like the traditional release engineering practice of publishing signed, user-facing "tar" snapshots of a source tree in addition to whatever version control mechanisms may be at work behind the scenes for day-to-day development. It diverges from that practice in that it takes advantage of the differential structure of V projects, keeping the text that the signer must cover at each step down to just the new patches and the trivially-modified script. It also rules out the madness wherein those luser-facing tarballs don't match what's tracked in the development tree but are derived by some arcane process exercised only by the maintainers, quite contrary to the professed "open source" ethos yet commonplace in that world. As a side benefit, it also conserves storage and network resources for both client and server by reusing any previously fetched components.

Several projects are included in this release as subtrees:

• bitcoin - aka bitcoind, JWRD's reference node implementation, a cleanup-in-progress of Satoshi's original codebase.
• gbw-node - Gales Bitcoin Wallet, node component: companion program that extends bitcoind to index addresses and outputs as required by gbw-signer.
• gbw-signer - Gales Bitcoin Wallet, offline transaction generation and signing component.
• gscm - Gales Scheme interpreter, used by gbw-signer.
• keksum - for verifying patch output hashes; as a concession to the bootstrapping conundrum, this checking is skipped with a warning displayed if this utility is not found on the system. (Signatures are always checked.)

And as a concession to messy reality, the Big Three dependency tarballs for bitcoind are fully included in the download and verification process.

System Requirements

The following operating systems are known to work, both for running the fetcher and building the bitcoind binary. An x86_64 (AMD64) CPU and OS is required in all cases.⁽ⁱ⁾

• Gales Linux (GCC 4.7.4/musl 1.1.21)
• Gentoo Linux hardened/musl, 2017 vintage (GCC 4.9.4/musl 1.1.15)
• Gentoo Linux, 2017 vintage (GCC 4.9.4/Glibc 2.23)
• CentOS 6.10 (GCC 4.4.7/Glibc 2.12)

While we can't offer full support for any arbitrary system, it's intended to work on a variety of Linuxen, so let me know how it goes if you try something not listed.

Around 82 MB of disk space is needed to complete the download and 1-2 GB for the build (though this will be system-dependent). The machines I tested all had at least 4 GB RAM, and this is recommended for running a node, but the build can likely be done with much less.

The following less-standard commands are required:

• wget - for downloading files; either the full-blown GNU version or the basic one included in BusyBox is fine.
• gpg - the GNU Privacy Guard version 1.4.x, for verifying signatures. The subverted 2.x series will not work (the old version is often still available even where the new is installed by default; check with your package manager).
• sha512sum - because the dependency hashes as inherited from the "TRB" project are still in the legacy format (ouch).

For no-DNS systems you'll need to include an entry for fixpoint.welshcomputing.com jfxpt.com in your /etc/hosts file.

Obtaining

You will need my GPG pubkey in order to verify the script itself; from that point on, all necessary keys are included. If you don't have it:

$ wget -O - http://jfxpt.com/jfw.asc | gpg --import
$ gpg --fingerprint jfw

Verify the key fingerprint as you see fit. Next, obtain the script and its signature, and verify:

$ wget http://jfxpt.com/code/trb/fetch-bitcoind-0001.sh
$ wget http://jfxpt.com/code/trb/fetch-bitcoind-0001.sh.asc
$ gpg --verify fetch-bitcoind-0001.sh.asc

Or to preview or download by browser: key, script, signature.

Usage

You must choose a directory for the build environment; this will contain all resulting files. It is safe and indeed encouraged to retain and reuse this directory on future invocations, e.g. if the download is interrupted or when running a newer script to sync the latest patches.

Pass the chosen path to the script as its first and only argument; the directory will be created automatically if needed. Example:

$ sh fetch-bitcoind-0001.sh ~/src/bitcoin
Fetching genesis.vpatch ...
Fetching genesis.vpatch.jfw_unchecked.sig ...
Fetching bitcoin-asciilifeform.1.vpatch ...
Fetching bitcoin-asciilifeform.1.vpatch.jfw.sig ...
[snip]
Fetching gbw-signer_usrbin.vpatch ...
Fetching gbw-signer_usrbin.vpatch.jfw.sig ...
Fetching boost_1_52_0.tar.bz2 ...
Fetching db-4.8.30.tar.gz ...
Fetching openssl-1.0.1g.tar.gz ...
Pressing...
Successfully pressed to /home/user/src/bitcoin/tree !
To build bitcoin or keksum, enter the respective subdirectory and run 'make'.
(For faster results try 'make -jN' where N is the max concurrent job count.)
For gbw-node, gbw-signer, or gscm, see the README in the respective subdir.

The result should be three subdirectories: "patches" (containing V patches and seals), "deps" (containing the special-case dependency tarballs), and "tree" (containing the pressed source tree). A fourth, "fetch-bitcoind.tmp" is used for temporary files during operation; it's left over only in case of error or interruption and gets deleted on the next run. To proceed to building each component, simply follow the hints provided by the script as seen above.

If you wish to save a full snapshot of the tree - for archival, transfer to another machine or the like - you can do so at this point.

Because "tree" is always used as the press directory name, you'll need to delete any conflicting one first or else the script will halt before trying to press. (This avoids the risk of clobbering uncommitted local changes and follows the behavior of the old v.pl.)

Hard links are used to make the "deps" visible within "tree/bitcoin/deps" after pressing, so that this all works without duplication.

History

As to why this was all necessary, the problem and the approach to solving were expressed at least by June of 2020, with the pain illustrated most clearly when I found myself surprised when a recipe Robinson posted a few weeks later featured needless typing out of voluminous, error-prone and unmemorable magic strings at the CLI. The idea with our training program is to expand your mind and options; this necessarily involves at times making your head hurt, in the right places - but not wearing out your fingers while putting you to sleep!

Despite my manifesting an initial response of trying to "fix" the messenger, it was clear that something had to be done; still, deciding on how to go about it wasn't easy. Much of it was documented through some deep-running #ossasepia conversations. For readability, I've extracted them from surrounding threads and captured them here. Looking back at it now, it seems to me that the underlying difficulty with the design was in the tension between my two conflicting desires: both to optimize the specific case of deploying Bitcoin and company, and to maximize the value of the work by keeping it fully general.

Wherein I come to rant and am challenged to defend or at least clarify my position:

jfw: Working through the process for a demonstration, deploying TRB in a fresh environment - even knowing where to look for all the pieces - is definitely a pain; and my own wallet too though less so
diana_coman: lol, a more familiar pain with own wallet?
jfw: V alone or at least the one I'm using so far doesn't get you over the hill of 'resolve this project name / key / whatever to a set of files ready to press'
jfw: diana_coman: perhaps, but own wallet has fewer pieces needing to be rounded up so far
diana_coman: that sounds like a script being sorely needed
diana_coman: and a write up afterwards.

jfw: that's probably the practical approach, yeah; but needing a special unique script for this one thing grates. I need a bitcoin-bootstrapper, why? haven't I already bootstrapped a unix and V?
diana_coman: well, that route it's already "why do I need to bootstrap V, haven't I already bootstrapped a unix?" heh

diana_coman: the "why" in the end is simple really - because there's no better alternative available and no, can't wait "however long it takes" until that better alternative is "ready" etc
jfw: "that's how we ended up in this software mess in the first place" sticks in my head but I can't disagree
jfw: comes right back to balancing present vs future costs.
diana_coman: no, it's not that how we ended up in this software mess in the first place, heh⁽ⁱⁱ⁾
diana_coman: choosing what is as opposed to what one might want is certainly not "how one ends up in mess"
diana_coman: note though that choosing what has to be chosen *now* doesn't mean that it has to remain like that forever or anything of the sort
diana_coman: you can argue perhaps that making a script now lowers the pain and thus makes the problem less visible/easier to ignore - in which case fine, don't make the script and keep the pain until the root trouble gets solved, sure
diana_coman: only I don't think it's your clients who are supposed to either solve it or wait until it's solved.

jfw: I think I get it - or at least I've got as far as I'm going to from chewing on this example.
jfw: this one script won't be creating bigger problems, in itself; my thinking was that by using something you become more dependent on it + the system it's a part of. perhaps that's more a result of using it blindly or allowing that to happen than of the mere use though ("don't blame the drug")
diana_coman: jfw, that is exactly what I mean above by makes the problem less visible to some extent; but this visible always depends inevitably on who and how is looking, heh.
sonofawitch: 2020-06-17 21:37:35 (#ossasepia) diana_coman: you can argue perhaps that making a script now lowers the pain and thus makes the problem less visible/easier to ignore - in which case fine, don't make the script and keep the pain until the root trouble gets solved, sure
jfw: I'm just slower to figure out what I'm arguing then!
diana_coman: well, came to rant and ended up arguing - it might take a while to switch!

Wherein a JWRD-internal disconnect is diagnosed as missed interactions (which I take to mean both between us and with the outside world):

jfw: diana_coman, dorion: I'm curious now how one ends up in this kind of self-punishment. Is it just my lack of providing a ready-made script at work here? Of all the manual ways this one seems about the most laborious possible, both for the doing and the writing up!
jfw: (and in this case there even was a readymade script, in the very previous step, but I can see the point of being sure the V really does build itself)
jfw: (and of having all the source patches oneself)
diana_coman: jfw - eh, the shortest answer to that is "as a result of all the missed interactions", really.
diana_coman: and not only from the side of "the one".
jfw: mhm

Wherein insanities are examined, and comparing their levels is regarded dimly as a defense, with aside on what gets done in the absence of other ways to commit:

dorion: http://ossasepia.com/2020/07/01/ossasepia-logs-for-Jul-2020/#1027502 - thanks diana_coman.
sonofawitch: 2020-07-01 21:08:02 (#ossasepia) diana_coman: dorion - your gbw article is quite an interesting read actually, as it covers a lot of ground and as a side effect, all sorts of tangles and weirds-if-coming-at-them-fresh pop all the clearer into view, hm.
dorion: http://ossasepia.com/2020/07/01/ossasepia-logs-for-Jul-2020/#1027540 - hm, I could've said, open http://fixpoint.welshcomputing.com/v/v/ in your browser and download the seals and vpatches, but I can't offer that as command. it was a pain to write that, but now it's done.
sonofawitch: 2020-07-01 22:17:01 (#ossasepia) jfw: diana_coman, dorion: I'm curious now how one ends up in this kind of self-punishment. Is it just my lack of providing a ready-made script at work here? Of all the manual ways this one seems about the most laborious
dorion: or I could've eschewed the brace expansion and written wget for each item. maybe it'd be easier to read, but the reasoning was it's not too hard to parse if you know what you're reading and easier to execute.
dorion: jfw do you have a ready example of a better way ?

jfw: dorion: several, though not meeting your desire for 'just type these 100k characters and it'll work': use external browser and 'scp'; use onboard 'links'; use external 'wget -r' (because yeah, the busybox one doesn't support crawling); paste the urls into a script and run that
jfw: not to imply that a fully-working recipe is a bad goal, if recipe is what you're going for
jfw: but - and I know this cuts both ways - can you actually imagine someone typing that all in from the blog page? even if the offscreen overflow gets cleaned up
jfw: steps 19+ look good tho!
dorion: jfw thanks. yeah, I was going for more of a recipe and for sure it'd be a big pain to type out.

diana_coman: http://ossasepia.com/2020/07/01/ossasepia-logs-for-Jul-2020/#1027559 - yeah, but it's not as much the recipe's fault really, it's more of the underlying mess and lack of sane alternative.
sonofawitch: 2020-07-01 23:02:51 (#ossasepia) dorion: jfw thanks. yeah, I was going for more of a recipe and for sure it'd be a big pain to type out.

diana_coman: http://ossasepia.com/2020/07/01/ossasepia-logs-for-Jul-2020/#1027565 - you know, all those are good but the trouble made plain there is essentially that current trb requires some level of desperation/insanity/passion-in-the-religious-sense to deploy, pretty much.
sonofawitch: 2020-07-01 23:23:38 (#ossasepia) jfw: I've indeed tended to use ~/bin before so this is probably a needn't-have-been-problem you've picked up from me; basically I'm making efforts here to cut down on unnecessary optionality.

jfw: diana_coman: I'm not sure that's just in the deployment either; e.g. the still unhealed miner/node externality
diana_coman: jfw - ah, certainly; the deployment is just the surface/first-contact part, sure.
diana_coman: and a consequence in the end.
jfw: I suppose the question is whether in the end it demands less insanity and works better than what they call bank accounts these days
diana_coman: for that matter and to add layers of either sadness or laughter to it all, there isn't even any discernible purpose or intention, only side effects.
diana_coman: jfw - eh, only different insanity and one that tends to be more familiar for most; but comparing levels of insanity is not all that meaningful, you know?
diana_coman: (the above re bank accounts)
jfw: diana_coman: purpose/intention in what?
diana_coman: jfw - in that requirement of passion.
diana_coman: ie it's not that anyone set on purpose to make it painful; nevertheless (and predictably once looking at it properly), that's how it ends up.
diana_coman: anyway, in more practical considerations, there's no reason why it has to remain as insane as that forever and ever or something.
jfw: certainly.

jfw: I see what you're saying now about making it painful but don't yet have the vantage point where it's predictable.
diana_coman: well, that's the top of a whole forest so maybe another time; but look at the simple fact that the moment the whole thing is exposed worse-than-usual, the first reaction is to "fix" ... the person exposing it, not the thing.

diana_coman: this is btw exactly why I was saying that dorion's gbw article is quite an interesting read - exactly because it's not written by the usual too-familiar-with-the-madness-to-find-an-easier-way-around-it.
sonofawitch: 2020-07-01 21:08:02 (#ossasepia) diana_coman: dorion - your gbw article is quite an interesting read actually, as it covers a lot of ground and as a side effect, all sorts of tangles and weirds-if-coming-at-them-fresh pop all the clearer into view, hm.
diana_coman: hm, that should be clearer perhaps as too familiar with it to go a longer/harder route.
jfw: on the surface those read as opposites, but I take your meaning as: too familiar with the trails through the forest to bother clearing & paving a road
diana_coman: too familiar to *not* find an easier route; apparently I lost the "not" in there, but it's long anyway.

diana_coman: http://ossasepia.com/2020/07/01/ossasepia-logs-for-Jul-2020/#1027592 - almost; a bit worse in that it's more like too familiar with dodging poisonous darts at each step to even pay much attention to all the crazy moves (sure, also doing as a result of familiarity the best moves there, most effective etc;)
sonofawitch: 2020-07-02 21:55:37 (#ossasepia) jfw: on the surface those read as opposites, but I take your meaning as: too familiar with the trails through the forest to bother clearing & paving a road
diana_coman: in the end the difference is simply "this is insane and has to change" vs "this is unfortunate/not pleasant but inevitable"

jfw: are you saying dorion had that first perspective while I had the second, or what? because I'm not seeing that at all
diana_coman: jfw - no, I'm saying that dorion's article is more useful to give one that first perspective.
jfw: ah ok, indeed.
diana_coman: what perspective each person has is up to them in the end; I don't have any need to highlight other people's perspectives on this now, lol.

jfw: I'd say I was in between - not "unfortunate but inevitable" but that there's a cost to improving it which I didn't see as a high priority, while the article's perhaps pushing me in that direction.
diana_coman: I can see it.
diana_coman: worth perhaps noting also that absent any other ways to commit to one or another piece of work, what gets done or not (and in a more general sense, not specifically about trb now) is always decided simply by perceived cost-of-improvement vs pain-of-current-state, there isn't much else left.
diana_coman: even decided and then re-decided, as the balance shifts, of course.
jfw: yep, and that got a chuckle from me.
diana_coman: heh, glad to hear it did!

Wherein I attempt to lighten my load while - seemingly inadvertently - increasing the generality of the contemplated tool, into a sort of proto-gossipd-for-code:

jfw: for the problem of the manual drudgery presently involved in getting started as well as infecting new systems with jfw-bitcoin, or jfw-wares more generally, I'm thinking to use the perfectly good sync script I already have; you'll simply get *all* my patches then take your pick of what to press. I was trying to prematurely optimize by thinking it needed an option to filter for a specific project.
jfw: that can be the low-hanging fruit for anyone who wants to improve their client; a higher fruit on the same branch would be filtering for just the antecedents of a given patch, e.g. based on the manifest as rendered by a btcbase style patch browser.
jfw: I think I'll be so bold as to house that script in the v.pl tree, remove the recursive-wget based sync that doesn't work on busybox, release a new V starter including my pubkeys, and that'll be that.
jfw: It'd be great to have a federated pool of such "full nodes" based on each operator's choice of signers; the present scheme doesn't quite get there as it does hash-only verification, leaving sig checking to the V presser. A trial press with comparison of flow to output manifest might make a good condition for automated patch import there.
jfw: although the priority on that optimization might go up if a V-enlightened Gales Linux starts pulling in gccs and kernels and such.
jfw: (the filtering by project, that is.)

diana_coman: http://ossasepia.com/2020/07/01/ossasepia-logs-for-Jul-2020/#1027925 - jfw, what would the current steps be if someone starts from 0 ie they go "I'll get a dedicated server for this, what is it going to take to have it running gales+jfw-bitcoin" ?
sonofawitch: 2020-07-17 05:42:41 (#ossasepia) jfw: for the problem of the manual drudgery presently involved in getting started as well as infecting new systems with jfw-bitcoin, or jfw-wares more generally, I'm thinking to use the perfectly good sync script I already have; you'll simply get *all* my patches then take your pick of what to press. I was trying to prematurely optimize by thinking it needed an option to filter for a specific project.

diana_coman: http://ossasepia.com/2020/07/01/ossasepia-logs-for-Jul-2020/#1027928 - as with all great things lately, it's more likely firmly in the camp of "if you make it, you'll have it"; but this aside for a moment, to the extent that you talk strictly of infrastructure what's the problem and otherwise if you talk of operators, well, the lack there is not going to be solved by better scripts or software in general.
sonofawitch: 2020-07-17 06:09:29 (#ossasepia) jfw: It'd be great to have a federated pool of such "full nodes" based on each operator's choice of signers; the present scheme doesn't quite get there as it does hash-only verification, leaving sig checking to the V presser. A trial press with comparison of flow to output manifest might make a good condition for automated patch import there.
diana_coman: that being said and to the extent that I follow what you are saying there, the only rather puzzling part for me is why "v-starter" if it's in fact a "trb-starter" you seem to be aiming for?

jfw: http://ossasepia.com/2020/07/01/ossasepia-logs-for-Jul-2020/#1027931 - that'd be a fairly lengthy list that I don't have available atm; starting from 0 you'd need to establish that your hardware is supported and create or ask nicely for a kernel config
sonofawitch: 2020-07-17 20:36:46 (#ossasepia) diana_coman: http://ossasepia.com/2020/07/01/ossasepia-logs-for-Jul-2020/#1027925 - jfw, what would the current steps be if someone starts from 0 ie they go "I'll get a dedicated server for this, what is it going to take to have it running gales+jfw-bitcoin" ?

diana_coman: jfw - do you have a list of supported hardware ie somewhere for the operator to go and look? or at least one relatively common/easily available config that one can find with hosters?
jfw: what do you mean by "with hosters"?
jfw: but no I haven't presently published kernel configurations.
jfw: they are made available to customers with purchase of hardware, for which we presently have the Thinkpad x200 option and are working on an APU one.

diana_coman: jfw - by configuration I meant hardware, not the kernel config as such; now though I get the impression that this is getting basically split between some conflicting aims there and possibly some not-yet-fully-clear strategy otherwise, hm.
diana_coman: jfw - to roll this back and get somewhere before I need to go as it's not that long today: is it correct to say that you are aiming there for a trb-starter rather than a v-starter as such? (sure, it includes V, of course)

jfw: diana_coman: a common config with plenty of hosters that's known to work is of course VPS, lol
diana_coman: jfw lol
diana_coman: are you now suggesting to run trb on a vps?
jfw: no, but hey, if you want "known to work" it's there
diana_coman: eh; forget about that part, it's clearly too big a jump and it doesn't yet make sense to you, let it be.

jfw: alright. I see what I'm going for as a V starter in the sense of V as a system including the software using it and a means of obtaining that, of which TRB is one of the more patch-heavy examples.
diana_coman: the issue there is that trb is *also* the ...least-mandatory part otherwise, lol
diana_coman: ie I would get it if it were gales the patch-heavy example, seeing how it is an OS, but...why trb, I have no idea.
diana_coman: this being said, you can of course package and name anything anyway you like, what's there to stop you.

jfw: because "given a working OS, how to get TRB / gbw ?" was a pain point as illustrated by http://dorion-mode.com/2020/07/gbw-node-gales-bitcoin-wallet-node-verified-acquisition-build-install-and-run-in-21ish-short-simple-steps/
jfw: Gales, while not in V form as we've discussed, already had an automated download process for all the necessary sources.

diana_coman: jfw - so exactly being "how to get trb/gbw" , it is therefore a ...trb/gbw starter, that's all.
jfw: diana_coman: if I then publish more things it becomes applicable to, does it then change to a trb+gbw+other_things starter, without my even touching it?

diana_coman: in other words: there isn't anything particularly wrong that I can see with the idea of "will package existing script and make the whole a best fit for the task at hand", that's fine; it just isn't more than that though and if there is indeed a longer term strategy of which this is a part it's not clear, nor does it seem to fit such thing, that's about it all.
diana_coman: jfw - so it's basically jfwsoftware-starter, pretty much.
diana_coman: nothing wrong with that.

Wherein I'm still aiming to make it as general as possible:

jfw: revisiting http://ossasepia.com/2020/07/01/ossasepia-logs-for-Jul-2020/#1027989 - I still see it as more general: it's adding to the v.pl tree the capability to fetch a complete set of patches from a given publisher, using a more streamlined scheme than mod6 had included. Indeed I'd be the only one publishing in that format at the start, but nothing says it has to stay that way.
sonofawitch: 2020-07-17 21:35:45 (#ossasepia) diana_coman: jfw - so it's basically jfwsoftware-starter, pretty much.
diana_coman: jfw - my understanding was that you were planning to add not only the capability but the hardcoded publisher=jfw since keys etc; ie adding the capability to my mind does not include specific keys of anyone.
jfw: diana_coman: I see what you mean about the keys.
jfw: so the change to the v.pl tree is general but the starter would indeed be optimized for my stuff albeit re-targetable.
diana_coman: so it sounds like a .vpatch that adds the capability itself and thus perfectly fine in the tree and otherwise a starter pack that is optimised for your use.
jfw: exactly
diana_coman: no problem that I can see there at all.
jfw: cool, ty.

Shortly after this I put something together along the lines discussed, calling it "fixpoint-v-starter". But as I tested and played, it soon started growing extra mini-scripts to deal with those TRB depwads, suggesting that at heart it was indeed a TRB-centric thing, just as Diana had first surmised, yet now weighed down by the generality baggage, both conceptually and practically in terms of directory structuring and script naming.

The depwads could perhaps have been ignored on the observation that there's still the "make ONLINE=1" mechanism. As it stood, this would add dependencies on "curl" and the external "deedbot" service; but moreover, it struck me as an imposition of extra complexity on the user (through the extra download stage) and a fundamental conflation of download and build stages. One way this manifested was that the resulting process would demand a fresh download each time a press was done.

So the thing sat on the shelf while the balance shifted to other pains-of-current-state. I got a couple students through the TRB vpatch download process, by treating it as an opportunity to showcase the power of the CLI (specifically the "awk one-liner" genre) for automating things that didn't otherwise provide for it. By the time I got back around to it, the continued reliance on v.pl was looking more like a liability than anything, so I threw it out and restarted in the direction that led to the present result.

1. For relatively shallow reasons: the OpenSSL build recipe for bitcoind hardcodes it, and gscm uses an assembly instruction for 64-bit extended integer division. [^]
2. I still haven't quite figured it out but as an updated attempt, I'd say it's because of being stuck in a world full of shallow people unable to perceive or care that a mess even exists and thus to do anything about it. [^]