Fixpoint

The problem

Will Haack reported an observation of the Bitcoin software returning an incorrect block through the "dumpblock" RPC command,⁽ⁱ⁾ as part of his work on a block explorer based on a modified gbw-node.⁽ⁱⁱ⁾

Because I designed gbw-node as a small component, trusting the data it receives from a local Bitcoin daemon, and I used dumpblock for lack of alternatives, it can thus import bad transaction data into its SQLite database and incorporate it in the derived reports: unspent-outs, balance, and register.

Some disambiguation of terms is in order to better explain the situation. Orphan blocks⁽ⁱⁱⁱ⁾ are blocks that may appear structurally valid but cannot be fully verified and connected to the block chain because the observing node does not have their claimed antecedents (parent blocks). Loser blocks are blocks that were at one point verified and connected to a node's current chain, but subsequently disconnected, with any included transactions reversed, due to a different branch overtaking theirs in the proof-of-work race. Both classifications are according to a given node's view of the block at a given time, rather than being intrinsic to the block. In past sloppy usage, loser blocks were sometimes called orphans; this is clearly incorrect because they do have valid parentage tracing all the way to the genesis block.

The problem, then, is that dumpblock searches the implicitly tree-shaped block index by height alone, returning the first match without checking if it's still on the active branch. Thus it can return loser blocks, and potentially at any depth, depending on the races observed over the node's lifetime.

For gbw-node users, the worst case impact would be seeing inbound payments or mining rewards as well-confirmed that are in fact no longer confirmed at all; if its reports are used to gate access to something of value, financial loss to the operator could result and not be discovered until attempting to spend the supposed funds. Perhaps equally severe would be repeating an outbound payment on the false view that an earlier attempt had failed. Presumably the alert user would check other available sources prior to doing this, but it's possible they'd all fail in the same way; and in any case you'd still be stuck having to unwind the mess.

Moderating the potential severity is that there appears to be some cost to pulling this off as an active attack, since creating a block race on demand requires giving up mining rewards. It's not clear to me how common the problematic blocks have been in practice; only one has been noted so far (at the fairly recent height of 685135) but we weren't specifically looking.

For someone using dumpblock to speed up initial sync for new nodes, the result of hitting the bug would be that "eatblock" would stop accepting the provided blocks after the discontinuity, unless the missing winner block is filled in from some other source (such as the normal p2p network).

The fix

In response, I initially considered removing dumpblock altogether in light of its many shortcomings, replacing it with a "getblock" that would return its data directly to the caller rather than through the side channel of the filesystem,^(iv) and updating gbw-node to match. Instead I went with minimizing the delay as well as costs and risks of remediation for the user by doing a minimal patch to the existing dumpblock:^(v)

• bitcoin_dumpblock_no_losers.vpatch (sigs: jfw)

I'll probably still go for the "getblock" addition at some point but don't have a concrete plan (and it's perhaps not fully clear yet what should become of "dumpblock" or its companion "eatblock").

Deploying the fix

A process for deploying the fix on an affected system would be something like the following. Needless to say, you'll want to have your backups in order first.

1. Download the above patch to the "patches" subdir of your existing bitcoin V workspace, e.g. ~/src/bitcoin/patches/, and the .sig likewise to the ".seals" subdir.

2. Do a press and enter the new tree:

$ v.pl press bitcoin_dumpblock_no_losers bitcoin_dumpblock_no_losers.vpatch
$ cd bitcoin_dumpblock_no_losers/bitcoin

3. Copy in the three pesky depwads: boost_1_52_0.tar.bz2 db-4.8.30.tar.gz openssl-1.0.1g.tar.gz as usual to the "deps" subdir; verify checksums if you're unsure of their provenance:

$ cd deps
$ sha512sum -c Manifest.sha512
$ cd ..

4. Compile using "make", shaking your fist at how long it takes to build all the extraneous cruft.

5. Install the resulting "build/bitcoind" and restart your node as usual.

6. Purge blockchain data from gbw-node's database and do a full rescan as per the heavy workaround noted in the comments. Like the first scan, expect this to take substantial time: on the order of days, depending on how many addresses you have it watching.

7. Audit the reports for your addresses of interest against your accounting records.

If you may have been impacted by this problem and are interested in further assistance or a more refined approach, you know where to reach me.

1. As far as the history is captured in my V-tree, this command originated in asciilifeform_and_now_we_have_block_dumper_corrected.vpatch and was updated in mod6_fix_dumpblock_params.vpatch. [^]
2. The network-connected data indexing half of Gales Bitcoin Wallet, which sadly still lacks a proper introductory article for its release, but source code has been available here (web view). [^]
3. As the term is used in the Bitcoin codebase; also called "bastard blocks" in the same, depending on the author. [^]
4. Incidentally, the novice wanting to use dumpblock from the CLI is apparently just expected to know that the path is taken relative to the daemon's working directory rather than the client's. [^]
5. I also took the liberty of correcting Datskovskiy's inconsistent indentation, though only for the changed line to keep the noise down. To head off any further dramaz, I'll note that I did not consult his previously linked patch when producing my own, but did compare afterwards: the code change is essentially identical, which is unsurprising as it's pretty much the obvious fix once you look into it; whereas his patch produces an incorrectly formatted manifest. Apparently that doesn't matter or someone else is supposed to fix the fix or whatever. [^]