Alert: dumpblock flaw in bitcoind can cause gbw-node to incorrectly report transactions

Filed under: Bitcoin, JWRD, News, Software — Jacob Welsh @ 05:21

The problem

Will Haack reported an observation of the Bitcoin software returning an incorrect block through the "dumpblock" RPC command,(i) as part of his work on a block explorer based on a modified gbw-node.(ii)

Because I designed gbw-node as a small component, trusting the data it receives from a local Bitcoin daemon, and I used dumpblock for lack of alternatives, it can thus import bad transaction data into its SQLite database and incorporate it in the derived reports: unspent-outs, balance, and register.

Some disambiguation of terms is in order to better explain the situation. Orphan blocks(iii) are blocks that may appear structurally valid but cannot be fully verified and connected to the block chain because the observing node does not have their claimed antecedents (parent blocks). Loser blocks are blocks that were at one point verified and connected to a node's current chain, but subsequently disconnected, with any included transactions reversed, due to a different branch overtaking theirs in the proof-of-work race. Both classifications are according to a given node's view of the block at a given time, rather than being intrinsic to the block. In past sloppy usage, loser blocks were sometimes called orphans; this is clearly incorrect because they do have valid parentage tracing all the way to the genesis block.

The problem, then, is that dumpblock searches the implicitly tree-shaped block index by height alone, returning the first match without checking if it's still on the active branch. Thus it can return loser blocks, and potentially at any depth, depending on the races observed over the node's lifetime.

For gbw-node users, the worst case impact would be seeing inbound payments or mining rewards as well-confirmed that are in fact no longer confirmed at all; if its reports are used to gate access to something of value, financial loss to the operator could result and not be discovered until attempting to spend the supposed funds. Perhaps equally severe would be repeating an outbound payment on the false view that an earlier attempt had failed. Presumably the alert user would check other available sources prior to doing this, but it's possible they'd all fail in the same way; and in any case you'd still be stuck having to unwind the mess.

Moderating the potential severity is that there appears to be some cost to pulling this off as an active attack, since creating a block race on demand requires giving up mining rewards. It's not clear to me how common the problematic blocks have been in practice; only one has been noted so far (at the fairly recent height of 685135) but we weren't specifically looking.

For someone using dumpblock to speed up initial sync for new nodes, the result of hitting the bug would be that "eatblock" would stop accepting the provided blocks after the discontinuity, unless the missing winner block is filled in from some other source (such as the normal p2p network).

The fix

In response, I initially considered removing dumpblock altogether in light of its many shortcomings, replacing it with a "getblock" that would return its data directly to the caller rather than through the side channel of the filesystem,(iv) and updating gbw-node to match. Instead I went with minimizing the delay as well as costs and risks of remediation for the user by doing a minimal patch to the existing dumpblock:(v)

I'll probably still go for the "getblock" addition at some point but don't have a concrete plan (and it's perhaps not fully clear yet what should become of "dumpblock" or its companion "eatblock").

Deploying the fix

A process for deploying the fix on an affected system would be something like the following. Needless to say, you'll want to have your backups in order first.

1. Download the above patch to the "patches" subdir of your existing bitcoin V workspace, e.g. ~/src/bitcoin/patches/, and the .sig likewise to the ".seals" subdir.

2. Do a press and enter the new tree:

$ press bitcoin_dumpblock_no_losers bitcoin_dumpblock_no_losers.vpatch
$ cd bitcoin_dumpblock_no_losers/bitcoin

3. Copy in the three pesky depwads: boost_1_52_0.tar.bz2 db-4.8.30.tar.gz openssl-1.0.1g.tar.gz as usual to the "deps" subdir; verify checksums if you're unsure of their provenance:

$ cd deps
$ sha512sum -c Manifest.sha512
$ cd ..

4. Compile using "make", shaking your fist at how long it takes to build all the extraneous cruft.

5. Install the resulting "build/bitcoind" and restart your node as usual.

6. Purge blockchain data from gbw-node's database and do a full rescan as per the heavy workaround noted in the comments. Like the first scan, expect this to take substantial time: on the order of days, depending on how many addresses you have it watching.

7. Audit the reports for your addresses of interest against your accounting records.

If you may have been impacted by this problem and are interested in further assistance or a more refined approach, you know where to reach me.

  1. As far as the history is captured in my V-tree, this command originated in asciilifeform_and_now_we_have_block_dumper_corrected.vpatch and was updated in mod6_fix_dumpblock_params.vpatch. [^]
  2. The network-connected data indexing half of Gales Bitcoin Wallet, which sadly still lacks a proper introductory article for its release, but source code has been available here (web view). [^]
  3. As the term is used in the Bitcoin codebase; also called "bastard blocks" in the same, depending on the author. [^]
  4. Incidentally, the novice wanting to use dumpblock from the CLI is apparently just expected to know that the path is taken relative to the daemon's working directory rather than the client's. [^]
  5. I also took the liberty of correcting Datskovskiy's inconsistent indentation, though only for the changed line to keep the noise down. To head off any further dramaz, I'll note that I did not consult his previously linked patch when producing my own, but did compare afterwards: the code change is essentially identical, which is unsurprising as it's pretty much the obvious fix once you look into it; whereas his patch produces an incorrectly formatted manifest. Apparently that doesn't matter or someone else is supposed to fix the fix or whatever. [^]


  1. Thanks for the clarification regarding orphan blocks vs loser blocks. I am guilty of using the incorrect terminology, referring to loser blocks as orphan blocks.

    Receiving a loser block via the pre-patch'd dumpblock is much greater once your node is sync'd. This is because, afaik, nodes do not feed known loser blocks to other nodes. So as your syncing the earlier blocks, fully sync'd nodes are only going to send you winner blocks. Then, once you are at the tip with all your peers, those peers can no longer filter out soon-to-be-loser blocks, seeing as it can only be known that the blocks will be losers in the future.

    I sync'd my node recently, and thus discovered my first problematic block at height 685,135. Since you and dorion and others sync'd your trb much earlier than mine, I'd imagine that you loaded a couple of loser blocks with heights between, say, ~500,000 and 685,135.

    Comment by whaack — 2021-06-27 @ 16:56

  2. > I am guilty of using the incorrect terminology, referring to loser blocks as orphan blocks.

    As far as I know we were all living in sin.

    > afaik, nodes do not feed known loser blocks to other nodes.

    Unless they're looking to make trouble.

    Comment by Jacob Welsh — 2021-06-28 @ 16:48

  3. [...] Investigation and fix for a bug discovered in earlier Bitcoin Foundation work on the underlying bitc... [...]

    Pingback by Gales Bitcoin Wallet (re)release « Fixpoint — 2021-12-03 @ 08:54

  4. [...] be valid. One would imagine that these can't show up on the active chain, but at the very least my orphans and losers classification was incomplete, and the "dumpblock" flaw worse than previously rea...: there's a third category of, shall we say, hopeful blocks, with validity unknown as they've never [...]

    Pingback by Mapping the bitcoind block acceptance code, with bonus patch for index access « Fixpoint — 2022-03-22 @ 04:12

  5. [...] A flurry of coding and testing ensued, and by the time it was done I had a stack of five patches, bringing: multiple usability improvements, improved portability to older Python interpreters, a minor reduction in scan time per byte, a dramatic reduction in scan time overhead per block, and finally a path toward full removal of the sorest point, the notorious dumpblock horror. [...]

    Pingback by Smartening up gbw-node « Fixpoint — 2023-04-05 @ 01:38

RSS feed for comments on this post. TrackBack URL

Leave a comment

Powered by MP-WP. Copyright Jacob Welsh.