This proposal will detail the software part of my wallet under development,(i) consisting of a security critical offline part and less sensitive online part.
All monetary values are input and displayed in decimal BTC but kept internally in exact integer (satoshi) form; conversion must be lossless.
Offline
Wallet data is stored in a directory tree containing a keys subdirectory and outputs, change, fee, and transactions files.
Private keys are represented in hex, with one key per file in the keys directory. (Probably little-endian; need to check existing conventions.)
Each key file is named by the Base58Check-encoded Bitcoin address corresponding to the key. (This implies a case-sensitive filesystem.) Support for compressed public keys may be added using an extra tag byte in the private key.(ii)
A list of outputs believed to be spendable is maintained in the tabular text file outputs, with records delimited by linefeeds and the following fields delimited by one or more space or tab characters:
- Address (Base58Check)
- Value (decimal BTC)
- TXID (hash of transaction containing the output, in the conventional little-endian hex format)
- Index (position in the transaction's output vector, decimal integer)
- Comment (optional; may contain spaces and tabs)
Encryption is supported by keeping these in a tmpfs and persisting with tar+gpg or similar.
The offline wallet program provides the following commands:
gen-key- generate a new key file and print the resulting address to stdout. (Nice to have: numeric argument to generate many.)send ADDR VAL ...- generate a signed transaction sending the given addresses the given amounts in BTC, pairwise. Inputs are selected from the outputs table in the order they appear. The change address is read from thechangefile; the fee in BTC per 1000 bytes is read from thefeefile. The hex-encoded raw transaction is appended to thetransactionsfile. (Nice to have: summarize the proposed transaction and prompt for confirmation.) The outputs file is overwritten to remove the spent ones and add the change and any others that pay the wallet's own addresses.
Once transferred, the transactions file can be removed or truncated at will.
Online
The online part has a database that serves to index a lightweight subset of blockchain data pertinent to a single offline wallet. It tracks the following objects:
- Watched addresses
- Confirmed outputs funding them and inputs spending from them
- Confirmed transaction metadata (hash, block hash+height+index, size, fee, comment)
- Raw transactions submitted by the operator
- Scan state
The online wallet program communicates with a Bitcoin node to populate its database and transmit new transactions. It provides sync commands:
scan- iterate blocks, filter for relevant transactions and update database.reset- clear the scan pointer, e.g. to pick up past transactions affecting newly watched addresses. (Nice to have: argument to set to a given height)
Input/output commands:
unspent-outs- print unspent output table in the format used by the offline wallet. (Nice to have: other views like all outputs or only new outputs)watch- import new addresses to watch from stdin, with optional comments.push- import raw transaction(s) to push from stdin, with optional comments.
Accounting commands:
balance- print the sum of unspent output values.register- print transaction history, format TBD.
Plan and time estimates
Online:
Write SQLite schema (currently drafted on paper): 1h
Import RPC client, block and transaction parsing code (I considered using python-bitcoinlib from Garzik et al. here; happily I have original implementations of the necessary parts): 1h
Block fetching ("dumpblock" from TRB to named pipe): 1h
Sync commands: 3h
I/O commands: 3h
Accounting commands (using something quick and dirty for "register" format for v1): 2h
Write manual: 3h
Total online part: 14h
Proposed deadline: Sunday, Dec 1.
Offline:
Port transaction signing prototype to Scheme: 2h
Key file I/O: 1h
BTC numeric I/O: 1h
Outputs table I/O: 3h
Update "gen-key" command for new spec: 1h
"send" command: 3h
Sample tar+gpg wallet open and close scripts: 1h
Design and implement tests: 5h
Write manual: 3h
Total offline part: 20h
Proposed deadline: Thursday, Dec 12.
Other:
"sendrawtransaction" RPC for TRB: 5h
Total: 39h
Deadline: Friday, Dec 13.
Now: I see 40 available hours remaining in my schedule for this time interval. Given that I'm far from confident in my estimation abilities here and there's always something unexpected, this is worrying. Perhaps the documentation would be a light enough job at this point to handle on the plane. Perhaps there's a "sendrawtransaction" patch floating around somewhere. Perhaps someone can be motivated to lend a hand - this would be the most separable task of the pile.
- Through further consideration of the coding tasks and data dependencies, in particular mapping out the online part as a relational database, I've realized my earlier notion of transactions as the central data structure had introduced unnecessary complications. If accounting of historical transactions is left to the online database, the offline signing part need only be aware of currently unspent outputs, which reduces its storage needs to a flat table, no S-expressions or JSON or similar. I believe this also brings it in closer alignment with the ideal. A second simplification is in the storage of addresses and keys, following a philosophy of "the filesystem is a database; use it", and naming of keys has been dropped. [^]
- This may be desirable to import keys or sweep funds from legacy wallets without having to execute their code. If used by default it would reduce outbound transaction size, thus perhaps fees, though as ever miner behavior is uncertain. [^]
The offline part need not be aware of ~anything~. You tell it what to do, it does. The online part tells you what you might be telling it to do. See ?
Comment by Mircea Popescu — 2019-11-28 @ 17:36
I think I see. In the design here, the way you tell it what to do is by editing the "outputs" file - which might mean appending new outputs that the online part told you about, but could mean anything you please because it's not automatic. It's only ever "aware" of what you've told it, directly or through its own "send" command. Is that the idea?
Comment by Jacob Welsh — 2019-11-28 @ 18:32
Well rather, the idea is that your online/node can produce unsigned rawtx for ~any~ inputs it knows of, it doesn't need to know which are "yours" for whatever definition of yours. In practice, it's as if it had in fact produced all of them.
You select some from this pile, arbitrarily and pass them to the offline/wallet to process (aka sign), which it does, on faith -- ie, it takes you at your word that those inputs so and so described do in fact exist. It's after all up to you whether you lie to it or not, seeing how it is... your thing.
That's the correct cut, disentangling the mess on the need to know basis -- neither node nor wallet need to know what they're doing better than you. In fact, it's quite ideal if they do not.
Comment by Mircea Popescu — 2019-11-28 @ 22:55
Intriguing; something continues to elude me here. I'll run with the node + wallet terminology (the distinction between node and "online part of wallet" as used here seems to be a technical artifact really).
If a node doesn't track which addresses are of interest to me (the "watch" command in the article), and the wallet stores nothing but keys, does it follow that neither node nor wallet can report a balance? I suppose I could track that elsewhere, but based on what - how would I learn when I've received a payment? How would I select inputs I can sign for, out of a global pile of millions, if the machine can't help?
Comment by Jacob Welsh — 2019-11-29 @ 01:41
Updated to s/pushrawtransaction/sendrawtransaction/ following the historical naming.
Comment by Jacob Welsh — 2019-11-29 @ 04:04
The node can track imaginary "balances" of any arbitrary selections of addresses one chooses. Which of those are xored how to get your real balance is ~your business~. The node really doesn't have to know anything about you at all.
Comment by Mircea Popescu — 2019-11-29 @ 05:26
Bridging in:
Comment by Jacob Welsh — 2019-11-30 @ 02:07
[...] this ended up stimulating the lines of thought necessary to clarify the spec and I was then able to write it. I think it ended up for the best to have taken the time, especially given the thought-provoking [...]
Pingback by JFW review, week of Nov 25 2019 « Young Hands Club — 2019-12-02 @ 06:05
I know you moved the deadline for the online part to this week and generally this was pushed a bit in lieu of Gales Linux publications.
Please make an update to your deadline for delivery of the offline part here when you dig back into that later this week.
Comment by Robinson Dorion — 2019-12-10 @ 22:20
[...] of them fills a still-unmet requirement of any split wallet so I'm reviewing it in hopes of saving some time on having to redo from scratch. Full patch is [...]
Pingback by Review of polarbeard_add_sendrawtransaction_rpc.vpatch « Fixpoint — 2019-12-15 @ 20:14
[...] Gales Bitcoin Wallet spec and battle plan. [...]
Pingback by Draft gbw-node schema « Fixpoint — 2019-12-16 @ 23:01
@Robinson Dorion: I suppose I failed to make the requested update here because I wanted to avoid a big fat "I don't know", and was still hoping to find the time somehow to better evaluate. Sorry about that.
I still don't know, except that finishing by Wednesday (departure Thursday early) is looking quite unlikely.
Comment by Jacob Welsh — 2019-12-17 @ 05:16
Ok, give it some more thought this week and we can discuss when you visit.
Comment by Robinson Dorion — 2019-12-17 @ 16:37
[...] my offline Bitcoin signer nears completion, it's a good time to introduce just what Bitcoin transactions are anyway, how they [...]
Pingback by Bitcoin transactions and their signing, 1 « Fixpoint — 2020-02-25 @ 23:43
[...] our wallet is built to a spec from last year, vpatches are at gbw-node, gbw-signer, and the interpreter gscm, what's missing is mainly [...]
Pingback by #ossasepia Logs for Jun 2020 « Ossa Sepia — 2020-06-26 @ 13:35
[...] think returning the results to queries as an sexpr, and/or something that interfaces well with Gale's Bitcoin Wallet is the way to [...]
Pingback by Block Explorer Update « whaack — 2021-05-15 @ 22:01
[...] and was updated in mod6_fix_dumpblock_params.vpatch. [^]The network-connected data indexing half of Gales Bitcoin Wallet, which sadly still lacks a proper introductory article for its release, but source code has been [...]
Pingback by Alert: dumpblock flaw in bitcoind can cause gbw-node to incorrectly report transactions « Fixpoint — 2021-06-20 @ 05:21
[...] User interface specification for two programs, an online "node" and offline "signer", with more deta... [...]
Pingback by Gales Bitcoin Wallet (re)release « Fixpoint — 2021-12-03 @ 08:58
[...] the program is a modification of the online component to jfw's Gales Bitcoin Wallet, the scope of the project has changed enough that I thought it proper to regenesis instead of apply [...]
Pingback by Trbexplorer Genesis Release « whaack — 2022-02-13 @ 20:51