Day changed to 2024-03-01
[01:28] jfw: dorion: try building with http://jfxpt.com/wp-content/uploads/2024/03/bitcoin_reorg_tracing.patch (based on my last, bitcoin_rebranding.vpatch) and run it through one reorg cycle, with the memory log going too for good measure
[01:29] jfw: mempool wrangling is one possible "leak" source but doesn't fit with the "one or a few big allocations at once" theory
[01:31] jfw: dorion: and get me the debug log from the "REORGANIZE" through the OOM exception.
[01:32] jfw: best if you can have it disconnected from peers while that runs.
[01:32] jfw: but I'm unsure if at least one new block is required to trigger the reorg
[05:02] dorion: alright, I had killed it ; will fire it back up, wait to see reorg and disconnect the network.
[18:55] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10498 - this appears to be so, and worse - if the new block is a bastard block, the afflicted node sends a 'getblocks' to try to catch up, so the peer sends a handful of inventory, which the afflicted node already has, just on its inactive, not-yet-connected reorg branch, so it doesn't ask for the blocks, so the
[18:55] sourcerer: 2024-03-01 01:32:46 (#jwrd) jfw: but I'm unsure if at least one new block is required to trigger the reorg
[18:55] jfw: continuation hash never triggers and no further inventory is sent, so the reorganize is never re-triggered.
[19:06] jfw: here btw is for the full linked dataset on memory use over 14 hours, where we can clearly see the node giving up on the reorg after 7 hours, presumably because it stopped getting fed new shiny blocks to chase.
[19:12] jfw: whaack: I have a keksum patch just about ready to fix the annoyance we ran into where it dies on the first error when you do something like 'keksum *' and there's subdirectories or otherwise unreadable files present (and fixing a blatant typo in the usage synopsis).
[19:15] jfw: I also started on a '-c' implementation i.e. to verify files against an existing hash listing, but this is turning out considerably more complicated (basically, parsing sucks and especially in C); it seems to me that the "make your own listing and diff them" remains an adequate solution for now but what do you think, would the builtin checking be very helpful? from the standpoint of the jwrd
[19:15] jfw: training client at least
[19:19] jfw: and dorion, have you run into that situation much (or does it perhaps hold back keksum adoption vs. sha*sum)?
[19:22] jfw: it's certainly doable, and with only a small bending of the "no buffered I/O or dynamic memory allocation" restrictions of keksum's current style, just going to be closer to the 200 LoC than 20 LoC mark.
[19:27] dorion: jfw, I can't say lacking the feature has held me back much and the extra step of using diff sounds like a 'good enough' alternative to have to rely on for now.
[20:02] jfw: in the reorg saga, a targeted 'eatblock' woke it up so we should get a clearer picture.
[20:05] jfw: whaack, we couldn't readily work out how to get a raw (binary) block out of your explorer; how would you suggest converting from hex? or something else?
[20:34] dorion: jfw, teh log , I trimmed out the connect/disconnect noise and added in the error that went to stdo, but not the log.
[20:43] dorion: for those following allow at home, bitcoind runs out of memory on a reorg attempt because it tries to connect 210 blocks at a time.. rather than just a few.
[20:43] dorion: or even a baker's dozen or w/e.
[20:43] dorion: s/allow/along/
Day changed to 2024-03-02
[03:55] whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10505 << I don't think this is helpful, it can already be easily done with existing tools. I was surprised when you showed me in class that such a feature existed for sha512sum
[03:55] sourcerer: 2024-03-01 19:15:58 (#jwrd) jfw: I also started on a '-c' implementation i.e. to verify files against an existing hash listing, but this is turning out considerably more complicated (basically, parsing sucks and especially in C); it seems to me that the "make your own listing and diff them" remains an adequate solution for now but what do you think, would the builtin checking be very helpful? from
[03:56] whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10511 << raw per my thought process meant the representation of the bits in hexadecimal, perhaps i need to change the wording and/or add an endpoint for the binary bits, sorry that you lost time for that
[03:56] sourcerer: 2024-03-01 20:05:53 (#jwrd) jfw: whaack, we couldn't readily work out how to get a raw (binary) block out of your explorer; how would you suggest converting from hex? or something else?
[04:18] dorion: whaack, we didn't lose too much time on it, no sweat. it was more an opportunity to test and when we couldn't figure it out, jfw just dumped the block we needed from his node and passed it over to me for eating.
[04:19] dorion: to test your explorer. btw, not sure if I ever mentioned, but while I'm thinking of it, I never got the push raw txn to work on your explorer.
[16:52] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10521 <-> http://jfxpt.com/2024/jwrd-logs-for-Feb-2024/#10488
[16:52] sourcerer: 2024-03-02 04:19:13 (#jwrd) dorion: to test your explorer. btw, not sure if I ever mentioned, but while I'm thinking of it, I never got the push raw txn to work on your explorer.
[16:52] sourcerer: 2024-02-29 18:39:04 (#jwrd) whaack: i still have to figure out what's wrong with the push command
[16:55] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10518 - there is that use of 'raw' as in 'raw transaction' indeed. it could certainly be dealt with client-side, just needs working out since it's not base64 or something a standard utility can eat directly. maybe some tricky 'od' recipe
[16:55] sourcerer: 2024-03-02 03:56:18 (#jwrd) whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10511 << raw per my thought process meant the representation of the bits in hexadecimal, perhaps i need to change the wording and/or add an endpoint for the binary bits, sorry that you lost time for that
[16:59] jfw: to add to the confusion though there's that "raw_data" checkbox, all it seems to do is remove the form header and links but is still returning html
[17:00] dorion: jfw, here's the memory view from the most recent bitcoind run : http://jfxpt.com/paste/gfhghneixu
[17:01] jfw: dorion: ty, looking.
[17:02] jfw: the log output from the reorg certainly looks much more helpful now, I'd say that patch is a keeper.
[17:02] dorion: de acuerdo.
[17:02] jfw: "Reorganize() : fork height=825321 with 0 to disconnect, 501 to connect
[17:03] jfw: " - seems odd that there would be none to disconnect, that just sounds like normal growth rather than fork/reorg
[17:10] jfw: the memory series in seconds with data archived alongside as before
[17:22] jfw: kind of inconsistent that uncaught exceptions from a peer network message get logged but those from rpc commands don't
[17:25] jfw: but assuming there wasn't anything missed in the log extract between the ConnectBlock messages and the exception, it seems to rule out the bdb commit as the culprit, since it never gets there, indeed only gets through 210 of the 501 blocks
[17:38] dorion: jfw, I don't think I did any excessive snipping of the log, but will double check.
[17:40] jfw: and 825321 is your current height, correct?
[17:43] jfw: ah no, it was 825821 (per my dump command history). a suspicious cooincidence with the 501 blocks to connect
[17:44] dorion: 825321 is where is it stuck. e.g. last SetBestChain and what getinfo reports.
[17:44] jfw: oh right.
[17:46] jfw: the 500 difference makes sense then at least.
[18:09] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10533 - this makes sense based on the code too: writing a block's data and adding it to the block index is done & committed prior to 'connecting' it (CBlock::SetBestChain). presumably the latter failed, recurringly, due to the disk I/O error. thereafter, it would continue recording the new blocks but being unable to connect. thus, the new block's
[18:09] sourcerer: 2024-03-02 17:03:10 (#jwrd) jfw: " - seems odd that there would be none to disconnect, that just sounds like normal growth rather than fork/reorg
[18:09] jfw: parent doesn't match the current best block, which is enough to get it treated as a reorg.
[18:11] jfw: so basically this is a preview of what might happen in a fork war with actual deep reorgs
[18:12] jfw: and what happens is that we get stuck
[19:06] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10513 - the logic here, such as it is, would be that the reorg needs to be tried as a whole in order to establish that the proposed chain is even valid
[19:06] sourcerer: 2024-03-01 20:43:36 (#jwrd) dorion: for those following allow at home, bitcoind runs out of memory on a reorg attempt because it tries to connect 210 blocks at a time.. rather than just a few.
[19:08] jfw: then an optimization would be to draw that line and commit at the earliest PoW-longer block which is not necessarily the tip of the new chain, and then things can proceed incrementally
[19:09] jfw: but perhaps more important would be to contain memory usage even for deep reorgs, because that optimization wouldn't solve it in general
[19:22] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10547 -- good thing we're working on it now then !!
[19:22] sourcerer: 2024-03-02 18:12:17 (#jwrd) jfw: and what happens is that we get stuck
[19:23] dorion: and once we get it solved on our side, would be interesting to investigate the extent to which prb deals with it.
[20:03] jfw: myeah, we're working on it at least but still in a reactive way
[20:06] jfw: and not saying that's wrong from the business standpoint, like, that I should have been fixing the unloved codebase instead of making money; more that we need to get more money flowing in its direction. for which I'm cautiously optimistic about the 'key management consulting'
[20:14] dorion: jfw, agreed. I think there may be some other creative ways too. need to investigate further though.
[20:16] jfw: in desktop rebuilding news, the Antec P101 arrived after a whopping 16 days in 'express' shipment from miami, 'vuelos diarios' doncha know; came well packaged and weighing a hearty 30+ lbs. I got the machine reassembled, fresh thermal paste and all that and all seems well so far, even boots Gales despite the UEFI infection of the motherboard.
[20:16] sourcerer: 2024-02-05 04:44:14 (#jwrd) jfw: looks like I'll go with Antec Performance Series P101 Silent, runner up Phanteks Enthoo Pro series PH-ES614PC_BK
[20:21] jfw: the background here was I shipped two desktops down to myself in panama back in 2018 or so, the cases got all bent up and the better one failed to run reliably; the hope was that the electronics were OK and just the connections were flaky due to the physical compromise of the enclosure.
[20:23] jfw: the plan is to use the machine for a home server, on the idea that it's a lower bar to find an ISP adequate to connect fiber to our premises than to find one adequate to physically entrust our data and iron.
[20:24] jfw: so far the search still isn't going well though.
[21:56] whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10527 <-- yeah i can see how that is quite confusing
[21:56] sourcerer: 2024-03-02 16:59:03 (#jwrd) jfw: to add to the confusion though there's that "raw_data" checkbox, all it seems to do is remove the form header and links but is still returning html
[23:11] whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10521 <-- yup, it does not work. i'm going to finally look into it.
[23:11] sourcerer: 2024-03-02 04:19:13 (#jwrd) dorion: to test your explorer. btw, not sure if I ever mentioned, but while I'm thinking of it, I never got the push raw txn to work on your explorer.
Day changed to 2024-03-03
[02:02] jfw: what a pile of stupid dependencies or limitations I hit when reproducing my last soft-RAID Gales server setup
[02:02] sourcerer: 2022-04-06 01:53:59 (#jwrd) jfw: to summarize my findings on the soft-RAID side: the current lilo build *can* load kernels from a RAID volume, but only if it's using the old metadata format version 0.90; the chief shortcoming of 0.90 compared to the current 1.x formats is a 2TB size limit; thus there's perhaps still some value in the old hack of making a separate small /boot partition.
[02:03] jfw: http://jfxpt.com/2022/jwrd-logs-for-Apr-2022/#3712 , http://jfxpt.com/2022/jwrd-logs-for-Apr-2022/#3713
[02:03] sourcerer: 2022-04-06 01:55:31 (#jwrd) jfw: of course in that case you'd need GPT (while if the other stuff didn't suck so hard you could just use the full drive with no partitioning)
[02:03] sourcerer: 2022-04-06 01:58:01 (#jwrd) jfw: further, the kernel is able to auto-assemble arrays so as to mount a root FS on RAID without needing to bother with an initramfs - but only with 0.90 format. this applies even if you use the explicit syntax to tell the kernel the array structure (eg "md=0,/dev/sda1,/dev/sdb1").
[02:05] jfw: meanwhile the initramfs is built by a util in the linux tree, building linux still demands bc, which demands m4, which demands autoconf, which demands perl
[02:05] sourcerer: 2022-04-06 06:18:13 (#jwrd) jfw: so now they're included as /etc/examples/initramfs.*, provide more helpful comments, and actually work on gales; in combination with the new mdadm port I've now got the server assembling its RAIDs unassisted; now there just remains to relocate the root FS onto the array.
[02:07] jfw: and having mdadm as a gport rather than in the base or at least included with the installer made for a rather towers-of-hanoi like situation where I've had to build that port in the installer's tmpfs multiple times in order to access the target filesystem
[02:12] jfw: the 'bc' thing is one relatively low hanging and important fruit here I'd think: the base system should be able to build its own damn kernel at the very least, given that gcc is included.
[02:13] jfw: running 'make menuconfig' requires ncurses too, there's not likely to be a cure for that, but at least you can build from existing configs without it.
[02:16] jfw: packaging the initramfs utility (usr/gen_init_cpio) outside of the linux tree or finding an alternative might not be too hard either
[02:27] whaack_: hm, just did a successful push from the command line version of bitcoindexplorer
[02:31] whaack_: dorion: any chance you saved the error message from your attmept?
[02:54] dorion: whaack, no I didn't save and don't recall, sorry. perhaps jfw recalls what it said.
[03:01] whaack_: dorion: i have a hunch to what the problem is
[03:02] whaack_: push *should* be working on the NZ mirror http://103.6.212.28/
[03:08] whaack_: it seems the user that apache is using doesn't have permission to do a bitcoin rpc call, not sure why that's true for only 1 of my servers
[03:11] dorion: alright, well, seems like you're getting closer in any case. nice.
[03:25] whaack_: i think it should work now
[03:25] whaack_: will try next time i need to do a tx
[03:32] dorion: cool.
[03:42] dorion: whaack, while you're back to the explorer work, you may want to consider jfw's comment from back when you released it. perhaps a regrind of the genesis before you get to far on the current ? also, are you going to create a new key now that you have some practice with the airgap ?
[03:42] sourcerer: 2022-02-16 03:03:59 (#jwrd) jfw: whaack: re http://ztkfg.com/2022/02/trbexplorer-genesis-release/ , since you seem to indeed take the stance that your thing is changed entirely, with no attempt at maintaining compatibility etc - which I reckon is for the best since we're not really collaborating on it - I can't for the life of
[03:43] dorion: in any case, perhaps you'd like to register in our [http://jfxpt.com/2023/grand-reopening-of-jwrd-the-irc-channel/?b=If you wish to register&e=#select][protowot] as well.
[17:26] whaack: dorion: i will regrind, and also rename to bitcoindexplorer from trbexplorer
[17:27] whaack: my current gpg key is airgapped. its security is in question only b.c. it's 2048 bit iirc and it was seeded using a linux distros urandom rather than a TRNG
[17:29] whaack: lol 'how to airgap a practical guide' on goolag search does not return trilema on the front page of results. i'm used to trilema being hard-coded out but usually it shows up if the string is right. it does show up if you search 'how to airgap. a practical guide' (note the added .)
[17:29] whaack: newz @ 11
[18:04] whaack: left hand is hurting for the first time in a bit, i mean they always hurt a little nowadays but this time i am considering stopping typing because of it. specifically it's hurting in the carpal tunnel of the palm side, as i type now though it seems to be ok
[18:37] jfw: whaack: bummer. in other time & keystroke saving ideas though, I was looking at how to streamline the gport build & install steps, and what seems to stick so far is to make a "smarter" gpkg-install
[18:38] whaack: thx. i'm pushing through it. it seems to have abadated
[18:38] jfw: where it can take a package file as before, or the port directory containing build.sh, from which it determines the package file name, and launches gbuild if necessary
[18:40] jfw: and can also stash the result to /gales/dist/pkg/
[18:47] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10590 - did you type that url by hand or something? normally when copied out of the browser spaces and such get properly %-encoded
[18:47] sourcerer: 2024-03-03 03:43:54 (#jwrd) dorion: in any case, perhaps you'd like to register in our [http://jfxpt.com/2023/grand-reopening-of-jwrd-the-irc-channel/?b=If you wish to register&e=#select][protowot] as well.
[18:47] jfw: (I'm seeing how the url isn't detected as desired by the bot)
[18:54] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10516 - thanks, and I recall that reaction now that you mention. and keksum makes it even easier to verify a single file hash programatically, since when fed from stdin it prints just the hash without the extra filename field to cut out
[18:54] sourcerer: 2024-03-02 03:55:02 (#jwrd) whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10505 << I don't think this is helpful, it can already be easily done with existing tools. I was surprised when you showed me in class that such a feature existed for sha512sum
[18:59] jfw: atm I'm back to looking into the reorganize fail.
[19:13] jfw: one issue here is that block read errors aren't really adequately handled
[19:15] jfw: yeah, the function returns a success status so the calling code dutifully checks it but then all it does is log the error and give up on the reorg as if it were an invalid block
[19:17] jfw: seems to me that if there's an I/O error on database operations, the node's only reasonable options are either to abort entirely or else enter some "cry for help" maintenance state
[19:18] jfw: where it could perhaps still answer some local queries but won't pretend to still be following the blockchain
[19:30] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10591 -- cool.
[19:30] sourcerer: 2024-03-03 17:26:18 (#jwrd) whaack: dorion: i will regrind, and also rename to bitcoindexplorer from trbexplorer
[19:30] jfw: obviously that would be more complex codewise and harder to get right (noting how the existing 'safe mode' misses the mark)
[19:30] jfw: ("cry for help" mode, that is)
[19:30] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10592 -- was it always airgapped or did you move it over at a later stage ?
[19:30] sourcerer: 2024-03-03 17:27:23 (#jwrd) whaack: my current gpg key is airgapped. its security is in question only b.c. it's 2048 bit iirc and it was seeded using a linux distros urandom rather than a TRNG
[19:32] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10593 -- when I'm looking for a trilema article, i typically go straight to prepending with 'site:trilema.com' ; myeah, notreallynoose google sux.
[19:32] sourcerer: 2024-03-03 17:29:35 (#jwrd) whaack: lol 'how to airgap a practical guide' on goolag search does not return trilema on the front page of results. i'm used to trilema being hard-coded out but usually it shows up if the string is right. it does show up if you search 'how to airgap. a practical guide' (note the added .)
[19:33] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10595 -- sorry to hear that.
[19:33] sourcerer: 2024-03-03 18:04:47 (#jwrd) whaack: left hand is hurting for the first time in a bit, i mean they always hurt a little nowadays but this time i am considering stopping typing because of it. specifically it's hurting in the carpal tunnel of the palm side, as i type now though it seems to be ok
[19:33] whaack: dorion: if at any point in time it was not airgapped, i would not consider the key airgapped
[19:34] whaack: my gpg key has, to the best of my knowledge, never been present on any device with an active internet connection
[19:34] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10600 -- no, I didn't hand type and confirmed again that's how it pastes on my system.
[19:34] sourcerer: 2024-03-03 18:47:00 (#jwrd) jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10590 - did you type that url by hand or something? normally when copied out of the browser spaces and such get properly %-encoded
[19:36] jfw: testing over the shoulder shows the behavior I expected so nfi.
[19:37] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10620 -- good to confirm we see it the same way. curious how you ferried data back and forth prior to the optical data diode ?
[19:37] sourcerer: 2024-03-03 19:33:41 (#jwrd) whaack: dorion: if at any point in time it was not airgapped, i would not consider the key airgapped
[19:38] whaack: dorion: usb stick. this is a sore point as well
[19:38] dorion: I see.
[19:39] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10608 -- makes sense to me.
[19:39] sourcerer: 2024-03-03 19:17:25 (#jwrd) jfw: seems to me that if there's an I/O error on database operations, the node's only reasonable options are either to abort entirely or else enter some "cry for help" maintenance state
[19:49] dorion: whaack, did you start to read that stem cell book at all ? the treatment dropped a good bit since you last considered it and while that's likely to continue you're not getting any younger either.
[19:50] dorion: ***dropped good bit in price in satoshi terms.
[19:57] whaack: dorion: no, maybe i'll give it a flip through today.
[20:01] dorion: alright. no pressure or anything, but, while a bit outdated, that text gives a decent overview of the field.
[20:17] whaack: dorion: i forget, did you try the treatment yourself?
[21:19] dorion: not yet, don't have anything acute, but am curious. q2 or q3 I expect to.
[21:40] jfw: dorion: let's load your bitcoind under gdb, do 'catch throw', run it disconnected, and feed the next block to trigger reorg. that ought to get us a stack trace, unless there's too many unrelated exceptions going on to push through in normal control flow but I'd think not
[21:40] jfw: besides the stack trace it should let us see RAM usage prior to the unwinding.
[21:43] jfw: this is for the higher level question, given that we know there's a slow leak via mempool bookkeeping, of why it fails even with plenty of RAM still available.
[23:13] jfw: from that experiment: debug log, gdb session, memory data (now with explicit time field) and
[23:13] jfw: visual looking the same as usual
[23:33] jfw: the txid seen in that stack trace translates to 6bf00af44faf92918bd335b42c1b668f23132235dce743adef8378bd4dd45d40 - dorion, can you resume the node and try a getrawtransaction on it, just to see if maybe it's something triggered by reading that specific item from your db?
[23:35] dorion: jfw, on it.
[23:50] jfw: worked fine except needed -disablesafemode (because of the broken error handling which treated the I/O error the same as an invalid block)
[23:50] sourcerer: 2024-03-03 19:15:38 (#jwrd) jfw: yeah, the function returns a success status so the calling code dutifully checks it but then all it does is log the error and give up on the reorg as if it were an invalid block
[23:52] dorion: for the log : 0100000001afd1a3182539803bec0098be560dacfce655a5afe5653846c52405bed29f85460000000000ffffffff0272af0100000000001976a914b63927ce18471fb052fdcae3c5d71f7fed25247388aca8340000000000001600140307cf037216c44306b544206f74f27d53c3e40900000000
Day changed to 2024-03-04
[00:26] dorion: from .bitcoin/db.log I have a bunch of "Lock table is out of available object entries"
[00:35] jfw: got that clue by way of Troubleshooting common Berkeley DB problems which some wrecker apparently removed in my version of the reference
[00:42] jfw: it would appear the 'maxint locks' fork/wedge saga with magic numbers handed down from MP himself
[00:42] jfw: ...has been exhumed
[02:14] jfw: it doesn't really look fixable by tweaking the numbers. there's no bound on the number of blocks in a reorg, and thus the number of database pages touched within the transaction, and transactions accumulate locks over the transaction lifetime.
[02:15] jfw: one idea is to break down the reorg, only processing one block per db transaction. basically making it a less stateful operation
[02:16] jfw: the trouble there is how to know the reorg is legit before you start reversing blocks
[02:19] jfw: this could perhaps be assured by fully checking blocks prior to admitting them into the database, i.e. no more hopeful blocks
[02:20] jfw: which might require changes to how transactions are indexed, at minimum indexing also the transactions from accepted but inactive blocks off the main chain, so that downstream blocks have what to reference
[02:21] jfw: and "where a transaction is spent" would no longer be an atom but a list of potentially many downstream blocks
[02:23] jfw: further there's the spamming issue where any number of transactions & blocks could be spewed downstream of the early chain where difficulty is low, and the node now has to index all those transactions rather than just storing the block data as it does already
[02:25] jfw: this together with the unreliable block download does seem to point to headers-first sync, i.e. show me the full proof of work chain that puts your blocks in the lead before I'm interested in what they contain
[02:27] jfw: but for a less invasive measure, I wonder if bdb locking & multithread support can't just be turned off altogether, not like we're doing any useful concurrency with it, pretty much everything is lock guarded anyway by the bitcoind code
[02:27] jfw: would have to validate that ofc
[02:49] jfw: with the contemplated change to transaction indexing, the reorg itself might get a lot faster, if all that's needed is updating the block index flags for which blocks are active
[02:49] jfw: and mempool updates
[05:33] dorion: jfw, I'm game to try turning off the multithread support.
[23:15] whaack: hm, noting here that per my recent lookup there are 5,751,194.63631089 coins in p2pkh addresses, however txstats.com reports ~7.8million
Day changed to 2024-03-05
[00:52] jfw: whaack: do they show that over time? how much does it vary?
[05:33] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10659 - so far so good on checking this: calls to ProcessMessage are all guarded by cs_main; likewise ProcessBlock and all RPC commands. CheckDiskBlocks is called only from CDB::LoadBlockIndex, in turn from AppInit2 i.e. at startup in the main thread before creating any other threads.
[05:33] sourcerer: 2024-03-04 02:27:14 (#jwrd) jfw: but for a less invasive measure, I wonder if bdb locking & multithread support can't just be turned off altogether, not like we're doing any useful concurrency with it, pretty much everything is lock guarded anyway by the bitcoind code
[05:48] jfw: and with the GUI absent, network messages, rpc commands and startup are really the only 'events' that could trigger any sort of action at all by the node, far as I can think
[05:51] jfw: there are the 'scheduled tasks' like ResendWalletTransactions but that's run by SendMessages, which - also holds cs_main.
[05:55] jfw: a dubious one is ThreadFlushWalletDB, it uses cs_db but it's not clear that all database operations also hold that :/
[05:58] jfw: there's also Shutdown with its stupid DBFlush business.
[13:13] dorion: jfw, nice, sounds like good progress there.
Day changed to 2024-03-06
[18:36] dorion: jfw, http://jfxpt.com/paste/2ktmht9gkd -- those tmp db files you asked for.
[19:14] jfw: http://jfxpt.com/paste/izn4ix7fzn - how those look after turning off bdb locks.
[19:14] sourcerer: 2024-03-04 02:27:14 (#jwrd) jfw: but for a less invasive measure, I wonder if bdb locking & multithread support can't just be turned off altogether, not like we're doing any useful concurrency with it, pretty much everything is lock guarded anyway by the bitcoind code
[19:25] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10608 - I'm thinking it's not so bad as is, it doesn't really need a new 'panic' state, the existing safe mode should be adequate; so it would just need to set a warning string to trigger that when an I/O error comes up.
[19:25] sourcerer: 2024-03-03 19:17:25 (#jwrd) jfw: seems to me that if there's an I/O error on database operations, the node's only reasonable options are either to abort entirely or else enter some "cry for help" maintenance state
[19:26] jfw: and it's not really preferable to abort entirely
[19:28] jfw: and with that settled, it should be straightforward to deal with the runaway heap usage by Reorganize - connect the blocks first, update the mempool later by re-reading the blocks.
[23:04] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10676 -- makes sense.
[23:04] sourcerer: 2024-03-06 19:25:22 (#jwrd) jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10608 - I'm thinking it's not so bad as is, it doesn't really need a new 'panic' state, the existing safe mode should be adequate; so it would just need to set a warning string to trigger that when an I/O error comes up.
[23:05] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10678-- agreed.
[23:05] sourcerer: 2024-03-06 19:26:27 (#jwrd) jfw: and it's not really preferable to abort entirely
[23:05] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10679 -- sweet.
[23:05] sourcerer: 2024-03-06 19:28:46 (#jwrd) jfw: and with that settled, it should be straightforward to deal with the runaway heap usage by Reorganize - connect the blocks first, update the mempool later by re-reading the blocks.
Day changed to 2024-03-07
[04:30] whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10665 <-- they show the last 6 months, its gone down from 8.08 mil or so since then, interestingly it seemed ~flat until jan 1st, then has declined to 7.8mil. forget my reported number, its not accurate at all, i'll try to make the proper query/report later
[04:30] sourcerer: 2024-03-05 00:52:36 (#jwrd) jfw: whaack: do they show that over time? how much does it vary?
[19:01] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10676 - the madness goes deeper though because there's absolutely no distinguishing in the code between I/O or otherwise external system errors and actually invalid blocks/transactions.
[19:01] sourcerer: 2024-03-06 19:25:22 (#jwrd) jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10608 - I'm thinking it's not so bad as is, it doesn't really need a new 'panic' state, the existing safe mode should be adequate; so it would just need to set a warning string to trigger that when an I/O error comes up.
[19:08] jfw: also the "invalid chain found" is the ~only way to make it loudly signal the condition to the operator. so seems the only option for now is to leave everything under the blanket "invalid chain" even though the different error cases in Reorganize certainly *could* be distinguished
[22:50] dorion: jfw, I applied patch pile. here's the extract from debug.log and the memory consumption while reorg was being processed. the 503 block reorg was reconnected successfully and memory consumption never rose above ~590 MB !!111!
[23:07] jfw: BEAUTIFUL.
[23:24] jfw: as that ran the bdb log dir .bitcoin/database grew to 3.8G over ~400 log files, which was freed up when the db checkpoint ran sometime thereafter. it's a bit odd to me that we don't see any time taken per the debug.log for that checkpointing (archived - the next line following the ACCEPTED was an outbound ping at the
[23:24] jfw: same exact timestamp)
Day changed to 2024-03-08
[17:49] dorion: jfw, the node caught up with the ~8k blocks it had fallen behind fine. synced connecting to one node and now opened it up and have 8 peers. also ran a back up of blk****.dat and blkindex.dat files.
[18:09] jfw: dorion: and of 'database' from the same point as the rest?
[19:21] dorion: yeah.
Day changed to 2024-03-09
[21:11] jfw: http://jfxpt.com/2024/file-level-error-recovery-for-keksum/
Day changed to 2024-03-11
[00:58] jfw: in gales linux annoyances, busybox tar not preserving extracted directory timestamps.
[03:10] jfw: or rather, it does when encountering a record in the tar file for the directory but if any files contained in it are listed later (as they typically are), their creation bumps the directory mtime and tar doesn't restore it.
[04:05] jfw: search turns up the addition of a related TODO but no fixes.
[04:09] jfw: this suggests a workaround at least when the source filesystem is still available: "find . -type d -exec tar --no-recursion -cf dirtimes.tar {} +" then extract that after the main one with the file data
[04:11] jfw: ha, worked perfect for my present case of migrating a directory containing many years of projects.
[04:13] jfw: I suppose fixing it would require, gasp, a tar that uses memory, hence busybox hasn't done it.
[04:15] jfw: or it could be done with a two-pass scan of the tar file, but that would have significant overhead (especially if compressed) and prevent streaming usage
[04:17] jfw: ah well, onto the TODOs it goes.
[04:23] jfw: ooh, maybe it creates a temporary file in the destination as it goes, into which it copies the directory records instead of acting on them, then feeds that right back into the usual extractor.
[04:42] jfw: perhaps with the unix trick of unlinking the temp file as soon as it's opened, so it's automatically freed on either exit or abort.
[18:51] whaack: so...logging some notes on my exercise routines here.
[18:52] whaack: last week i had some bad flare ups, i had been going strong for about a month and dare i say improving in some ways, but perhaps i overtrained or something, i have a couple of theories as to why i had the flar eup
[18:53] whaack: 1) muscles were fatigued from so much use of grip-strength. just about all lifts require grip strength and perhaps the nerves that replenish the forearm muscles are pinch/working poorly so the grip needs extra time to recover (that i did not give it)
[18:55] whaack: 2) my right elbow is a particular sore point, seems i have some form of epicondylitis, so specifically exercises where i bend the elbow hurt me (again that's a lot of them, bench press, overhead press, lat pulldowns, bent over rows, etc etc
[18:55] whaack: 3) allergy to milk/dairy , i say this because i started supplementing with a 'milk shake' around the time pain creeped in
[18:56] whaack: i'm testing theory 3 today, so i'm resting and still drinking my supplements
[18:57] jfw: whaack: alright, how much recovery time do you normally take?
[18:57] whaack: a flare up takes like 2 days to calm down
[18:58] whaack: and a flare up = chronic burning in the forearm muscles
[18:58] jfw: ok, and between workouts normally? per theory 1
[18:58] whaack: i was working out almost 6 days a week, rotating muscle groups
[18:59] jfw: so the grip and elbows would be involved most every day?
[18:59] whaack: yes
[18:59] whaack: i hurt my foot as well, so the leg day rest also got scrapped
[19:00] whaack: (the calf thing i mentioned in the logs, that seems to be healing fine)
[19:01] jfw: right. sounds like that might not be helping, then, depending on how hard you're going. I do 3x a week, always a full rest day, which is what at least two programs I saw over the years (or was attracted to) recommended
[19:03] jfw: if you want more activity you could do cardio, stretching, yoga etc on the off days
[19:04] whaack: yup. i was more relaxed but then i got addicted to the idea of hypertrophy and i was seeing good results so pushed harder and harder
[19:04] whaack: so i guess i'll pull back for now
[19:05] jfw: addicted to an idea? heh
[19:05] whaack: lol. addicted to the process
[19:06] jfw: well, results are good motivation but gotta be sustainable, it's a marathon not a sprint and all that (though sprints can be a solid exercise too)
[19:06] jfw: gains come easier at first I gather
[19:06] whaack: word. 6 days a week was fine apart from the flareups
[19:11] jfw: but for reference on the 3-day, 5x5 routine - since starting at the gym 4 months ago now, I've gone from the empty 20kg bar to 60kg on back squats which I do every session
[19:11] jfw: upper body stuff goes much slower of course
[19:12] whaack: nice! It's pretty amazing what lifting does for you
[19:29] jfw: inb4 AMAZING COMPANY
[20:32] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10711 -- you can look for dynamometers to measure grip strength and start collecting and tracking your data to give you an idea if you're in an overtrained/fatigued state and then take it easier those days.
[20:32] sourcerer: 2024-03-11 18:53:45 (#jwrd) whaack: 1) muscles were fatigued from so much use of grip-strength. just about all lifts require grip strength and perhaps the nerves that replenish the forearm muscles are pinch/working poorly so the grip needs extra time to recover (that i did not give it)
[20:34] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10719 -- are you lifting every day ? or mixing in cardio, stretching, yoga etc days too ?
[20:34] sourcerer: 2024-03-11 18:58:43 (#jwrd) whaack: i was working out almost 6 days a week, rotating muscle groups
[20:34] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10725 -- heh.
[20:34] sourcerer: 2024-03-11 19:03:20 (#jwrd) jfw: if you want more activity you could do cardio, stretching, yoga etc on the off days
[20:37] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10726 -- key is patience and setting realistic expectations. there is only so much muscle you can add in X time and rest is really important part of the building process. are you putting as much focus on good sleep habits as you are on good workout routines ? as you're experiencing now, overtraining to injury causes the set backs.
[20:37] sourcerer: 2024-03-11 19:04:05 (#jwrd) whaack: yup. i was more relaxed but then i got addicted to the idea of hypertrophy and i was seeing good results so pushed harder and harder
[20:47] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10704 -- oh noes !!
[20:47] sourcerer: 2024-03-11 04:13:59 (#jwrd) jfw: I suppose fixing it would require, gasp, a tar that uses memory, hence busybox hasn't done it.
[21:44] whaack: im lifting everyday, but i always mix in a decent amount of stretching
[21:45] whaack: i also usually warm up with.cardio but due to my calf injury i stopped doing that , that may be a reason i had a bad flareup
[21:45] whaack: it would make sense if nerves are squished that having a higher heart rate would be a good idea before lifting
[21:46] whaack: jfw: is it ok to plug a usb stick into 'offline machine' provided usb stick forever stays offline?
[21:47] whaack: also getting gscm/gbw-signer on the offline machine requires the dioptic usb right?
[21:48] jfw: whaack: it certainly seems a low risk of data exfiltration if the usb stick stays forever offline - in the sense that you protect it physically along with the machine or else reduce it to NAND dust
[21:49] jfw: can't be ruled out that the 'foreign' chip could harbor some kernel exploit or w/e, though
[21:50] whaack: yes re sleep habits. i'm following advice from schwarzenegger's tomb on building muscle. nutrition + sleep are top priorities
[21:50] whaack: jfw: those were my thoughts
[21:51] jfw: but yes, in effect you'll need some 'breach of protocol' to finish the initial software load, much like installing the OS, so that's fine if the machine is still in its 'gestation'
[21:51] jfw: whaack: tomb or tome?
[21:52] whaack: 'here lies Arnold, who always prioritized diet and rest'
[21:52] whaack: it was..uh.. STT!
[21:52] jfw: hehe
Day changed to 2024-03-12
[16:47] whaack: jfw: im having trouble w. the data diode. after setting the baud rates, i run cat <filename> > /dev/ttyUSB0 on the transmitter machine, and then try to receive the data via cat /dev/ttyUSB0 > <filename>
[17:05] jfw: whaack: alright, and what happens?
[17:25] whaack: i get either nothing in the file or some random chars
[17:40] whaack: if i feed /dev/zero i get a block of zeros but not a continuous stream
[17:59] dorion: whaack, I typically have to play with the where exactly in the port the fiber is and also how tightly I tighten it. like tuning a radio. I feed in /dev/urandom on the transmit side and hexdump -Cv on the receive side while I'm tuning.
[17:59] dorion: s/with the where/with where/
[18:02] whaack: k ill give it another whirl, i did that b4 and was getting a nice stream of random bites, but still no luck with actual files
[18:03] whaack: is the process i outlined the correct way to obtain a file?
[18:14] jfw: whaack: it should work, yes; you have to interrupt the receiving 'cat' when the transfer is done since there's no signalling of such completion; maybe check also that you're setting the raw/noecho modes; another test stream generator you could try is "yes testing", then you can more easily see in the hexdump if it's transmitting correctly
Day changed to 2024-03-13
[03:45] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10763 - another possibility pops into mind: if there's another process (hexdump test or w/e) still reading from the device; the stream gets shared with each byte going only to one or the other reader
[03:45] sourcerer: 2024-03-12 17:25:58 (#jwrd) whaack: i get either nothing in the file or some random chars
Day changed to 2024-03-15
[20:10] jfw: http://jfxpt.com/2024/jwrd-logs-for-Feb-2024/#10378 - I noticed this function also requires some new standard library imports: from cStringIO import StringIO; import struct.
[20:10] sourcerer: 2024-02-18 18:46:46 (#jwrd) jfw: whaack: cmd_decode; afaik it should work against gbw-node_direct_block_read.vpatch, just put it in the script and link it into the cmds table at the end
[20:10] jfw: whaack, dorion: ^
Day changed to 2024-03-16
[02:41] dorion: jfw, thanks. are you planning to include it in the coming release ?
[17:43] jfw: dorion: if you mean the coming release of the bitcoind patches, no. if you mean the next batch of gbw-node work when it gets its turn, maybe. there's a lot building up on the list there.
[17:46] jfw: now if those who asked for the sneak preview will put it to use and send feedback, that could inform what goes into the release :)
[20:21] dorion: jfw, alright, makes sense.
[20:53] whaack: jfw: im embarrassed to ask, but can you send the paste again?
[20:54] whaack: also ive been playing with the diode cable..i cannot get that thing straight for the life of me. i get 90% of the 'testing ' transmission, but ive twiddled it for 20-30m, any tips?
[22:22] jfw: whaack: http://jfxpt.com/paste/4u2mbjmcet
[22:23] jfw: since I'd already been asked to re-post it elsewhere
[22:23] jfw: it does start to bug me to implement a frontend for the link archiver though.
[22:24] jfw: whaack: what means 90%, like it's fine for some interval then glitches?
[22:25] jfw: there's also the sensitivity adjustment screws on the boards, have you tried those? I turn them all the way "down", let me check which way that means
[22:27] jfw: to the left / counterclockwise
[22:27] whaack: I mean 9 out of 10 characters come through correctly. Sometimes the transmission just stops. It's also very difficult for me to get anything resembling the correct output
[22:28] whaack: Are those the screws you turn by hand?
[22:29] jfw: needs a pretty small flathead driver
[22:30] jfw: it's a potentiometer, to be precise
[22:30] whaack: Is it the one in the center?
[22:30] jfw: yeah, in between the fiber ports
[22:30] whaack: Alright I think I see it. I will try that and get back to you
Day changed to 2024-03-17
[18:27] jfw: http://jfxpt.com/2024/on-the-fact-that-bitcoind-fails-to-implement-the-bitcoin-protocol/
Day changed to 2024-03-18
[04:10] whaack: jfw: Nice write up. I've given it a pass but I'm going to have to chew on it. I wouldn't be surprised if we do or do not experience a major reorganization in our lifetime.
[05:30] jfw: whaack: thanks. could be one of those things where being ready for the potential crisis prevents it from materializing
[05:35] jfw: it also occurs to me that only now with my log patch do you have something to grep for to monitor the depth of reorg events.
[20:43] dorion: whaack, i don't have a return to mexico city confirmed yet. how's your schedule shaping up ?
[20:45] whaack: The best time for me to go is towards the end of April
[20:48] dorion: that might work for me, but can't commit right now.
[20:52] whaack: Alright, atm I plan to go regardless of whether or not you can make it. I should have flights booked by the end of the week
[20:52] whaack: I'm at checkout El Salvador as well
[20:52] whaack: may*
[20:58] whaack: I want to live in a city in Central America a couple months out of the year. I want to be in an environment that's more conducive to getting work done. It'll be either Panama or Mexico most likely.
[21:01] dorion: whaack, I think mexico is the best bet, especially starting out new. mexico city isn't the only place worth a look either. I'm in panama to make the most out of my network here, but the environment generally is becoming more of a drag.
[21:03] dorion: the annoying stuff I've been putting up with for all this time is wearing and getting worse.
[21:04] whaack: dorion: alright, you and Jacob's presence is a massive plus that would lead me to Panama, but if you're thinking of abandoning ship then I will start looking in Mexico.
[21:07] dorion: whaack, I get that and I'd like to have you here for sure. Our network here is a big plus and you're always welcome to visit, but I'd suggest shorter stays here. mexico will have a much better vibe, weather is way better for one and it's an actual city and on many level outcompetes NYC.
[21:08] whaack: That's great to hear, I'm excited to check it out
[21:08] dorion: I need to write an article already about my experience...
[21:09] dorion: whaack, if cdmx, look for places around chapultapec and ave reforma. a more pricey/touristy part of town, but good landing spot at least.
[21:11] dorion: in the 3 weeks I was there last month, I did a week each in cuauhtemoc, roma norte y polanco. condessa is quite nice as well, perhaps my fav.
[21:12] dorion: I liked bouncing around because gives different perspectives.
[21:12] dorion: and ofc I can share the contacts I met there too.
[21:13] whaack: Awesome. Maybe I will plan a couple week trip and check out those areas as well. And thanks, i will probably take you up on grabbing some of those contacts
[21:20] dorion: whaack, how's it going with your stuff otherwise ?
[21:23] dorion: have you gotten any further on the maths research jfw asked you about ?
[21:32] whaack: dorion: nope, I have been slacking. http://ztkfg.com/2024/03/moving-in-all-sorts-of-directions/ that's a reason for my desire to change my environment
[21:36] whaack: I can blame part of it on my hands, no matter what I do they always seem to be getting a little worse, the gym did help for a bit but as I mentioned in the logs I think I overdid it. But even with my crippled condition I should be doing way more. There are ways to work around it
[21:37] dorion: whaack, alright. change in environment/new beginnings can certainly be energizing, but as diana_coman advised me a few years ago, real, lasting change has to come from within.
[21:38] jfw: I might be able to do a mexico visit end of april
[21:40] whaack: I took a very brief skim of the stem cell text you sent me, unfortunately I really can't believe it will help. Another time I might elaborate on why I don't trust it, but the short version is that it is possible that my condition comes from the body producing the wrong type of tissue, and I could only see stem cells accelerating that process
[21:41] dorion: hm, myeah, better to be conservative. is there a name for the production of the wrong tissues ?
[21:42] whaack: I know this maybe a pessimistic view, but it seems to me that the mechanism in which stem cells help is not well understood, and my problem is not well understood
[21:42] whaack: Wallerian degeneration
[21:43] whaack: Essentially the nerve tissue gets damaged, and it grows insulating material in the place where there should be conductivity
[21:43] dorion: did you have tests to confirm that ? (sorry if I'm forgetting if you mentioned already).
[21:45] whaack: I have No confirmation, I've done one nerve test that showed that I had reduced conductivity, but I've had physical therapists tell me the tests are bullshit and I'm inclined to believe them
[21:46] whaack: brb food
Day changed to 2024-03-19
[01:05] whaack: the antecedent of them is the physical therapists, if that wasnt clear
[01:05] whaack: jfw: that's great, extra incentive for me to go
[22:42] whaack: jfw: I'm still having no luck with the diode. I tried adjusting that tiny knob like you said, I was able to but still could not get proper output
[22:42] whaack: I'm not sure how to proceed
Day changed to 2024-03-20
[03:37] jfw: whaack, does the problem occur in either directions? how about with a loopback test (looping the fiber from transmit to receive port of the same board, sending data in and reading it out the same tty device) ?
[03:37] jfw: perhaps one of the boards is defective
[04:00] whaack: jfw: oh dear, I thought there was only one board. Well, at first I was confused because having one board did not make much sense to me. But then I figured perhaps there was some isolation on the chip. And I didn't find the second board. In other words, I've been doing only a loopback test
[04:00] whaack: I just looked through my bag of jwrd goodies and found the second board, I will try again tomorrow morning
[04:12] jfw: ahaha, seems like the bigger question would have been 'how am I supposed to get the bits from point A to point B if it only plugs into one computer?!'
[17:48] jfw: I'll point out that local namespaces (aka classes or packages or what-have-you), unavailable in C but so much loved by C++, Python, CL, Scheme, Ada and other such langs aiming for "scalable to large system programming", are an absolute pain in the ass when it comes to searching a codebase to make global conclusions.
[17:50] jfw: "I don't have to worry about colliding with someone else's names" - yeah, because you push the costs multiple times over onto someone downwind.
[17:51] jfw: especially when you deliberately collide the names for the 'uniform interface' or whatever like in bitcoind as encouraged by c++ practice.
[17:55] jfw: basically, the busybox machinery is smarter than they knew: the grep program shouldn't be allowed+required to have a function named 'main', let it use 'grep_main' and the compiler enforces global non-collision
[17:57] jfw: scheme is in the middle because its primitives are powerful enough that it's easy to build class/namespace/module/package like machinery, out of lambda, symbols, macros and such, but it's not a required or pushed part of the base language
[18:04] jfw: recalling a trilema where MP said something much like this, seemingly in favor of scheme over common lisp in this aspect, but not finding.
[18:15] jfw: scheme however doesn't exactly provide support for loading a set of definitions into the top/global namespace with collision checking (redefinition of anything is allowed in the 'interaction environment'; you can request evaluation in standard environments but there top-level definition isn't allowed at all iirc)
[18:40] jfw: dorion: one question to perhaps feed into your road map is how to approach the mining code. perhaps it's useful as a working reference miner, even if not a viable hash rate competitor. it's pretty clear we want to cut out the wallet code, but there's some dependence there, eg the miner needs an
[18:40] jfw: address to pay the block reward which it gets from the wallet, so how far do we go with patching that over to keep the miner working? then, do we want to add support for what miners actually use (stratum protocol or w/e it is now)? approach known mining pools about advice & testing?
[18:44] jfw: and obviously that first roadmap doesn't have to be perfect, is expected to have open questions, get iterated and refined as we go
[18:44] whaack: jfw: I had two USB cables connected to the same board
[18:45] jfw: whaack: oh like one with power, ground & transmit, the other ground & recieve or some such?
[18:46] whaack: Yes. Except I did not ground both of them
[18:46] whaack: One had power, ground, and transit. The other just had receive
[18:47] jfw: dodgier still, ha. the confusions I didn't even know were possible!
[18:47] whaack: I'm going to go try again with the two boards. I guess I've contaminated my off-line machine to some degree
[18:48] jfw: so it wasn't a loopback in the sense of the same machine reading & writing?
[18:48] whaack: correct
[18:48] whaack: just same board
[18:49] jfw: I see. one side not having a ground to reference the signal against is certainly more than enough to account for the errors you were getting.
[18:51] jfw: on the bright side, that same flakiness makes it pretty unlikely that any attempted contamination would have succeeded, it seems to me.
[18:52] jfw: since whichever machine wasn't even properly communicating with the board.
[19:15] whaack: I'm still having troubl, with two boards
[19:20] whaack: argh 1sec
[19:22] whaack: got it. I forgot to set the baud rate
[20:49] jfw: http://jfxpt.com/2024/auditing-bitcoind-for-concurrent-database-objects-the-call-graph-from-hell/
Day changed to 2024-03-21
[05:10] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10561 - this desktop is now happily promoted to 'workstation' status, with its 16G of plain consumer DDR3 replaced with 32G ECC - AMD cpus & mobos ftw! at the linux 'EDAC' driver claims to recognize it and have ECC enabled.
[05:10] sourcerer: 2024-03-02 20:23:45 (#jwrd) jfw: the plan is to use the machine for a home server, on the idea that it's a lower bar to find an ISP adequate to connect fiber to our premises than to find one adequate to physically entrust our data and iron.
[05:11] jfw: *at least the linux...
[05:14] jfw: otherwise discussions are coming along well for the real estate to house it, the network to connect and the bureaucratic interfacing for the braindead ISPs who won't sell 'biz grade service' to individual without local biz papers
[05:16] jfw: s/16G/8G/ - whoops, bigger upgrade still.
[05:21] jfw: (and by 'without local biz papers' I mean more like 'utterly uninterested in getting local biz papers in 3rdworld landfill that already punished me once for playing along')
[21:51] whaack_: If I were to lose my gpg key/ gpg uid used for the gbw-signer, would I still be able to open the encrypted Wallet, assuming I have the password?
[21:51] whaack_: jfw: ^
[21:51] whaack_: I believe the answer is yes but I want to make sure
[21:54] dorion: whaack, what would the password decrypt if you didn't have the private key ?
[21:58] whaack: hm i guess i understood that the password itself was what created the cypher
[21:58] whaack: id be decrypting the wallet
[21:58] dorion: whaack, the password encrypts the priv key. if someone steals your privkey, but doesn't have your password, they'd have to brute force the password.
[21:59] dorion: the cipher texts are encrypted/decrypted by the privkey.
[21:59] whaack: ok ty
[22:03] whaack: how do i export an encrypted gpg priv key to a file s.t. it still requires the pw to unlock?
[22:37] whaack: nvm i figured it out
[22:59] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10875 - the password is used to symmetrically encrypt the asymmetric (rsa) private key, so no, the rsa key is not derived from the password at all.
[22:59] sourcerer: 2024-03-21 21:58:00 (#jwrd) whaack: hm i guess i understood that the password itself was what created the cypher
[23:00] jfw: gbw-signer could theoretically use a symmetric mode to encrypt the wallet, but then you'd have to type the password every time you saved (rather than just opened), and somehow verify it was typed right etc.
[23:02] jfw: the bitcoind internal wallet does this, by holding the password in memory basically.
[23:03] jfw: anyway I don't really see the problem, you have to back up the wallet anyway so just include the gpg key with that, it changes even less often.
Day changed to 2024-03-22
[05:36] jfw: http://jfxpt.com/2024/dropping-bdb-locking-bitcoind-finally-follows-the-bitcoin-protocol/
[06:00] jfw: dorion: if not using the miner or wallet, there's unlikely to be any problem resulting from the missed locking cases in the old version of bitcoin_drop_bdb_locking.vpatch, but best to update anyway.
[06:00] jfw: (the current one is the only signed one anyway.)
Day changed to 2024-03-23
[19:03] whaack: i'm having cold feet about mexico city
[19:03] whaack: specifically due to the air quality
[19:03] whaack: that's a dealbreaker for me, so now i'm toying with the idea of getting an aprtment in SJ in CR
[19:04] whaack: or perhaps Panama City
[19:16] jfw: I've heard mixed reports about mx air quality so open to seeing for myself
[19:20] jfw: PTY is anti-recommended for air quality right now and who knows how much longer. there's normally fires from time to time in the dry season but this year - apparently we didn't yet mention it here - the city's overflowing garbage dump caught (or was set) on fire back in january, and was eventually subdued but has been repeatedly flaring back up.
[19:21] jfw: this in turn means garbage pickup has been more backed up than usual, hence the landfill epithet is quite fair.
[19:21] sourcerer: 2024-03-21 05:21:55 (#jwrd) jfw: (and by 'without local biz papers' I mean more like 'utterly uninterested in getting local biz papers in 3rdworld landfill that already punished me once for playing along')
[19:25] whaack: yeah i may visit CDMX still but chances of me living there are looking grim, a good friend of mine was just there and he confirmed the air quality is a problem
Day changed to 2024-03-30
[00:15] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10844 - http://trilema.com/2018/on-namespaces/ of course.
[00:15] sourcerer: 2024-03-20 18:04:36 (#jwrd) jfw: recalling a trilema where MP said something much like this, seemingly in favor of scheme over common lisp in this aspect, but not finding.
Day changed to 2024-03-31
[16:42] jfw: on reread, the linked item doesn't support the "much like this" interpretation. what it does or doesn't support I'm not sure I quite grasp well enough to say
[17:39] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10842 - am I really arguing, as a sysadmin even, that everything has to spell out the whole absolute path, all the time? it would make that particular case easier of "find all the inbound references" i.e. doing the linker's work by hand
[17:39] sourcerer: 2024-03-20 17:55:33 (#jwrd) jfw: basically, the busybox machinery is smarter than they knew: the grep program shouldn't be allowed+required to have a function named 'main', let it use 'grep_main' and the compiler enforces global non-collision
[17:43] jfw: but all that really seems to say is that: we have to work harder than it seems we ought to, whether in this way or that way, because our tools suck.
[17:44] jfw: and why do they suck? because there's not enough of them or not enough code in them? that strains credulity; seems more likely there's something amiss at a fundamental level
[17:51] jfw: okay now this is the weirdest: arrow keys stopped working for my yrc-in-tmux, displaying as A, B, C and D as if the leading ESC [ codes are missing. other commands, control keys and otherwise seem to still work fine, and arrows work in other tmux windows
[17:57] jfw: if I type in full "ESC [ A", it works like the up key. so doesn't seem like it could be a problem in the yrc input state machine.
[18:04] jfw: by 'cat' test, I see that particular tmux window is now sending arrow keys with an ESC O prefix rather than ESC [, which apparently bash recognizes but yrc never heard of
[18:07] jfw: it certainly ain't the ANSI mode cursor movements
[18:41] jfw: looks like they're SS3 sequences which apply under DECCKM, a "DEC private mode" to make the cursor keys send "application functions" for I can't imagine what reason.
[18:54] jfw: sure enough xterm shows this mode state under VT Options (ctrl-middleclick) as "Enable Application Cursor Keys". but I don't have it on at the xterm level, and it seems invisible & inaccessible at the tmux level, no mention in the manual.
[18:58] jfw: it gets enabled for example if the host sends the terminal an "ESC [ ? 1 h". but yrc doesn't do this. mysterious
[19:16] jfw: what's more, in xterm's version the Home/End keys are similarly affected but in tmux's they aren't.
[19:18] jfw: it's easy enough to tweak yrc to accept all such forms, much as I'd be inclined to take out this stupid invisible mode from the terminal code...
[21:04] jfw: turns out not so easy, it's not just a matter of tweaking the keymaps because commands using this SS3 prefix don't actually fit the definition of control sequence per ECMA-48, so they're discarded at an earlier point, and would require extending the input state machine (but how? what's the general form of such sequences? I'm invited to go look in some other standard...)
[21:04] jfw: at least I can make its terminal initialization sequence reset that mode to known-good state on startup.
[21:14] jfw: though, F1-F4 also generate such SS3 sequences in practice so for general-purpose terminal input I'd have to dig it up anyway
[21:51] jfw: to quote, "SS3 is used for code extension purposes. It causes the meanings of the bit combinations following it in the data stream to be changed. The use of SS3 is defined in Standard ECMA-35." Turns out I had that doc already ("Character Code Structure and Extension Techniques"), right beside ECMA-48; it's a stupid PDF with no index or clickable refs. When I search for SS3, it comes up only in
[21:51] jfw: discussions of its coded representation and other such tediousness, and its usage to specify a graphical character set, i.e. on the output side, not input.
[21:53] jfw: so... it changes the meaning of the following bits in an unspecified way. great.
[22:21] jfw: and ecma-35's discussion of escape sequences only gets me as far as confirming that yes, ESC O is a representation of the SS3 character from the C1 set of supplementary control functions. and that's the end of the escape sequence, nothing to inform a parser of how many following characters of input are modified by it.
[22:35] jfw: well, at least the xterm doc says SS3 "affects next character only", the keys it lists using it match that pattern, and it seems to be in the spirit of the 'single shift' name. so, I can make the state machine handle that.
[01:28] jfw: dorion: try building with http://jfxpt.com/wp-content/uploads/2024/03/bitcoin_reorg_tracing.patch (based on my last, bitcoin_rebranding.vpatch) and run it through one reorg cycle, with the memory log going too for good measure
[01:29] jfw: mempool wrangling is one possible "leak" source but doesn't fit with the "one or a few big allocations at once" theory
[01:31] jfw: dorion: and get me the debug log from the "REORGANIZE" through the OOM exception.
[01:32] jfw: best if you can have it disconnected from peers while that runs.
[01:32] jfw: but I'm unsure if at least one new block is required to trigger the reorg
[05:02] dorion: alright, I had killed it ; will fire it back up, wait to see reorg and disconnect the network.
[18:55] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10498 - this appears to be so, and worse - if the new block is a bastard block, the afflicted node sends a 'getblocks' to try to catch up, so the peer sends a handful of inventory, which the afflicted node already has, just on its inactive, not-yet-connected reorg branch, so it doesn't ask for the blocks, so the
[18:55] sourcerer: 2024-03-01 01:32:46 (#jwrd) jfw: but I'm unsure if at least one new block is required to trigger the reorg
[18:55] jfw: continuation hash never triggers and no further inventory is sent, so the reorganize is never re-triggered.
[19:06] jfw: here btw is for the full linked dataset on memory use over 14 hours, where we can clearly see the node giving up on the reorg after 7 hours, presumably because it stopped getting fed new shiny blocks to chase.
[19:12] jfw: whaack: I have a keksum patch just about ready to fix the annoyance we ran into where it dies on the first error when you do something like 'keksum *' and there's subdirectories or otherwise unreadable files present (and fixing a blatant typo in the usage synopsis).
[19:15] jfw: I also started on a '-c' implementation i.e. to verify files against an existing hash listing, but this is turning out considerably more complicated (basically, parsing sucks and especially in C); it seems to me that the "make your own listing and diff them" remains an adequate solution for now but what do you think, would the builtin checking be very helpful? from the standpoint of the jwrd
[19:15] jfw: training client at least
[19:19] jfw: and dorion, have you run into that situation much (or does it perhaps hold back keksum adoption vs. sha*sum)?
[19:22] jfw: it's certainly doable, and with only a small bending of the "no buffered I/O or dynamic memory allocation" restrictions of keksum's current style, just going to be closer to the 200 LoC than 20 LoC mark.
[19:27] dorion: jfw, I can't say lacking the feature has held me back much and the extra step of using diff sounds like a 'good enough' alternative to have to rely on for now.
[20:02] jfw: in the reorg saga, a targeted 'eatblock' woke it up so we should get a clearer picture.
[20:05] jfw: whaack, we couldn't readily work out how to get a raw (binary) block out of your explorer; how would you suggest converting from hex? or something else?
[20:34] dorion: jfw, teh log , I trimmed out the connect/disconnect noise and added in the error that went to stdo, but not the log.
[20:43] dorion: for those following allow at home, bitcoind runs out of memory on a reorg attempt because it tries to connect 210 blocks at a time.. rather than just a few.
[20:43] dorion: or even a baker's dozen or w/e.
[20:43] dorion: s/allow/along/
Day changed to 2024-03-02
[03:55] whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10505 << I don't think this is helpful, it can already be easily done with existing tools. I was surprised when you showed me in class that such a feature existed for sha512sum
[03:55] sourcerer: 2024-03-01 19:15:58 (#jwrd) jfw: I also started on a '-c' implementation i.e. to verify files against an existing hash listing, but this is turning out considerably more complicated (basically, parsing sucks and especially in C); it seems to me that the "make your own listing and diff them" remains an adequate solution for now but what do you think, would the builtin checking be very helpful? from
[03:56] whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10511 << raw per my thought process meant the representation of the bits in hexadecimal, perhaps i need to change the wording and/or add an endpoint for the binary bits, sorry that you lost time for that
[03:56] sourcerer: 2024-03-01 20:05:53 (#jwrd) jfw: whaack, we couldn't readily work out how to get a raw (binary) block out of your explorer; how would you suggest converting from hex? or something else?
[04:18] dorion: whaack, we didn't lose too much time on it, no sweat. it was more an opportunity to test and when we couldn't figure it out, jfw just dumped the block we needed from his node and passed it over to me for eating.
[04:19] dorion: to test your explorer. btw, not sure if I ever mentioned, but while I'm thinking of it, I never got the push raw txn to work on your explorer.
[16:52] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10521 <-> http://jfxpt.com/2024/jwrd-logs-for-Feb-2024/#10488
[16:52] sourcerer: 2024-03-02 04:19:13 (#jwrd) dorion: to test your explorer. btw, not sure if I ever mentioned, but while I'm thinking of it, I never got the push raw txn to work on your explorer.
[16:52] sourcerer: 2024-02-29 18:39:04 (#jwrd) whaack: i still have to figure out what's wrong with the push command
[16:55] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10518 - there is that use of 'raw' as in 'raw transaction' indeed. it could certainly be dealt with client-side, just needs working out since it's not base64 or something a standard utility can eat directly. maybe some tricky 'od' recipe
[16:55] sourcerer: 2024-03-02 03:56:18 (#jwrd) whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10511 << raw per my thought process meant the representation of the bits in hexadecimal, perhaps i need to change the wording and/or add an endpoint for the binary bits, sorry that you lost time for that
[16:59] jfw: to add to the confusion though there's that "raw_data" checkbox, all it seems to do is remove the form header and links but is still returning html
[17:00] dorion: jfw, here's the memory view from the most recent bitcoind run : http://jfxpt.com/paste/gfhghneixu
[17:01] jfw: dorion: ty, looking.
[17:02] jfw: the log output from the reorg certainly looks much more helpful now, I'd say that patch is a keeper.
[17:02] dorion: de acuerdo.
[17:02] jfw: "Reorganize() : fork height=825321 with 0 to disconnect, 501 to connect
[17:03] jfw: " - seems odd that there would be none to disconnect, that just sounds like normal growth rather than fork/reorg
[17:10] jfw: the memory series in seconds with data archived alongside as before
[17:22] jfw: kind of inconsistent that uncaught exceptions from a peer network message get logged but those from rpc commands don't
[17:25] jfw: but assuming there wasn't anything missed in the log extract between the ConnectBlock messages and the exception, it seems to rule out the bdb commit as the culprit, since it never gets there, indeed only gets through 210 of the 501 blocks
[17:38] dorion: jfw, I don't think I did any excessive snipping of the log, but will double check.
[17:40] jfw: and 825321 is your current height, correct?
[17:43] jfw: ah no, it was 825821 (per my dump command history). a suspicious cooincidence with the 501 blocks to connect
[17:44] dorion: 825321 is where is it stuck. e.g. last SetBestChain and what getinfo reports.
[17:44] jfw: oh right.
[17:46] jfw: the 500 difference makes sense then at least.
[18:09] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10533 - this makes sense based on the code too: writing a block's data and adding it to the block index is done & committed prior to 'connecting' it (CBlock::SetBestChain). presumably the latter failed, recurringly, due to the disk I/O error. thereafter, it would continue recording the new blocks but being unable to connect. thus, the new block's
[18:09] sourcerer: 2024-03-02 17:03:10 (#jwrd) jfw: " - seems odd that there would be none to disconnect, that just sounds like normal growth rather than fork/reorg
[18:09] jfw: parent doesn't match the current best block, which is enough to get it treated as a reorg.
[18:11] jfw: so basically this is a preview of what might happen in a fork war with actual deep reorgs
[18:12] jfw: and what happens is that we get stuck
[19:06] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10513 - the logic here, such as it is, would be that the reorg needs to be tried as a whole in order to establish that the proposed chain is even valid
[19:06] sourcerer: 2024-03-01 20:43:36 (#jwrd) dorion: for those following allow at home, bitcoind runs out of memory on a reorg attempt because it tries to connect 210 blocks at a time.. rather than just a few.
[19:08] jfw: then an optimization would be to draw that line and commit at the earliest PoW-longer block which is not necessarily the tip of the new chain, and then things can proceed incrementally
[19:09] jfw: but perhaps more important would be to contain memory usage even for deep reorgs, because that optimization wouldn't solve it in general
[19:22] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10547 -- good thing we're working on it now then !!
[19:22] sourcerer: 2024-03-02 18:12:17 (#jwrd) jfw: and what happens is that we get stuck
[19:23] dorion: and once we get it solved on our side, would be interesting to investigate the extent to which prb deals with it.
[20:03] jfw: myeah, we're working on it at least but still in a reactive way
[20:06] jfw: and not saying that's wrong from the business standpoint, like, that I should have been fixing the unloved codebase instead of making money; more that we need to get more money flowing in its direction. for which I'm cautiously optimistic about the 'key management consulting'
[20:14] dorion: jfw, agreed. I think there may be some other creative ways too. need to investigate further though.
[20:16] jfw: in desktop rebuilding news, the Antec P101 arrived after a whopping 16 days in 'express' shipment from miami, 'vuelos diarios' doncha know; came well packaged and weighing a hearty 30+ lbs. I got the machine reassembled, fresh thermal paste and all that and all seems well so far, even boots Gales despite the UEFI infection of the motherboard.
[20:16] sourcerer: 2024-02-05 04:44:14 (#jwrd) jfw: looks like I'll go with Antec Performance Series P101 Silent, runner up Phanteks Enthoo Pro series PH-ES614PC_BK
[20:21] jfw: the background here was I shipped two desktops down to myself in panama back in 2018 or so, the cases got all bent up and the better one failed to run reliably; the hope was that the electronics were OK and just the connections were flaky due to the physical compromise of the enclosure.
[20:23] jfw: the plan is to use the machine for a home server, on the idea that it's a lower bar to find an ISP adequate to connect fiber to our premises than to find one adequate to physically entrust our data and iron.
[20:24] jfw: so far the search still isn't going well though.
[21:56] whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10527 <-- yeah i can see how that is quite confusing
[21:56] sourcerer: 2024-03-02 16:59:03 (#jwrd) jfw: to add to the confusion though there's that "raw_data" checkbox, all it seems to do is remove the form header and links but is still returning html
[23:11] whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10521 <-- yup, it does not work. i'm going to finally look into it.
[23:11] sourcerer: 2024-03-02 04:19:13 (#jwrd) dorion: to test your explorer. btw, not sure if I ever mentioned, but while I'm thinking of it, I never got the push raw txn to work on your explorer.
Day changed to 2024-03-03
[02:02] jfw: what a pile of stupid dependencies or limitations I hit when reproducing my last soft-RAID Gales server setup
[02:02] sourcerer: 2022-04-06 01:53:59 (#jwrd) jfw: to summarize my findings on the soft-RAID side: the current lilo build *can* load kernels from a RAID volume, but only if it's using the old metadata format version 0.90; the chief shortcoming of 0.90 compared to the current 1.x formats is a 2TB size limit; thus there's perhaps still some value in the old hack of making a separate small /boot partition.
[02:03] jfw: http://jfxpt.com/2022/jwrd-logs-for-Apr-2022/#3712 , http://jfxpt.com/2022/jwrd-logs-for-Apr-2022/#3713
[02:03] sourcerer: 2022-04-06 01:55:31 (#jwrd) jfw: of course in that case you'd need GPT (while if the other stuff didn't suck so hard you could just use the full drive with no partitioning)
[02:03] sourcerer: 2022-04-06 01:58:01 (#jwrd) jfw: further, the kernel is able to auto-assemble arrays so as to mount a root FS on RAID without needing to bother with an initramfs - but only with 0.90 format. this applies even if you use the explicit syntax to tell the kernel the array structure (eg "md=0,/dev/sda1,/dev/sdb1").
[02:05] jfw: meanwhile the initramfs is built by a util in the linux tree, building linux still demands bc, which demands m4, which demands autoconf, which demands perl
[02:05] sourcerer: 2022-04-06 06:18:13 (#jwrd) jfw: so now they're included as /etc/examples/initramfs.*, provide more helpful comments, and actually work on gales; in combination with the new mdadm port I've now got the server assembling its RAIDs unassisted; now there just remains to relocate the root FS onto the array.
[02:07] jfw: and having mdadm as a gport rather than in the base or at least included with the installer made for a rather towers-of-hanoi like situation where I've had to build that port in the installer's tmpfs multiple times in order to access the target filesystem
[02:12] jfw: the 'bc' thing is one relatively low hanging and important fruit here I'd think: the base system should be able to build its own damn kernel at the very least, given that gcc is included.
[02:13] jfw: running 'make menuconfig' requires ncurses too, there's not likely to be a cure for that, but at least you can build from existing configs without it.
[02:16] jfw: packaging the initramfs utility (usr/gen_init_cpio) outside of the linux tree or finding an alternative might not be too hard either
[02:27] whaack_: hm, just did a successful push from the command line version of bitcoindexplorer
[02:31] whaack_: dorion: any chance you saved the error message from your attmept?
[02:54] dorion: whaack, no I didn't save and don't recall, sorry. perhaps jfw recalls what it said.
[03:01] whaack_: dorion: i have a hunch to what the problem is
[03:02] whaack_: push *should* be working on the NZ mirror http://103.6.212.28/
[03:08] whaack_: it seems the user that apache is using doesn't have permission to do a bitcoin rpc call, not sure why that's true for only 1 of my servers
[03:11] dorion: alright, well, seems like you're getting closer in any case. nice.
[03:25] whaack_: i think it should work now
[03:25] whaack_: will try next time i need to do a tx
[03:32] dorion: cool.
[03:42] dorion: whaack, while you're back to the explorer work, you may want to consider jfw's comment from back when you released it. perhaps a regrind of the genesis before you get to far on the current ? also, are you going to create a new key now that you have some practice with the airgap ?
[03:42] sourcerer: 2022-02-16 03:03:59 (#jwrd) jfw: whaack: re http://ztkfg.com/2022/02/trbexplorer-genesis-release/ , since you seem to indeed take the stance that your thing is changed entirely, with no attempt at maintaining compatibility etc - which I reckon is for the best since we're not really collaborating on it - I can't for the life of
[03:43] dorion: in any case, perhaps you'd like to register in our [http://jfxpt.com/2023/grand-reopening-of-jwrd-the-irc-channel/?b=If you wish to register&e=#select][protowot] as well.
[17:26] whaack: dorion: i will regrind, and also rename to bitcoindexplorer from trbexplorer
[17:27] whaack: my current gpg key is airgapped. its security is in question only b.c. it's 2048 bit iirc and it was seeded using a linux distros urandom rather than a TRNG
[17:29] whaack: lol 'how to airgap a practical guide' on goolag search does not return trilema on the front page of results. i'm used to trilema being hard-coded out but usually it shows up if the string is right. it does show up if you search 'how to airgap. a practical guide' (note the added .)
[17:29] whaack: newz @ 11
[18:04] whaack: left hand is hurting for the first time in a bit, i mean they always hurt a little nowadays but this time i am considering stopping typing because of it. specifically it's hurting in the carpal tunnel of the palm side, as i type now though it seems to be ok
[18:37] jfw: whaack: bummer. in other time & keystroke saving ideas though, I was looking at how to streamline the gport build & install steps, and what seems to stick so far is to make a "smarter" gpkg-install
[18:38] whaack: thx. i'm pushing through it. it seems to have abadated
[18:38] jfw: where it can take a package file as before, or the port directory containing build.sh, from which it determines the package file name, and launches gbuild if necessary
[18:40] jfw: and can also stash the result to /gales/dist/pkg/
[18:47] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10590 - did you type that url by hand or something? normally when copied out of the browser spaces and such get properly %-encoded
[18:47] sourcerer: 2024-03-03 03:43:54 (#jwrd) dorion: in any case, perhaps you'd like to register in our [http://jfxpt.com/2023/grand-reopening-of-jwrd-the-irc-channel/?b=If you wish to register&e=#select][protowot] as well.
[18:47] jfw: (I'm seeing how the url isn't detected as desired by the bot)
[18:54] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10516 - thanks, and I recall that reaction now that you mention. and keksum makes it even easier to verify a single file hash programatically, since when fed from stdin it prints just the hash without the extra filename field to cut out
[18:54] sourcerer: 2024-03-02 03:55:02 (#jwrd) whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10505 << I don't think this is helpful, it can already be easily done with existing tools. I was surprised when you showed me in class that such a feature existed for sha512sum
[18:59] jfw: atm I'm back to looking into the reorganize fail.
[19:13] jfw: one issue here is that block read errors aren't really adequately handled
[19:15] jfw: yeah, the function returns a success status so the calling code dutifully checks it but then all it does is log the error and give up on the reorg as if it were an invalid block
[19:17] jfw: seems to me that if there's an I/O error on database operations, the node's only reasonable options are either to abort entirely or else enter some "cry for help" maintenance state
[19:18] jfw: where it could perhaps still answer some local queries but won't pretend to still be following the blockchain
[19:30] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10591 -- cool.
[19:30] sourcerer: 2024-03-03 17:26:18 (#jwrd) whaack: dorion: i will regrind, and also rename to bitcoindexplorer from trbexplorer
[19:30] jfw: obviously that would be more complex codewise and harder to get right (noting how the existing 'safe mode' misses the mark)
[19:30] jfw: ("cry for help" mode, that is)
[19:30] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10592 -- was it always airgapped or did you move it over at a later stage ?
[19:30] sourcerer: 2024-03-03 17:27:23 (#jwrd) whaack: my current gpg key is airgapped. its security is in question only b.c. it's 2048 bit iirc and it was seeded using a linux distros urandom rather than a TRNG
[19:32] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10593 -- when I'm looking for a trilema article, i typically go straight to prepending with 'site:trilema.com' ; myeah, notreallynoose google sux.
[19:32] sourcerer: 2024-03-03 17:29:35 (#jwrd) whaack: lol 'how to airgap a practical guide' on goolag search does not return trilema on the front page of results. i'm used to trilema being hard-coded out but usually it shows up if the string is right. it does show up if you search 'how to airgap. a practical guide' (note the added .)
[19:33] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10595 -- sorry to hear that.
[19:33] sourcerer: 2024-03-03 18:04:47 (#jwrd) whaack: left hand is hurting for the first time in a bit, i mean they always hurt a little nowadays but this time i am considering stopping typing because of it. specifically it's hurting in the carpal tunnel of the palm side, as i type now though it seems to be ok
[19:33] whaack: dorion: if at any point in time it was not airgapped, i would not consider the key airgapped
[19:34] whaack: my gpg key has, to the best of my knowledge, never been present on any device with an active internet connection
[19:34] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10600 -- no, I didn't hand type and confirmed again that's how it pastes on my system.
[19:34] sourcerer: 2024-03-03 18:47:00 (#jwrd) jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10590 - did you type that url by hand or something? normally when copied out of the browser spaces and such get properly %-encoded
[19:36] jfw: testing over the shoulder shows the behavior I expected so nfi.
[19:37] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10620 -- good to confirm we see it the same way. curious how you ferried data back and forth prior to the optical data diode ?
[19:37] sourcerer: 2024-03-03 19:33:41 (#jwrd) whaack: dorion: if at any point in time it was not airgapped, i would not consider the key airgapped
[19:38] whaack: dorion: usb stick. this is a sore point as well
[19:38] dorion: I see.
[19:39] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10608 -- makes sense to me.
[19:39] sourcerer: 2024-03-03 19:17:25 (#jwrd) jfw: seems to me that if there's an I/O error on database operations, the node's only reasonable options are either to abort entirely or else enter some "cry for help" maintenance state
[19:49] dorion: whaack, did you start to read that stem cell book at all ? the treatment dropped a good bit since you last considered it and while that's likely to continue you're not getting any younger either.
[19:50] dorion: ***dropped good bit in price in satoshi terms.
[19:57] whaack: dorion: no, maybe i'll give it a flip through today.
[20:01] dorion: alright. no pressure or anything, but, while a bit outdated, that text gives a decent overview of the field.
[20:17] whaack: dorion: i forget, did you try the treatment yourself?
[21:19] dorion: not yet, don't have anything acute, but am curious. q2 or q3 I expect to.
[21:40] jfw: dorion: let's load your bitcoind under gdb, do 'catch throw', run it disconnected, and feed the next block to trigger reorg. that ought to get us a stack trace, unless there's too many unrelated exceptions going on to push through in normal control flow but I'd think not
[21:40] jfw: besides the stack trace it should let us see RAM usage prior to the unwinding.
[21:43] jfw: this is for the higher level question, given that we know there's a slow leak via mempool bookkeeping, of why it fails even with plenty of RAM still available.
[23:13] jfw: from that experiment: debug log, gdb session, memory data (now with explicit time field) and
[23:13] jfw: visual looking the same as usual
[23:33] jfw: the txid seen in that stack trace translates to 6bf00af44faf92918bd335b42c1b668f23132235dce743adef8378bd4dd45d40 - dorion, can you resume the node and try a getrawtransaction on it, just to see if maybe it's something triggered by reading that specific item from your db?
[23:35] dorion: jfw, on it.
[23:50] jfw: worked fine except needed -disablesafemode (because of the broken error handling which treated the I/O error the same as an invalid block)
[23:50] sourcerer: 2024-03-03 19:15:38 (#jwrd) jfw: yeah, the function returns a success status so the calling code dutifully checks it but then all it does is log the error and give up on the reorg as if it were an invalid block
[23:52] dorion: for the log : 0100000001afd1a3182539803bec0098be560dacfce655a5afe5653846c52405bed29f85460000000000ffffffff0272af0100000000001976a914b63927ce18471fb052fdcae3c5d71f7fed25247388aca8340000000000001600140307cf037216c44306b544206f74f27d53c3e40900000000
Day changed to 2024-03-04
[00:26] dorion: from .bitcoin/db.log I have a bunch of "Lock table is out of available object entries"
[00:35] jfw: got that clue by way of Troubleshooting common Berkeley DB problems which some wrecker apparently removed in my version of the reference
[00:42] jfw: it would appear the 'maxint locks' fork/wedge saga with magic numbers handed down from MP himself
[00:42] jfw: ...has been exhumed
[02:14] jfw: it doesn't really look fixable by tweaking the numbers. there's no bound on the number of blocks in a reorg, and thus the number of database pages touched within the transaction, and transactions accumulate locks over the transaction lifetime.
[02:15] jfw: one idea is to break down the reorg, only processing one block per db transaction. basically making it a less stateful operation
[02:16] jfw: the trouble there is how to know the reorg is legit before you start reversing blocks
[02:19] jfw: this could perhaps be assured by fully checking blocks prior to admitting them into the database, i.e. no more hopeful blocks
[02:20] jfw: which might require changes to how transactions are indexed, at minimum indexing also the transactions from accepted but inactive blocks off the main chain, so that downstream blocks have what to reference
[02:21] jfw: and "where a transaction is spent" would no longer be an atom but a list of potentially many downstream blocks
[02:23] jfw: further there's the spamming issue where any number of transactions & blocks could be spewed downstream of the early chain where difficulty is low, and the node now has to index all those transactions rather than just storing the block data as it does already
[02:25] jfw: this together with the unreliable block download does seem to point to headers-first sync, i.e. show me the full proof of work chain that puts your blocks in the lead before I'm interested in what they contain
[02:27] jfw: but for a less invasive measure, I wonder if bdb locking & multithread support can't just be turned off altogether, not like we're doing any useful concurrency with it, pretty much everything is lock guarded anyway by the bitcoind code
[02:27] jfw: would have to validate that ofc
[02:49] jfw: with the contemplated change to transaction indexing, the reorg itself might get a lot faster, if all that's needed is updating the block index flags for which blocks are active
[02:49] jfw: and mempool updates
[05:33] dorion: jfw, I'm game to try turning off the multithread support.
[23:15] whaack: hm, noting here that per my recent lookup there are 5,751,194.63631089 coins in p2pkh addresses, however txstats.com reports ~7.8million
Day changed to 2024-03-05
[00:52] jfw: whaack: do they show that over time? how much does it vary?
[05:33] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10659 - so far so good on checking this: calls to ProcessMessage are all guarded by cs_main; likewise ProcessBlock and all RPC commands. CheckDiskBlocks is called only from CDB::LoadBlockIndex, in turn from AppInit2 i.e. at startup in the main thread before creating any other threads.
[05:33] sourcerer: 2024-03-04 02:27:14 (#jwrd) jfw: but for a less invasive measure, I wonder if bdb locking & multithread support can't just be turned off altogether, not like we're doing any useful concurrency with it, pretty much everything is lock guarded anyway by the bitcoind code
[05:48] jfw: and with the GUI absent, network messages, rpc commands and startup are really the only 'events' that could trigger any sort of action at all by the node, far as I can think
[05:51] jfw: there are the 'scheduled tasks' like ResendWalletTransactions but that's run by SendMessages, which - also holds cs_main.
[05:55] jfw: a dubious one is ThreadFlushWalletDB, it uses cs_db but it's not clear that all database operations also hold that :/
[05:58] jfw: there's also Shutdown with its stupid DBFlush business.
[13:13] dorion: jfw, nice, sounds like good progress there.
Day changed to 2024-03-06
[18:36] dorion: jfw, http://jfxpt.com/paste/2ktmht9gkd -- those tmp db files you asked for.
[19:14] jfw: http://jfxpt.com/paste/izn4ix7fzn - how those look after turning off bdb locks.
[19:14] sourcerer: 2024-03-04 02:27:14 (#jwrd) jfw: but for a less invasive measure, I wonder if bdb locking & multithread support can't just be turned off altogether, not like we're doing any useful concurrency with it, pretty much everything is lock guarded anyway by the bitcoind code
[19:25] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10608 - I'm thinking it's not so bad as is, it doesn't really need a new 'panic' state, the existing safe mode should be adequate; so it would just need to set a warning string to trigger that when an I/O error comes up.
[19:25] sourcerer: 2024-03-03 19:17:25 (#jwrd) jfw: seems to me that if there's an I/O error on database operations, the node's only reasonable options are either to abort entirely or else enter some "cry for help" maintenance state
[19:26] jfw: and it's not really preferable to abort entirely
[19:28] jfw: and with that settled, it should be straightforward to deal with the runaway heap usage by Reorganize - connect the blocks first, update the mempool later by re-reading the blocks.
[23:04] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10676 -- makes sense.
[23:04] sourcerer: 2024-03-06 19:25:22 (#jwrd) jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10608 - I'm thinking it's not so bad as is, it doesn't really need a new 'panic' state, the existing safe mode should be adequate; so it would just need to set a warning string to trigger that when an I/O error comes up.
[23:05] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10678-- agreed.
[23:05] sourcerer: 2024-03-06 19:26:27 (#jwrd) jfw: and it's not really preferable to abort entirely
[23:05] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10679 -- sweet.
[23:05] sourcerer: 2024-03-06 19:28:46 (#jwrd) jfw: and with that settled, it should be straightforward to deal with the runaway heap usage by Reorganize - connect the blocks first, update the mempool later by re-reading the blocks.
Day changed to 2024-03-07
[04:30] whaack: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10665 <-- they show the last 6 months, its gone down from 8.08 mil or so since then, interestingly it seemed ~flat until jan 1st, then has declined to 7.8mil. forget my reported number, its not accurate at all, i'll try to make the proper query/report later
[04:30] sourcerer: 2024-03-05 00:52:36 (#jwrd) jfw: whaack: do they show that over time? how much does it vary?
[19:01] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10676 - the madness goes deeper though because there's absolutely no distinguishing in the code between I/O or otherwise external system errors and actually invalid blocks/transactions.
[19:01] sourcerer: 2024-03-06 19:25:22 (#jwrd) jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10608 - I'm thinking it's not so bad as is, it doesn't really need a new 'panic' state, the existing safe mode should be adequate; so it would just need to set a warning string to trigger that when an I/O error comes up.
[19:08] jfw: also the "invalid chain found" is the ~only way to make it loudly signal the condition to the operator. so seems the only option for now is to leave everything under the blanket "invalid chain" even though the different error cases in Reorganize certainly *could* be distinguished
[22:50] dorion: jfw, I applied patch pile. here's the extract from debug.log and the memory consumption while reorg was being processed. the 503 block reorg was reconnected successfully and memory consumption never rose above ~590 MB !!111!
[23:07] jfw: BEAUTIFUL.
[23:24] jfw: as that ran the bdb log dir .bitcoin/database grew to 3.8G over ~400 log files, which was freed up when the db checkpoint ran sometime thereafter. it's a bit odd to me that we don't see any time taken per the debug.log for that checkpointing (archived - the next line following the ACCEPTED was an outbound ping at the
[23:24] jfw: same exact timestamp)
Day changed to 2024-03-08
[17:49] dorion: jfw, the node caught up with the ~8k blocks it had fallen behind fine. synced connecting to one node and now opened it up and have 8 peers. also ran a back up of blk****.dat and blkindex.dat files.
[18:09] jfw: dorion: and of 'database' from the same point as the rest?
[19:21] dorion: yeah.
Day changed to 2024-03-09
[21:11] jfw: http://jfxpt.com/2024/file-level-error-recovery-for-keksum/
Day changed to 2024-03-11
[00:58] jfw: in gales linux annoyances, busybox tar not preserving extracted directory timestamps.
[03:10] jfw: or rather, it does when encountering a record in the tar file for the directory but if any files contained in it are listed later (as they typically are), their creation bumps the directory mtime and tar doesn't restore it.
[04:05] jfw: search turns up the addition of a related TODO but no fixes.
[04:09] jfw: this suggests a workaround at least when the source filesystem is still available: "find . -type d -exec tar --no-recursion -cf dirtimes.tar {} +" then extract that after the main one with the file data
[04:11] jfw: ha, worked perfect for my present case of migrating a directory containing many years of projects.
[04:13] jfw: I suppose fixing it would require, gasp, a tar that uses memory, hence busybox hasn't done it.
[04:15] jfw: or it could be done with a two-pass scan of the tar file, but that would have significant overhead (especially if compressed) and prevent streaming usage
[04:17] jfw: ah well, onto the TODOs it goes.
[04:23] jfw: ooh, maybe it creates a temporary file in the destination as it goes, into which it copies the directory records instead of acting on them, then feeds that right back into the usual extractor.
[04:42] jfw: perhaps with the unix trick of unlinking the temp file as soon as it's opened, so it's automatically freed on either exit or abort.
[18:51] whaack: so...logging some notes on my exercise routines here.
[18:52] whaack: last week i had some bad flare ups, i had been going strong for about a month and dare i say improving in some ways, but perhaps i overtrained or something, i have a couple of theories as to why i had the flar eup
[18:53] whaack: 1) muscles were fatigued from so much use of grip-strength. just about all lifts require grip strength and perhaps the nerves that replenish the forearm muscles are pinch/working poorly so the grip needs extra time to recover (that i did not give it)
[18:55] whaack: 2) my right elbow is a particular sore point, seems i have some form of epicondylitis, so specifically exercises where i bend the elbow hurt me (again that's a lot of them, bench press, overhead press, lat pulldowns, bent over rows, etc etc
[18:55] whaack: 3) allergy to milk/dairy , i say this because i started supplementing with a 'milk shake' around the time pain creeped in
[18:56] whaack: i'm testing theory 3 today, so i'm resting and still drinking my supplements
[18:57] jfw: whaack: alright, how much recovery time do you normally take?
[18:57] whaack: a flare up takes like 2 days to calm down
[18:58] whaack: and a flare up = chronic burning in the forearm muscles
[18:58] jfw: ok, and between workouts normally? per theory 1
[18:58] whaack: i was working out almost 6 days a week, rotating muscle groups
[18:59] jfw: so the grip and elbows would be involved most every day?
[18:59] whaack: yes
[18:59] whaack: i hurt my foot as well, so the leg day rest also got scrapped
[19:00] whaack: (the calf thing i mentioned in the logs, that seems to be healing fine)
[19:01] jfw: right. sounds like that might not be helping, then, depending on how hard you're going. I do 3x a week, always a full rest day, which is what at least two programs I saw over the years (or was attracted to) recommended
[19:03] jfw: if you want more activity you could do cardio, stretching, yoga etc on the off days
[19:04] whaack: yup. i was more relaxed but then i got addicted to the idea of hypertrophy and i was seeing good results so pushed harder and harder
[19:04] whaack: so i guess i'll pull back for now
[19:05] jfw: addicted to an idea? heh
[19:05] whaack: lol. addicted to the process
[19:06] jfw: well, results are good motivation but gotta be sustainable, it's a marathon not a sprint and all that (though sprints can be a solid exercise too)
[19:06] jfw: gains come easier at first I gather
[19:06] whaack: word. 6 days a week was fine apart from the flareups
[19:11] jfw: but for reference on the 3-day, 5x5 routine - since starting at the gym 4 months ago now, I've gone from the empty 20kg bar to 60kg on back squats which I do every session
[19:11] jfw: upper body stuff goes much slower of course
[19:12] whaack: nice! It's pretty amazing what lifting does for you
[19:29] jfw: inb4 AMAZING COMPANY
[20:32] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10711 -- you can look for dynamometers to measure grip strength and start collecting and tracking your data to give you an idea if you're in an overtrained/fatigued state and then take it easier those days.
[20:32] sourcerer: 2024-03-11 18:53:45 (#jwrd) whaack: 1) muscles were fatigued from so much use of grip-strength. just about all lifts require grip strength and perhaps the nerves that replenish the forearm muscles are pinch/working poorly so the grip needs extra time to recover (that i did not give it)
[20:34] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10719 -- are you lifting every day ? or mixing in cardio, stretching, yoga etc days too ?
[20:34] sourcerer: 2024-03-11 18:58:43 (#jwrd) whaack: i was working out almost 6 days a week, rotating muscle groups
[20:34] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10725 -- heh.
[20:34] sourcerer: 2024-03-11 19:03:20 (#jwrd) jfw: if you want more activity you could do cardio, stretching, yoga etc on the off days
[20:37] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10726 -- key is patience and setting realistic expectations. there is only so much muscle you can add in X time and rest is really important part of the building process. are you putting as much focus on good sleep habits as you are on good workout routines ? as you're experiencing now, overtraining to injury causes the set backs.
[20:37] sourcerer: 2024-03-11 19:04:05 (#jwrd) whaack: yup. i was more relaxed but then i got addicted to the idea of hypertrophy and i was seeing good results so pushed harder and harder
[20:47] dorion: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10704 -- oh noes !!
[20:47] sourcerer: 2024-03-11 04:13:59 (#jwrd) jfw: I suppose fixing it would require, gasp, a tar that uses memory, hence busybox hasn't done it.
[21:44] whaack: im lifting everyday, but i always mix in a decent amount of stretching
[21:45] whaack: i also usually warm up with.cardio but due to my calf injury i stopped doing that , that may be a reason i had a bad flareup
[21:45] whaack: it would make sense if nerves are squished that having a higher heart rate would be a good idea before lifting
[21:46] whaack: jfw: is it ok to plug a usb stick into 'offline machine' provided usb stick forever stays offline?
[21:47] whaack: also getting gscm/gbw-signer on the offline machine requires the dioptic usb right?
[21:48] jfw: whaack: it certainly seems a low risk of data exfiltration if the usb stick stays forever offline - in the sense that you protect it physically along with the machine or else reduce it to NAND dust
[21:49] jfw: can't be ruled out that the 'foreign' chip could harbor some kernel exploit or w/e, though
[21:50] whaack: yes re sleep habits. i'm following advice from schwarzenegger's tomb on building muscle. nutrition + sleep are top priorities
[21:50] whaack: jfw: those were my thoughts
[21:51] jfw: but yes, in effect you'll need some 'breach of protocol' to finish the initial software load, much like installing the OS, so that's fine if the machine is still in its 'gestation'
[21:51] jfw: whaack: tomb or tome?
[21:52] whaack: 'here lies Arnold, who always prioritized diet and rest'
[21:52] whaack: it was..uh.. STT!
[21:52] jfw: hehe
Day changed to 2024-03-12
[16:47] whaack: jfw: im having trouble w. the data diode. after setting the baud rates, i run cat <filename> > /dev/ttyUSB0 on the transmitter machine, and then try to receive the data via cat /dev/ttyUSB0 > <filename>
[17:05] jfw: whaack: alright, and what happens?
[17:25] whaack: i get either nothing in the file or some random chars
[17:40] whaack: if i feed /dev/zero i get a block of zeros but not a continuous stream
[17:59] dorion: whaack, I typically have to play with the where exactly in the port the fiber is and also how tightly I tighten it. like tuning a radio. I feed in /dev/urandom on the transmit side and hexdump -Cv on the receive side while I'm tuning.
[17:59] dorion: s/with the where/with where/
[18:02] whaack: k ill give it another whirl, i did that b4 and was getting a nice stream of random bites, but still no luck with actual files
[18:03] whaack: is the process i outlined the correct way to obtain a file?
[18:14] jfw: whaack: it should work, yes; you have to interrupt the receiving 'cat' when the transfer is done since there's no signalling of such completion; maybe check also that you're setting the raw/noecho modes; another test stream generator you could try is "yes testing", then you can more easily see in the hexdump if it's transmitting correctly
Day changed to 2024-03-13
[03:45] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10763 - another possibility pops into mind: if there's another process (hexdump test or w/e) still reading from the device; the stream gets shared with each byte going only to one or the other reader
[03:45] sourcerer: 2024-03-12 17:25:58 (#jwrd) whaack: i get either nothing in the file or some random chars
Day changed to 2024-03-15
[20:10] jfw: http://jfxpt.com/2024/jwrd-logs-for-Feb-2024/#10378 - I noticed this function also requires some new standard library imports: from cStringIO import StringIO; import struct.
[20:10] sourcerer: 2024-02-18 18:46:46 (#jwrd) jfw: whaack: cmd_decode; afaik it should work against gbw-node_direct_block_read.vpatch, just put it in the script and link it into the cmds table at the end
[20:10] jfw: whaack, dorion: ^
Day changed to 2024-03-16
[02:41] dorion: jfw, thanks. are you planning to include it in the coming release ?
[17:43] jfw: dorion: if you mean the coming release of the bitcoind patches, no. if you mean the next batch of gbw-node work when it gets its turn, maybe. there's a lot building up on the list there.
[17:46] jfw: now if those who asked for the sneak preview will put it to use and send feedback, that could inform what goes into the release :)
[20:21] dorion: jfw, alright, makes sense.
[20:53] whaack: jfw: im embarrassed to ask, but can you send the paste again?
[20:54] whaack: also ive been playing with the diode cable..i cannot get that thing straight for the life of me. i get 90% of the 'testing ' transmission, but ive twiddled it for 20-30m, any tips?
[22:22] jfw: whaack: http://jfxpt.com/paste/4u2mbjmcet
[22:23] jfw: since I'd already been asked to re-post it elsewhere
[22:23] jfw: it does start to bug me to implement a frontend for the link archiver though.
[22:24] jfw: whaack: what means 90%, like it's fine for some interval then glitches?
[22:25] jfw: there's also the sensitivity adjustment screws on the boards, have you tried those? I turn them all the way "down", let me check which way that means
[22:27] jfw: to the left / counterclockwise
[22:27] whaack: I mean 9 out of 10 characters come through correctly. Sometimes the transmission just stops. It's also very difficult for me to get anything resembling the correct output
[22:28] whaack: Are those the screws you turn by hand?
[22:29] jfw: needs a pretty small flathead driver
[22:30] jfw: it's a potentiometer, to be precise
[22:30] whaack: Is it the one in the center?
[22:30] jfw: yeah, in between the fiber ports
[22:30] whaack: Alright I think I see it. I will try that and get back to you
Day changed to 2024-03-17
[18:27] jfw: http://jfxpt.com/2024/on-the-fact-that-bitcoind-fails-to-implement-the-bitcoin-protocol/
Day changed to 2024-03-18
[04:10] whaack: jfw: Nice write up. I've given it a pass but I'm going to have to chew on it. I wouldn't be surprised if we do or do not experience a major reorganization in our lifetime.
[05:30] jfw: whaack: thanks. could be one of those things where being ready for the potential crisis prevents it from materializing
[05:35] jfw: it also occurs to me that only now with my log patch do you have something to grep for to monitor the depth of reorg events.
[20:43] dorion: whaack, i don't have a return to mexico city confirmed yet. how's your schedule shaping up ?
[20:45] whaack: The best time for me to go is towards the end of April
[20:48] dorion: that might work for me, but can't commit right now.
[20:52] whaack: Alright, atm I plan to go regardless of whether or not you can make it. I should have flights booked by the end of the week
[20:52] whaack: I'm at checkout El Salvador as well
[20:52] whaack: may*
[20:58] whaack: I want to live in a city in Central America a couple months out of the year. I want to be in an environment that's more conducive to getting work done. It'll be either Panama or Mexico most likely.
[21:01] dorion: whaack, I think mexico is the best bet, especially starting out new. mexico city isn't the only place worth a look either. I'm in panama to make the most out of my network here, but the environment generally is becoming more of a drag.
[21:03] dorion: the annoying stuff I've been putting up with for all this time is wearing and getting worse.
[21:04] whaack: dorion: alright, you and Jacob's presence is a massive plus that would lead me to Panama, but if you're thinking of abandoning ship then I will start looking in Mexico.
[21:07] dorion: whaack, I get that and I'd like to have you here for sure. Our network here is a big plus and you're always welcome to visit, but I'd suggest shorter stays here. mexico will have a much better vibe, weather is way better for one and it's an actual city and on many level outcompetes NYC.
[21:08] whaack: That's great to hear, I'm excited to check it out
[21:08] dorion: I need to write an article already about my experience...
[21:09] dorion: whaack, if cdmx, look for places around chapultapec and ave reforma. a more pricey/touristy part of town, but good landing spot at least.
[21:11] dorion: in the 3 weeks I was there last month, I did a week each in cuauhtemoc, roma norte y polanco. condessa is quite nice as well, perhaps my fav.
[21:12] dorion: I liked bouncing around because gives different perspectives.
[21:12] dorion: and ofc I can share the contacts I met there too.
[21:13] whaack: Awesome. Maybe I will plan a couple week trip and check out those areas as well. And thanks, i will probably take you up on grabbing some of those contacts
[21:20] dorion: whaack, how's it going with your stuff otherwise ?
[21:23] dorion: have you gotten any further on the maths research jfw asked you about ?
[21:32] whaack: dorion: nope, I have been slacking. http://ztkfg.com/2024/03/moving-in-all-sorts-of-directions/ that's a reason for my desire to change my environment
[21:36] whaack: I can blame part of it on my hands, no matter what I do they always seem to be getting a little worse, the gym did help for a bit but as I mentioned in the logs I think I overdid it. But even with my crippled condition I should be doing way more. There are ways to work around it
[21:37] dorion: whaack, alright. change in environment/new beginnings can certainly be energizing, but as diana_coman advised me a few years ago, real, lasting change has to come from within.
[21:38] jfw: I might be able to do a mexico visit end of april
[21:40] whaack: I took a very brief skim of the stem cell text you sent me, unfortunately I really can't believe it will help. Another time I might elaborate on why I don't trust it, but the short version is that it is possible that my condition comes from the body producing the wrong type of tissue, and I could only see stem cells accelerating that process
[21:41] dorion: hm, myeah, better to be conservative. is there a name for the production of the wrong tissues ?
[21:42] whaack: I know this maybe a pessimistic view, but it seems to me that the mechanism in which stem cells help is not well understood, and my problem is not well understood
[21:42] whaack: Wallerian degeneration
[21:43] whaack: Essentially the nerve tissue gets damaged, and it grows insulating material in the place where there should be conductivity
[21:43] dorion: did you have tests to confirm that ? (sorry if I'm forgetting if you mentioned already).
[21:45] whaack: I have No confirmation, I've done one nerve test that showed that I had reduced conductivity, but I've had physical therapists tell me the tests are bullshit and I'm inclined to believe them
[21:46] whaack: brb food
Day changed to 2024-03-19
[01:05] whaack: the antecedent of them is the physical therapists, if that wasnt clear
[01:05] whaack: jfw: that's great, extra incentive for me to go
[22:42] whaack: jfw: I'm still having no luck with the diode. I tried adjusting that tiny knob like you said, I was able to but still could not get proper output
[22:42] whaack: I'm not sure how to proceed
Day changed to 2024-03-20
[03:37] jfw: whaack, does the problem occur in either directions? how about with a loopback test (looping the fiber from transmit to receive port of the same board, sending data in and reading it out the same tty device) ?
[03:37] jfw: perhaps one of the boards is defective
[04:00] whaack: jfw: oh dear, I thought there was only one board. Well, at first I was confused because having one board did not make much sense to me. But then I figured perhaps there was some isolation on the chip. And I didn't find the second board. In other words, I've been doing only a loopback test
[04:00] whaack: I just looked through my bag of jwrd goodies and found the second board, I will try again tomorrow morning
[04:12] jfw: ahaha, seems like the bigger question would have been 'how am I supposed to get the bits from point A to point B if it only plugs into one computer?!'
[17:48] jfw: I'll point out that local namespaces (aka classes or packages or what-have-you), unavailable in C but so much loved by C++, Python, CL, Scheme, Ada and other such langs aiming for "scalable to large system programming", are an absolute pain in the ass when it comes to searching a codebase to make global conclusions.
[17:50] jfw: "I don't have to worry about colliding with someone else's names" - yeah, because you push the costs multiple times over onto someone downwind.
[17:51] jfw: especially when you deliberately collide the names for the 'uniform interface' or whatever like in bitcoind as encouraged by c++ practice.
[17:55] jfw: basically, the busybox machinery is smarter than they knew: the grep program shouldn't be allowed+required to have a function named 'main', let it use 'grep_main' and the compiler enforces global non-collision
[17:57] jfw: scheme is in the middle because its primitives are powerful enough that it's easy to build class/namespace/module/package like machinery, out of lambda, symbols, macros and such, but it's not a required or pushed part of the base language
[18:04] jfw: recalling a trilema where MP said something much like this, seemingly in favor of scheme over common lisp in this aspect, but not finding.
[18:15] jfw: scheme however doesn't exactly provide support for loading a set of definitions into the top/global namespace with collision checking (redefinition of anything is allowed in the 'interaction environment'; you can request evaluation in standard environments but there top-level definition isn't allowed at all iirc)
[18:40] jfw: dorion: one question to perhaps feed into your road map is how to approach the mining code. perhaps it's useful as a working reference miner, even if not a viable hash rate competitor. it's pretty clear we want to cut out the wallet code, but there's some dependence there, eg the miner needs an
[18:40] jfw: address to pay the block reward which it gets from the wallet, so how far do we go with patching that over to keep the miner working? then, do we want to add support for what miners actually use (stratum protocol or w/e it is now)? approach known mining pools about advice & testing?
[18:44] jfw: and obviously that first roadmap doesn't have to be perfect, is expected to have open questions, get iterated and refined as we go
[18:44] whaack: jfw: I had two USB cables connected to the same board
[18:45] jfw: whaack: oh like one with power, ground & transmit, the other ground & recieve or some such?
[18:46] whaack: Yes. Except I did not ground both of them
[18:46] whaack: One had power, ground, and transit. The other just had receive
[18:47] jfw: dodgier still, ha. the confusions I didn't even know were possible!
[18:47] whaack: I'm going to go try again with the two boards. I guess I've contaminated my off-line machine to some degree
[18:48] jfw: so it wasn't a loopback in the sense of the same machine reading & writing?
[18:48] whaack: correct
[18:48] whaack: just same board
[18:49] jfw: I see. one side not having a ground to reference the signal against is certainly more than enough to account for the errors you were getting.
[18:51] jfw: on the bright side, that same flakiness makes it pretty unlikely that any attempted contamination would have succeeded, it seems to me.
[18:52] jfw: since whichever machine wasn't even properly communicating with the board.
[19:15] whaack: I'm still having troubl, with two boards
[19:20] whaack: argh 1sec
[19:22] whaack: got it. I forgot to set the baud rate
[20:49] jfw: http://jfxpt.com/2024/auditing-bitcoind-for-concurrent-database-objects-the-call-graph-from-hell/
Day changed to 2024-03-21
[05:10] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10561 - this desktop is now happily promoted to 'workstation' status, with its 16G of plain consumer DDR3 replaced with 32G ECC - AMD cpus & mobos ftw! at the linux 'EDAC' driver claims to recognize it and have ECC enabled.
[05:10] sourcerer: 2024-03-02 20:23:45 (#jwrd) jfw: the plan is to use the machine for a home server, on the idea that it's a lower bar to find an ISP adequate to connect fiber to our premises than to find one adequate to physically entrust our data and iron.
[05:11] jfw: *at least the linux...
[05:14] jfw: otherwise discussions are coming along well for the real estate to house it, the network to connect and the bureaucratic interfacing for the braindead ISPs who won't sell 'biz grade service' to individual without local biz papers
[05:16] jfw: s/16G/8G/ - whoops, bigger upgrade still.
[05:21] jfw: (and by 'without local biz papers' I mean more like 'utterly uninterested in getting local biz papers in 3rdworld landfill that already punished me once for playing along')
[21:51] whaack_: If I were to lose my gpg key/ gpg uid used for the gbw-signer, would I still be able to open the encrypted Wallet, assuming I have the password?
[21:51] whaack_: jfw: ^
[21:51] whaack_: I believe the answer is yes but I want to make sure
[21:54] dorion: whaack, what would the password decrypt if you didn't have the private key ?
[21:58] whaack: hm i guess i understood that the password itself was what created the cypher
[21:58] whaack: id be decrypting the wallet
[21:58] dorion: whaack, the password encrypts the priv key. if someone steals your privkey, but doesn't have your password, they'd have to brute force the password.
[21:59] dorion: the cipher texts are encrypted/decrypted by the privkey.
[21:59] whaack: ok ty
[22:03] whaack: how do i export an encrypted gpg priv key to a file s.t. it still requires the pw to unlock?
[22:37] whaack: nvm i figured it out
[22:59] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10875 - the password is used to symmetrically encrypt the asymmetric (rsa) private key, so no, the rsa key is not derived from the password at all.
[22:59] sourcerer: 2024-03-21 21:58:00 (#jwrd) whaack: hm i guess i understood that the password itself was what created the cypher
[23:00] jfw: gbw-signer could theoretically use a symmetric mode to encrypt the wallet, but then you'd have to type the password every time you saved (rather than just opened), and somehow verify it was typed right etc.
[23:02] jfw: the bitcoind internal wallet does this, by holding the password in memory basically.
[23:03] jfw: anyway I don't really see the problem, you have to back up the wallet anyway so just include the gpg key with that, it changes even less often.
Day changed to 2024-03-22
[05:36] jfw: http://jfxpt.com/2024/dropping-bdb-locking-bitcoind-finally-follows-the-bitcoin-protocol/
[06:00] jfw: dorion: if not using the miner or wallet, there's unlikely to be any problem resulting from the missed locking cases in the old version of bitcoin_drop_bdb_locking.vpatch, but best to update anyway.
[06:00] jfw: (the current one is the only signed one anyway.)
Day changed to 2024-03-23
[19:03] whaack: i'm having cold feet about mexico city
[19:03] whaack: specifically due to the air quality
[19:03] whaack: that's a dealbreaker for me, so now i'm toying with the idea of getting an aprtment in SJ in CR
[19:04] whaack: or perhaps Panama City
[19:16] jfw: I've heard mixed reports about mx air quality so open to seeing for myself
[19:20] jfw: PTY is anti-recommended for air quality right now and who knows how much longer. there's normally fires from time to time in the dry season but this year - apparently we didn't yet mention it here - the city's overflowing garbage dump caught (or was set) on fire back in january, and was eventually subdued but has been repeatedly flaring back up.
[19:21] jfw: this in turn means garbage pickup has been more backed up than usual, hence the landfill epithet is quite fair.
[19:21] sourcerer: 2024-03-21 05:21:55 (#jwrd) jfw: (and by 'without local biz papers' I mean more like 'utterly uninterested in getting local biz papers in 3rdworld landfill that already punished me once for playing along')
[19:25] whaack: yeah i may visit CDMX still but chances of me living there are looking grim, a good friend of mine was just there and he confirmed the air quality is a problem
Day changed to 2024-03-30
[00:15] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10844 - http://trilema.com/2018/on-namespaces/ of course.
[00:15] sourcerer: 2024-03-20 18:04:36 (#jwrd) jfw: recalling a trilema where MP said something much like this, seemingly in favor of scheme over common lisp in this aspect, but not finding.
Day changed to 2024-03-31
[16:42] jfw: on reread, the linked item doesn't support the "much like this" interpretation. what it does or doesn't support I'm not sure I quite grasp well enough to say
[17:39] jfw: http://jfxpt.com/2024/jwrd-logs-for-Mar-2024/#10842 - am I really arguing, as a sysadmin even, that everything has to spell out the whole absolute path, all the time? it would make that particular case easier of "find all the inbound references" i.e. doing the linker's work by hand
[17:39] sourcerer: 2024-03-20 17:55:33 (#jwrd) jfw: basically, the busybox machinery is smarter than they knew: the grep program shouldn't be allowed+required to have a function named 'main', let it use 'grep_main' and the compiler enforces global non-collision
[17:43] jfw: but all that really seems to say is that: we have to work harder than it seems we ought to, whether in this way or that way, because our tools suck.
[17:44] jfw: and why do they suck? because there's not enough of them or not enough code in them? that strains credulity; seems more likely there's something amiss at a fundamental level
[17:51] jfw: okay now this is the weirdest: arrow keys stopped working for my yrc-in-tmux, displaying as A, B, C and D as if the leading ESC [ codes are missing. other commands, control keys and otherwise seem to still work fine, and arrows work in other tmux windows
[17:57] jfw: if I type in full "ESC [ A", it works like the up key. so doesn't seem like it could be a problem in the yrc input state machine.
[18:04] jfw: by 'cat' test, I see that particular tmux window is now sending arrow keys with an ESC O prefix rather than ESC [, which apparently bash recognizes but yrc never heard of
[18:07] jfw: it certainly ain't the ANSI mode cursor movements
[18:41] jfw: looks like they're SS3 sequences which apply under DECCKM, a "DEC private mode" to make the cursor keys send "application functions" for I can't imagine what reason.
[18:54] jfw: sure enough xterm shows this mode state under VT Options (ctrl-middleclick) as "Enable Application Cursor Keys". but I don't have it on at the xterm level, and it seems invisible & inaccessible at the tmux level, no mention in the manual.
[18:58] jfw: it gets enabled for example if the host sends the terminal an "ESC [ ? 1 h". but yrc doesn't do this. mysterious
[19:16] jfw: what's more, in xterm's version the Home/End keys are similarly affected but in tmux's they aren't.
[19:18] jfw: it's easy enough to tweak yrc to accept all such forms, much as I'd be inclined to take out this stupid invisible mode from the terminal code...
[21:04] jfw: turns out not so easy, it's not just a matter of tweaking the keymaps because commands using this SS3 prefix don't actually fit the definition of control sequence per ECMA-48, so they're discarded at an earlier point, and would require extending the input state machine (but how? what's the general form of such sequences? I'm invited to go look in some other standard...)
[21:04] jfw: at least I can make its terminal initialization sequence reset that mode to known-good state on startup.
[21:14] jfw: though, F1-F4 also generate such SS3 sequences in practice so for general-purpose terminal input I'd have to dig it up anyway
[21:51] jfw: to quote, "SS3 is used for code extension purposes. It causes the meanings of the bit combinations following it in the data stream to be changed. The use of SS3 is defined in Standard ECMA-35." Turns out I had that doc already ("Character Code Structure and Extension Techniques"), right beside ECMA-48; it's a stupid PDF with no index or clickable refs. When I search for SS3, it comes up only in
[21:51] jfw: discussions of its coded representation and other such tediousness, and its usage to specify a graphical character set, i.e. on the output side, not input.
[21:53] jfw: so... it changes the meaning of the following bits in an unspecified way. great.
[22:21] jfw: and ecma-35's discussion of escape sequences only gets me as far as confirming that yes, ESC O is a representation of the SS3 character from the C1 set of supplementary control functions. and that's the end of the escape sequence, nothing to inform a parser of how many following characters of input are modified by it.
[22:35] jfw: well, at least the xterm doc says SS3 "affects next character only", the keys it lists using it match that pattern, and it seems to be in the spirit of the 'single shift' name. so, I can make the state machine handle that.