Fixpoint

Day changed to 2022-04-01
[05:30] jfw: meanwhile in another castle, I have a guy who wants to try running a node on a raspberry pi (32-bit ARM, hard-float, little-endian). the build is failing on openssl because its deranged build system is passing -m64 to gcc, probably because it just doesn't recognize the architecture and is misapplying predetermined settings from another.
[05:31] jfw: iirc on aarch64 (as seen in Rockchip) dorion hit a similar snag except in Boost
[14:13] dorion: jfw, sounds pretty annoying and you recall correctly, it was boost.
[16:48] jfw: funny how after all that talk about "TRB on Pogo", with heroic efforts to track down memory leaks so it could fit, contemplation of attack vectors presented by the USG.NTP dependence, etc... nobody actually got the thing to build on ARM ?!
[16:50] jfw: ofc it could be a verschlimmbessert gcc version here, or glibc or who knows what; we shall see.
[18:32] jfw: the problem was the obvious, tracing back to makefiles.vpatch of 2016 vintage
[19:16] jfw: this particular Pi probably won't cut it, as it has only 512 MB RAM, and is now struggling with g++ memory usage on compiling bitcoinrpc.cpp. even just the in-memory block index is going to be tight at best.
[19:19] jfw: the situation's shown me another mark against tmux: it goes very unresponsive under this memory pressure. more specifically, as unresponsive as the programs running under it, which have the excuse that they actually need disk ops while the terminal is just supposed to be a damn state machine.
[19:19] jfw: this is with swap disabled, and ye olde gnu screen works fine.
[19:29] jfw: caai: to get closer to the root of your thinkpad problems, an easy & cheap step would be to replace the DRAM modules, because memory errors are the most likely explanation for the observed corruption. unfortunately the machines don't support error correcting (ECC) RAM, but there's some luck of the draw in how error-prone a module is in the first place; it's not well understood but one theory I saw
[19:29] jfw: is varying trace amounts of radioactive elements in the chip packaging.
[19:31] jfw: in theory any DDR3 SODIMM up to 4G ought to work; in practice there's fine print, and I've found Hynix 4GB PC3-10600 DDR3 1333MHz to work. you'd just need one of those to match your current capacity, or get two to double it.
[19:32] jfw: and you'll need a philips head #0 ("jeweller's") screwdriver to get at it; a handy thing to have in any case if you don't yet.
[23:47] jfw: http://fixpoint.welshcomputing.com/2022/spec-for-a-usable-ip-tunneling-system/
Day changed to 2022-04-02
[15:58] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3629 -- that was the whole point of the buildroot orchestra as I understand.
[15:58] sourcerer: 2022-04-01 16:50:56 (#jwrd) jfw: ofc it could be a verschlimmbessert gcc version here, or glibc or who knows what; we shall see.
[15:59] dorion: I gather you're not using that here, why not ? I get 100% why not in Gales, but why not in this case ?
[16:22] dorion: I can see the argument for trying out the build first w/o orchestra, why go through it all if you don't need to.
[17:05] jfw: dorion: because I killed that crawling horror two years ago and good riddance.
[17:12] jfw: in this case, the task was to try working with this imperfect yet available & working sliver of computer, as it is, and see what we learn. what I learned is that the build was broken because of an "x86_64" magic string introduced by the TRB people, NOT because of "heathen gcc" or the boost, bdb, or openssl configuration systems, or any part of the c & c++ code; nonetheless it was an easy diagnosis
[17:12] jfw: with 1-line fix within a relatively well known codebase.
[17:14] jfw: compared with this, buildroot is *an entire linux distribution* of its own right, which can *and has* failed to build even on relatively similar systems for a much wider array of possible causes.
[17:17] jfw: if we find it worthwhile to cut out mystery gcc's and the like, the approach would be to pick back up the Gales on ARM work and use that in place of buildroot.
[17:20] jfw: fwiw I thought the primary goal of 'rotor' was to eventually produce bitwise-identical binaries for auditing purposes.
[17:23] jfw: dorion: hope that clarifies; I was kinda surprised to see we weren't on the same page there.
[18:21] dorion: jfw, yeah, my bad. I don't know why I was confused there.
[18:22] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3644 -- interesting. so do you have it built now ?
[18:22] sourcerer: 2022-04-02 17:12:55 (#jwrd) jfw: in this case, the task was to try working with this imperfect yet available & working sliver of computer, as it is, and see what we learn. what I learned is that the build was broken because of an "x86_64" magic string introduced by the TRB people, NOT because of "heathen gcc" or the boost, bdb, or openssl configuration systems, or any part of the c & c++ code; nonetheless it was an easy diagnosis
[18:23] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3647 -- right, on the same page there.
[18:23] sourcerer: 2022-04-02 17:17:21 (#jwrd) jfw: if we find it worthwhile to cut out mystery gcc's and the like, the approach would be to pick back up the Gales on ARM work and use that in place of buildroot.
[19:04] jfw: ah, no it consistently bombed out on lack of memory for compiling bitcoin sources. I tried a couple basic tweaks but no luck; it's just not enough Pi.
[19:06] jfw: I suppose this could be labeled a gcc problem since wtf, why should it need a gig just to compile 2k lines of code... but moreover I don't think it's worth it because it will be too tight to run the node.
[19:08] jfw: the latest 'raspberry pi 4' generation reportedly has models with much more RAM, fwiw.
Day changed to 2022-04-03
[19:30] jfw: In unsurprising but still aggravating progress, github no longer supports git.
[19:33] jfw: (their proposed workaround of using "https" urls for anonymous read access isn't supported by git itself, it requires a "curl" dependency in which I'm not especially interested in adding.)
[19:33] jfw: *dependency which
[19:34] jfw: (a curl which itself must pull in openssl, obviously)
Day changed to 2022-04-04
[16:35] jfw: continuing from http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Mar-2022/#3623 - it didn't work; the node is still stuck in safe mode at the same height; and we confirmed the binary edit was done correctly by re-extracting the block and checking its hash.
[16:35] sourcerer: 2022-03-31 18:29:55 (#jwrd) jfw: once successful, resume the node and we'll see if it worked.
[16:42] jfw: caai: to complete the data gathering, can you please repeat this exercise in log extraction to track down the relevant error? except using http://welshcomputing.com/paste/ instead
[16:42] sourcerer: 2022-03-18 21:09:42 (#jwrd) jfw: can you do: "grep -FC 10 REORGANIZE ~/.bitcoin/debug.log > reorgs.log"
[16:43] jfw: in those grep options, -F means to match literal text rather than regexp, which may help performance given the large file, while -C means "context" i.e. includes the 10 lines before and after each match.
[19:30] jfw: in Gales Linux news, I'm trying it out on a leased dedicated server from Hetzner - a German provider and one of the very few such automated farms whose shit managed to actually work in my browser.
[19:34] jfw: earlier I got it working fine after the requisite kernel driver tweaks on an older Opteron server with 3ware RAID card, but that horse hasn't found a stable yet. the added challenges on this one are dealing with software RAID (and LVM while I'm at it) and an NVMe SSD.
[19:40] jfw: so I'm working on gports for mdadm and lvm2 and looking into lilo's troubles with both the device-mapper-encumbered HDDs and the SSD as previously reported.
[19:42] jfw: since lilo's come up I grabbed its git history and might have a look at the changes between 24.0 and 24.2, maybe rebase my patches as needed.
[19:45] jfw: on the nvme issue, I'd say this "rlaanemets" has a solid point - the linux people did it wrong according to their own docs (and never updated the docs). probably have to roll with it though.
[19:48] jfw: the seeming origin of the slackware patch that bvt picked up
[20:35] jfw: lilo dude didn't do the best job of explaining the "why" - and at times even the "what" - of his changes, or referencing external discussions.
[20:36] jfw: but boy did he update all those copyright dates like a good little worker bee.
[20:57] jfw: probably the most significant change from first skimming is also the latest, "Use system derived types to make code safe for both the 32-bit and the 64-bit compilation environment. (thanks to TAMUKI Shoichi)", yet it's not at all clear what was unsafe or how, and the change introduces inconsistency and doesn't fully do what it purports (eg int is still assumed to be 32 bits)
[22:38] jfw: there's also a bulk update of its known major device IDs - but since that didn't help with the actually popular NVMe, I expect it was driven by those dubious kernel docs rather than experience.
[22:43] jfw: after a fresh look at the complexity of the lvm codebase I'm rather loosing my appetite for it. and nothing that I wanted from it *can't* be done through other means; perhaps with performance penalties but that isn't known.
[22:43] jfw: *losing
[22:45] jfw: and for lilo to boot from it requires it to be built with libdevmapper from said lvm tree, so it's pulled into the gales bootstrap too.
[22:47] jfw: I suppose I'll stick with a classic MBR partition table for now, bite the bullet on supporting GPT when the time comes, and use a plain /boot partition outside of the RAID volume.
[22:48] jfw: and this path doesn't lock me into a decision on LVM vs GPT anyway.
[22:52] jfw: and for lilo, I'll take the minimal step of adapting the nvme patch since I took the time to look at it.
Day changed to 2022-04-05
[00:10] jfw: sooo, that slackware lilo nvme patch is broken, and not just in comment text like "set to none zero" : the usage of strcmp in setting the nvme_pr flag is backwards from what was intended, it concludes that nvme is present when it sees any string *other* than "nvme" in /proc/devices
[00:13] jfw: the search for that string itself is broken as of this commit, which was okay because "I can't test but I sincerely hope nobody's doing that".
[00:14] jfw: and to be fair to that Neil Brown, I'd also say the whole nvme_pr[esent] thing is stupid, because it's looking whether nvme is present somewhere in the kernel and using that to conclude that a particular device is nvme.
[00:15] jfw: The difference between Gales and ~every other distributor, folks.
[00:26] jfw: the "int dm_major_list[32]; /* increased from 16 to allow for nvme disks */" is also total nonsense.
[01:20] jfw: the rest is a copypasta of the prior stanza ("MASK15"), and it's quite unclear to me that the derivation of the BIOS drive number has any basis in reality, for a system with multiple drives.
[01:23] jfw: in fact it's clearly wrong since it'll come up with the same device number for the first nvme disk as for the first scsi disk.
[03:40] jfw: the lilo commit that introduced the error is perplexing too, nominally it's adding some GPT support but I don't see where it's doing anything at all besides misinterpreting the linux docs which already existed
[03:40] jfw: and introducing that strange error which invites the user to fuck off and try a totally different bootloader.
[03:42] jfw: the top part where it appears to assign MAJOR_SATA1 and MAJOR_SATA2 a 4-bit partition mask is unused, it's in a "#if BETA_TEST" block that's not enabled now and wasn't enabled at the time.
[03:57] jfw: lilo.conf(5) under "static-BIOS-codes" suggests that the BIOS drive number derivation I critiqued is merely guesswork and "often fails on mixed IDE/SCSI systems" anyway... gah. because totally, that's how you code, doesn't matter if it works or not as long as it slows down the reader.
[04:00] jfw: on the bright side, thus relieved of the burdens of doing anything right, I should be able to make a simple patch.
[05:04] jfw: such as this; testing the build now by way of a full re-bootstrap for good measure.
[16:15] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3677 -- ok. good that you looked at it. makes sense not to incorporate the unnecessary complexity.
[16:15] sourcerer: 2022-04-04 22:43:44 (#jwrd) jfw: after a fresh look at the complexity of the lvm codebase I'm rather loosing my appetite for it. and nothing that I wanted from it *can't* be done through other means; perhaps with performance penalties but that isn't known.
[16:27] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3683 -- good catches.
[16:27] sourcerer: 2022-04-05 00:10:11 (#jwrd) jfw: sooo, that slackware lilo nvme patch is broken, and not just in comment text like "set to none zero" : the usage of strcmp in setting the nvme_pr flag is backwards from what was intended, it concludes that nvme is present when it sees any string *other* than "nvme" in /proc/devices
[16:33] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3691 -- pretty bizarre.
[16:33] sourcerer: 2022-04-05 03:40:28 (#jwrd) jfw: and introducing that strange error which invites the user to fuck off and try a totally different bootloader.
[16:49] jfw: oh hey, there's a zlib compressor bug making the rounds with a 17-year exposure window... and the Gales version just so happens to be unaffected.
[16:51] jfw: unfortunately some stuff embeds its own copy of zlib; I tried to eliminate these but some were broken by the old version iirc.
[16:54] jfw: such as mysql
[16:56] jfw: ah, the better reference re zlib & mysql
[16:57] jfw: that fixpoint search box is starting to be pretty handy.
[19:03] jfw: and this "Ryzen" server with NVMe drive certainly rips; 'perl' gport built in 58 seconds, 3.9x faster than my thinkpad (X200).
[19:04] jfw: (thinkpad with SATA SSD)
[20:04] jfw: lilo patch is a success, ran without issue & rebooted on its own two feet. (this was a blind remote boot situation - no room for mistakes.)
[22:33] jfw: http://fixpoint.welshcomputing.com/2022/the-root-filesystem-and-mount-options-weirdness/
Day changed to 2022-04-06
[01:53] jfw: to summarize my findings on the soft-RAID side: the current lilo build *can* load kernels from a RAID volume, but only if it's using the old metadata format version 0.90; the chief shortcoming of 0.90 compared to the current 1.x formats is a 2TB size limit; thus there's perhaps still some value in the old hack of making a separate small /boot partition.
[01:55] jfw: of course in that case you'd need GPT (while if the other stuff didn't suck so hard you could just use the full drive with no partitioning)
[01:58] jfw: further, the kernel is able to auto-assemble arrays so as to mount a root FS on RAID without needing to bother with an initramfs - but only with 0.90 format. this applies even if you use the explicit syntax to tell the kernel the array structure (eg "md=0,/dev/sda1,/dev/sdb1").
[02:02] jfw: I kinda miss the days before I knew about partition tables, when it seemed that drive C: was just drive C: and that's all there was to it...
[06:00] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3707 << and the full scripted gales bootstrap in 5m33s, not bad.
[06:00] sourcerer: 2022-04-05 19:03:57 (#jwrd) jfw: and this "Ryzen" server with NVMe drive certainly rips; 'perl' gport built in 58 seconds, 3.9x faster than my thinkpad (X200).
[06:13] jfw: I've adapted the simple initramfs setup files I was using on gentoo; they happened to ship in the Gales source tree without being referenced anywhere, though they were the starting point for what became the installer initramfs machinery.
[06:18] jfw: so now they're included as /etc/examples/initramfs.*, provide more helpful comments, and actually work on gales; in combination with the new mdadm port I've now got the server assembling its RAIDs unassisted; now there just remains to relocate the root FS onto the array.
[06:34] jfw: http://welshcomputing.com/paste/d << the very shiny "df -hT", for those following from home.
[06:36] jfw: and /proc/mdstat
[14:22] caai: jfw: fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3662: Alright. I will do so.
[14:27] caai: I see that one must include http: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3662
[14:27] sourcerer: 2022-04-04 16:35:47 (#jwrd) jfw: continuing from http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Mar-2022/#3623 - it didn't work; the node is still stuck in safe mode at the same height; and we confirmed the binary edit was done correctly by re-extracting the block and checking its hash.
[14:42] caai: please note the final result: http://welshcomputing.com/paste/f | As you can see, I ran 'tail -n 1000' on the reorgs.log
[16:11] jfw: thanks caai.
[16:13] jfw: indeed the http is necessary
[16:13] sourcerer: 2020-11-17 18:29:20 (#jwrd) jfw: as a side note, plz not to skimp on the http: prefix for links - it's the only reliable way for a text scanner to identify them
[16:16] jfw: caai: the interesting thing there is the dates are from February; indeed it's identical to the previous log extract.
[16:26] jfw: caai: so assuming you didn't re-paste the previous file by accident or something, the current problem is different from the previous, and the reorganize code isn't involved this time
[16:31] jfw: caai: let's try a broader search term for the log, and this time add in the trick to skip the old bulk of the log to start with a smaller search space...
[16:32] jfw: hm, thought there was an example here already but perhaps it was elsewhere.
[16:42] jfw: tail -c 4000m ~/.bitcoin/debug.log | grep -FC 10 'received: block' > recv-blocks.log
[16:43] jfw: hopefully 4 gigs are enough to get something, if not then we've overflowed the arithmetics of this 'tail' implementation :D
[16:44] jfw: and again it'll be best to paste just the most recent results via "tail -n 1000 recv-blocks.log"
Day changed to 2022-04-07
[13:52] caai: jfw: Yes, that is very interesting that the dates are from February. Before creating this log extract file, I moved the previous one into a separate directory that I created outside of the .bitcoin directory, which I named '~/bcrepairdir'
[13:55] caai: When I run ls -l, I see that the date of the old log in ~/bcrepairdir is MAR 20 | the date of the new log in ~/.bitcoin is APR 6
[14:03] caai: please note the tail results: http://welshcomputing.com/paste/10
[14:07] caai: the results for recv.blocks-tail.log
[14:56] caai: jfw/dorion: my suggestion for the April schedule: Management: April 20 - 23:00 UTC | Operator: April 27 - 23:00 UTC
[16:20] jfw: caai: rats, foiled again by the BASTARD BLOCKs.
[16:32] caai: those BASTARD BLOCKS!
[16:32] jfw: ok, here's one that should filter those out while still capturing the context for whatever else is going on: tail -c 4000m ~/.bitcoin/debug.log | grep -Fv BASTARD | grep -FC 10 ProcessBlock
[16:33] jfw: redirect into a temporary file named as you see fit, then this time let's not cut it to the last 1000 lines, but do 'gzip' it before pasting.
[16:34] jfw: caai: since you're around, I'll stand by.
[16:35] jfw: the proposed schedule works for me, I take it the order is swapped from last month.
[17:35] caai: sorry, I had to step away. I am running it now
[17:41] caai: please note the result: http://welshcomputing.com/paste.11
[17:46] caai: Yes, I thought it best to change the order this month in order to allow me to study the new material more, becuase I have almost solely been focusing on the node repair during my study time. In addition, I think Management tasks are more urgent at this juncture.
[18:01] jfw: caai: sounds good. corrected link: http://welshcomputing.com/paste/11 , and a good thing we gzipped it.
[18:02] jfw: so basically the poor node hasn't received anything besides bastards since returning from surgery.
[18:03] jfw: I'll post the one that it appears to need for a manual feed (eatblock), see if that resuscitates it.
[18:07] jfw: caai: http://fixpoint.welshcomputing.com/wp-content/uploads/2022/04/blk721421
[18:08] jfw: wget that then feed it using 'bitcoind eatblock $PWD/blk721421', again, the $PWD will expand it to the absolute path which is needed because bitcoind may have been started in a different working directory.
[18:10] jfw: 721421 is the next block after the last that it received in the 'invalid chain' episode, according to the posted log.
[18:11] jfw: thus, the hypothesis is that it has all the blocks up to that one, just 720921 was corrupted and once another reorg is triggered it'll chug right through them.
[18:13] jfw: dorion: I'm thinking to start a node syncing on the new server, since that's (now) easily done and will take a while but will give us a known-probably-good node e.g. that we can have people -connect to in case of p2p network weirdness like this one.
Day changed to 2022-04-08
[13:11] caai: jfw: Good morning! Sounds good.
[13:15] caai: I just ran wget to download blk721421 and I received the following message: wget: server returned error: HTTP/1.1 403 Forbidden
[14:31] caai: I see that the ip address matches what I have in /etc/hosts, but there is an ':80' at the end
[15:04] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3756 -- cool.
[15:04] sourcerer: 2022-04-07 18:13:48 (#jwrd) jfw: dorion: I'm thinking to start a node syncing on the new server, since that's (now) easily done and will take a while but will give us a known-probably-good node e.g. that we can have people -connect to in case of p2p network weirdness like this one.
[17:14] jfw: caai: d'oh, I missed the read permissions again. try now
[17:17] jfw: not sure why 'dumpblock' results are ending up with those permissions, I've tried adding a 'umask 022' to my run script but no difference.
[17:24] jfw: ah, that would do it.
Day changed to 2022-04-09
[04:30] caai: jfw: I downloaded the block successfully, then I started the node. After that, I ran 'bitcoind eatblock $PWD/blk721421'. The resultant output: 'false'
[04:31] caai: I then stopped the node because I assumed that wasn't a positive sign
[05:26] jfw: caai: can you dig deeper, e.g. look in the log from the time of running the command?
[05:28] jfw: supposedly 'false' means it already had the block which doesn't seem to fit with the observed logs
[15:30] caai: Upon running the eatblock command, the log reads: Attempting to create block #720921 from file /home/user/.bitcoin/blk721421
[17:29] jfw: caai: and then??
Day changed to 2022-04-10
[05:54] jfw: dorion: you asked about my tmux.conf, and on working in a fresh machine I'm reminded of just how much better it makes things. http://fixpoint.welshcomputing.com/wp-content/uploads/2022/04/tmux.conf
[05:56] jfw: I might go so far as to say we should standardize & teach it this way; ctrl-b is such a terrible default.
[05:59] jfw: see how the F-keys suit you though.
[12:03] caai: jfw: I assumed that the best way to ascertain what occurred upon feeding the block was to create a temp file that displayed the tail of the log and search said file for 'blk721421', then view the succeeding lines (i.e., how the node responded upon feeding it block 721421)
[12:09] caai: Rather than try to explain it, I have uploaded a deedbot paste: welshcomputing.com/paste/12
[12:14] caai: You can see that I ran the command once on 04/08 and once on 04/09.
[12:17] caai: http://welshcomputing.com/paste/12
[19:11] jfw: caai: that is more helpful for sure (despite the 10MB haystack that it comes in), thank you.
[19:13] jfw: what I see is that you ran it twice on the 8th, thus on what was actually the second and third tries you got the 'false' because it already had the block. the first one is the relevant part, where it attempts to reorganize.
[19:23] jfw: and from the messages there, I conclude that the 'surgery' was technically a success in that the "ERROR: CheckBlock() : hashMerkleRoot mismatch" is gone; unfortunately something else is still amiss and we're even more in the dark than last time because there's no further detail prior to "ERROR: Reorganize() : ConnectBlock failed".
[19:26] jfw: conceivably some other clues are that it correctly figures the height of the newly-fed block, shows its hash as the "invalid block", and no perceptible time elapsed since the 'eatblock' and this result.
[19:27] jfw: so I figure it's still failing on the original problem block 720921, and it just means the new one is invalid because it builds on that 'invalid' chain.
[19:30] jfw: I also see I messed up my last patch by missing the damn newline in the edited log message.
[19:35] jfw: caai, dorion: we could perhaps continue along this line, further studying the code, adding more debug output and database inspection RPCs or whatnot, in hopes of tracking down the current problem; but strategically thinking, I'd say the situation is different from before: we no longer have nearly as much cause to suspect anything amiss in the bitcoin code, since it's been proven that the machine
[19:35] jfw: flipped at least one bit; and where there's one, there could well be more, especially considering the witnessed kernel crash.
[19:37] jfw: Attempting the surgery made sense to me because I had to track down the corruption anyway to prove the machine at fault, so once we knew the exact problem we might as well try fixing it. But now we've got no such further intel.
[19:39] jfw: So my thought is that it's time to abort; let caai get that replacement memory module at minimum, zap ~/.bitcoin/blk*.dat and restart the sync.
[19:39] sourcerer: 2022-04-01 19:31:54 (#jwrd) jfw: in theory any DDR3 SODIMM up to 4G ought to work; in practice there's fine print, and I've found Hynix 4GB PC3-10600 DDR3 1333MHz to work. you'd just need one of those to match your current capacity, or get two to double it.
[19:40] jfw: doing the proper sync should make a decent "burn-in" test for the new memory, too.
[19:41] jfw: perhaps we can supply a block collection to accelerate things through eatblock.
[19:43] jfw: thoughts?
[19:47] jfw: I suppose doing the full Gales bootstrap & reinstall first wouldn't hurt either, since it's coming up in the training.
Day changed to 2022-04-11
[14:08] caai: jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3779: Yes, I suppose I did run it twice on the 8th.
[14:08] sourcerer: 2022-04-10 19:13:15 (#jwrd) jfw: what I see is that you ran it twice on the 8th, thus on what was actually the second and third tries you got the 'false' because it already had the block. the first one is the relevant part, where it attempts to reorganize.
[14:23] caai: I don't mind purchasing the new memory(RAM). Shall I procure 2? I would prefer syncing the node from the genesis block. I know that it will take time, but I am curious to observe the process from the beginning.
[14:52] jfw: caai: if it were me I'd get two for sure, even if just to have the extra on hand.
[18:56] jfw: dorion: the question of "large random" vs "small but predictable" IDs for the pastebin has been simmering. a rule of thumb for the "birthday problem" would be that if you want odds of 1 in 2^J of any collision existing in a set of 2^K items, each of 2^L randomly chosen bits, then L must be at least J+2K.
[18:57] jfw: er, each of L randomly chosen bits, so a space of 2^L possibile distinct items.
[19:00] jfw: so 50 bits would get ~ 1 in a thousand odds of collision (10 bits) among a million IDs (20 bits); going with base32 to avoid mixed case, that would give 10 digits for URLs like http://welshcomputing.com/paste/477yuify81 , which all seems pretty reasonable to me
[19:03] jfw: or on the more conservative end it'd be one in a million odds of collision among 32k items
[19:20] jfw: or from another angle, if you had 32k items live at a time, the amount of trawling needed for one in a thousand odds of hitting anything at all would be 32M requests. certainly a noisy attack, though not necessarily prohibitive on bandwidth alone.
[19:24] jfw: another two digits would bring that to 32G requests.
[19:29] jfw: of course, nothing stops us from starting at 10 digits and lengthening in the event it actually found some kind of high-volume usage.
Day changed to 2022-04-12
[01:55] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3799 -- thanks for spelling it out. 10 digit base32 endpoint sounds good to me. big enough key space to start yet still typeable if/when needed.
[01:55] sourcerer: 2022-04-11 19:00:30 (#jwrd) jfw: so 50 bits would get ~ 1 in a thousand odds of collision (10 bits) among a million IDs (20 bits); going with base32 to avoid mixed case, that would give 10 digits for URLs like http://welshcomputing.com/paste/477yuify81 , which all seems pretty reasonable to me
[01:55] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3803 -- right.
[01:55] sourcerer: 2022-04-11 19:29:54 (#jwrd) jfw: of course, nothing stops us from starting at 10 digits and lengthening in the event it actually found some kind of high-volume usage.
[01:58] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3792 -- we talked about this in sidebar today and decided it's better to start the resync once the new RAM is installed than delay for the full gales rebuild.
[01:58] sourcerer: 2022-04-10 19:47:09 (#jwrd) jfw: I suppose doing the full Gales bootstrap & reinstall first wouldn't hurt either, since it's coming up in the training.
[01:58] dorion: caai, fyi ^^^
[14:25] caai: dorion, alright. I just purchased two of them
[15:29] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3771 -- thanks, I've tried giving this a spin. I have it working on an ubuntu toilet box, but not on my gentoo system. in both cases it's installed at /etc/tmux.conf with 644 permission. when I hit F5 on the gentoo system, a ~ is emitted in the terminal. the standard C-b is still operational.
[15:29] sourcerer: 2022-04-10 05:54:30 (#jwrd) jfw: dorion: you asked about my tmux.conf, and on working in a fresh machine I'm reminded of just how much better it makes things. http://fixpoint.welshcomputing.com/wp-content/uploads/2022/04/tmux.conf
[15:31] dorion: and trying on Gales yields same result as gentoo.
[15:32] dorion: jfw, any hints or is there more background info I could provide you ?
[15:33] dorion: oooo, looks like I need to kill all the active sessions on the system.
[17:09] jfw: dorion: ah, that sounds like a bizarre design, to have separate instances colluding with each other; I certainly wouldn't put it past them though. I would have asked if there were a ~/.tmux.conf possibly conflicting
[17:12] jfw: I'm noticing the inability to send a literal F6-8 is at least a theoretical hole in my keymap config. can probably just escape them with F5, like the F5 itself.
[17:18] jfw: The technical reason for the tilde is that function keys, since they don't have ASCII codes, are transmitted as escape sequences, with F5 being ESC [ 1 5 ~ (at least on my present terminal). Pretty much an old-school form of multi-byte encoding. If the application (such as the shell) doesn't recognize it, one possible symptom is that the first ESC ... characters are dropped, the parser resets and
[17:18] jfw: the remaining characters show up literally.
[17:21] jfw: ESC is a true ASCII control code, namely Ctrl-[ and the same as sent by the physical Escape key.
[17:24] jfw: hah, found another proliferated ~/.tmux.conf of mine that *did* close the F6-8 hole.
[17:37] jfw: updated version.
[18:05] jfw: regarding the typo in my recent patch, my inclination is to issue a regrind but also add some sort of "listing" counterpart to the "getting", as otherwise there's no way to discover what block hashes the node has.
[18:05] sourcerer: 2022-04-10 19:30:09 (#jwrd) jfw: I also see I messed up my last patch by missing the damn newline in the edited log message.
[18:09] jfw: one option would be to have, say, a "getblockindex" with no arguments return a json list of all the index entries in full detail. the "getblockindex <hash>" form should then be changed to return a list-of-1 for consistency. a downside is that throwing such volume of data by default might be unexpected.
[18:10] jfw: another would be to have a separate RPC eg "listblockindex" that returns a json list of just the block hash strings, then you can script it to fetch detail about any given one or all of them as desired.
[18:14] jfw: prot
[18:15] jfw: ^ half-dead tcp connection after power outage, heh.
[18:17] jfw: anyway, this is probably in the "nice to have" category still, since I was able to do without earlier, and 'getblockindex' was still helpful on its own, so probably best to just fix the typo for now.
[21:06] jfw: now I discover that busybox 'tr' is broken. it supports backslash escapes; it supports ranges; but it silently fails to support backslash escapes as range endpoints.
[21:10] jfw: I was trying to specify the set of all byte values as '\000-\377'
[21:16] jfw: perhaps fixed elsewhere
[23:13] jfw: I've digested & applied a slight variant of the patch - just leaving out some of the "const char *" back-and-forth which ultimately read to me as more obfuscation and self-delusion than anything.
[23:31] jfw: http://welshcomputing.com/paste/t69pan3n2v << and we have random IDs.
[23:34] jfw: still http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-mar-2022/#3574 except by adding the 'just trust urandom' dependency/assumption and some further pruning it's down to 12 lines.
[23:34] sourcerer: 2022-03-30 02:01:48 (#jwrd) jfw: 19 lines of bash/ksh, durable commits & no RDBMS or PHP etc.
Day changed to 2022-04-16
[05:17] jfw: Preparing for a long-absent update of the public release of Gales Linux and greasing the wheels to keep them coming, I've eliminated all binary/non-textual characters from the tree. This doesn't cover the tarballs pulled in for gports, the toolchain, lilo and similars, but does cover musl & busybox which have been imported directly to the tree.
[05:21] jfw: The idea is to use a linux kernel style release mechanism, with a full updated tarball from time to time and a series of patches based on the latest of those otherwise. It's still a ways from something properly V-like, but one step closer.
[05:22] jfw: It also means I can finally put it up in codeview.
Day changed to 2022-04-17
[13:41] caai: 4096 Jul 19 2019 ttyS0
[13:46] caai: Good morning! That last message was simply a tmux test and had no meaning. I was testing to see if I can copy and paste if yrc is opened within tmux.
[17:33] jfw: caai: nice, looks like you can.
[17:52] jfw: dorion: some historical fail I came across in tmux: around 2016 they quietly took it "UTF-8 is now widely enough adopted that supporting it alone is practical." (the thread gets sidetracked by "iso 8859", a strawman in my view since it seems to take the "infinite alphabet" / "no user's language matters more than any other's" premise as a given)
[17:54] jfw: I'd been wondering why the multibyte utf8 sequences were showing up so poorly in my environment. vim is sensibly handling them as just separate bytes, the linux console can be set to do so too so they show up as whatever dingbats are in the 128-255 range of the current console font, but tmux insists on parsing utf8, thereby throwing off cursor positioning.
[17:55] jfw: (while eliminating them from the Gales tree.)
[18:04] jfw: this might just be the last straw re tmux; it keeps coming up with these fundamental defects among the superficial ones. It only ever got in over GNU Screen because it was the default on openbsd when I was exploring that, and I came to like its tiled panes.
[18:09] jfw: ie not a particularly considered decision, and incremental-cost-minimizing since then. not sure as yet if this is a high priority to deal with.
[18:16] jfw: as far as that patch-friendly Gales release, I've tightened up my audit script and turned up some further difficulties: empty files (mainly in musl), missing trailing newlines ("noeol", in busybox shell test cases), and one DOS line ending in an otherwise unix-format file (in a 'links' patch)
Day changed to 2022-04-18
[17:05] jfw: http://welshcomputing.com/paste/hhz52895bm << error encountered in building bitcoind on a stock ubuntu with gcc "10.3.0", coming from improper redefinition of __atomic_compare_exchange. I swear I'd seen this before but haven't turned up any discussion.
[17:46] jfw: welcome to irc, jwm!
[17:56] jfw: the fix at https://fsanmartin.co/compiling-berkeley-db-4-8-30-in-ubuntu-19/ looks correct to me.
[17:59] jfw: specifically: at some point gcc introduced the atomic builtins which conflict with the implementation provided by bdb under the same name; the idea is to keep using the provided version by renaming it.
[18:12] jfw: C89 section 4.1.2 reads, "All external identifiers that begin with an underscore are reserved. All other identifiers that begin with an underscore and either an upper-case letter or another underscore are reserved. If the program defines an external identifier with the same name as a reserved external identifier, even in a semantically equivalent form, the behavior is undefined."
[18:14] jfw: what's perhaps left unclear there is "reserved for whom"; as I understand the standard, it means for the language implementation, which in concrete terms includes compiler builtins and libc.
[18:15] jfw: and in any case bdb is certainly not part of the language implementation, so AFAICS it's in the wrong to be defining such names itself.
[18:20] jfw: so what I don't like about the linked fix is that it maintains the __ prefix while renaming. I suppose the intent of such naming was to exploit that "reserved" status to avoid conflicts with the bdb application's code. A kind of counterfeiting I'd say - relying on others to respect the reservation but not doing so themselves.
[18:21] jfw: jwm: to be clear, it's fine to give it a try, see what happens.
[18:23] jfw: but for a proper fix, I'd rather either rename to something without leading underscores, or perhaps take the lead of the comment and use the builtin __sync_bool_compare_and_swap available since gcc 4.1
[18:28] jfw: "New code should always use the `__atomic' builtins rather than the `__sync' builtins", the gcc geniuses say, without mentioning when those were introduced.
[18:31] jfw: an advantage to going with the __sync builtins might be cross platform support; the bdb assembly code is x86 only and otherwise falls back to a (slower) mutex based implementation.
[18:32] jfw: and the situation triggers me afresh about the bdb code being hidden away in a tarball rather than in the supposed V tree.
[19:38] jfw: The edit worked, then there was some shit about std::array from a newer c++ standard conflicting with boost::array. cure was to add -ansi to CXXFLAGS
[19:39] jfw: (that was in the bitcoin code itself, not bdb etc.)
[20:00] jfw: ah, the atomics were already added in gcc 4.7; the conflict just shows up as warnings in the Gales environment while the newer compiler has perhaps promoted it to full error.
Day changed to 2022-04-19
[05:11] jfw: 15 down, 5 to go on diagnosis & treatment of the empty files in my musl tree
[05:11] sourcerer: 2022-04-17 18:16:33 (#jwrd) jfw: as far as that patch-friendly Gales release, I've tightened up my audit script and turned up some further difficulties: empty files (mainly in musl), missing trailing newlines ("noeol", in busybox shell test cases), and one DOS line ending in an otherwise unix-format file (in a 'links' patch)
[05:14] jfw: and a corrected getblockindex vpatch upcoming after final review & signing.
[05:16] jfw: I tried the CXXFLAGS change on my Gales system: it built, but the binary differed from before which seems to warrant further investigation.
[05:18] jfw: possibly just the "line numbers embedded in strings for mutex debugging" bother again.
[20:04] jfw: in amusing reports from a user of the "ledger" "hardware wallet", their software wouldn't allow access to the coin stored thereon until it was allowed to update the device's firmware; and upon allowing such update, its memory was wiped and it now reports a balance of zero. The only option appears to be digging up the 'seed phrase' to recover it.
Day changed to 2022-04-20
[06:41] jfw: those last empty files in musl proved the trickiest for tracking down what was going on, but it's done, and the noeols too; so I now have a mechanically-certified all-well-formed-text Gales tree. (I'm not going so far as to ban CR line endings, though my script does warn of their presence.)
[19:02] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3830 -- I agree with the "nice to have" categorization. I think the approach of separate RPC for "listblockindex" is better than adding to getblockindex.
[19:02] sourcerer: 2022-04-12 18:17:55 (#jwrd) jfw: anyway, this is probably in the "nice to have" category still, since I was able to do without earlier, and 'getblockindex' was still helpful on its own, so probably best to just fix the typo for now.
[19:02] sourcerer: 2022-04-12 18:10:22 (#jwrd) jfw: another would be to have a separate RPC eg "listblockindex" that returns a json list of just the block hash strings, then you can script it to fetch detail about any given one or all of them as desired.
[19:03] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3835 -- nice, and win.
[19:03] sourcerer: 2022-04-12 23:31:41 (#jwrd) jfw: http://welshcomputing.com/paste/t69pan3n2v << and we have random IDs.
[19:03] sourcerer: 2022-04-12 23:34:38 (#jwrd) jfw: still http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-mar-2022/#3574 except by adding the 'just trust urandom' dependency/assumption and some further pruning it's down to 12 lines.
[19:03] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3838, http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3839 -- sounds like a good start and path forward for incremental improvements.
[19:03] sourcerer: 2022-04-16 05:17:31 (#jwrd) jfw: Preparing for a long-absent update of the public release of Gales Linux and greasing the wheels to keep them coming, I've eliminated all binary/non-textual characters from the tree. This doesn't cover the tarballs pulled in for gports, the toolchain, lilo and similars, but does cover musl & busybox which have been imported directly to the tree.
[19:03] sourcerer: 2022-04-16 05:21:44 (#jwrd) jfw: The idea is to use a linux kernel style release mechanism, with a full updated tarball from time to time and a series of patches based on the latest of those otherwise. It's still a ways from something properly V-like, but one step closer.
[19:04] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3840 -- that'll be slick.
[19:04] sourcerer: 2022-04-16 05:22:43 (#jwrd) jfw: It also means I can finally put it up in codeview.
[19:05] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3872 -- big win, congrats.
[19:05] sourcerer: 2022-04-20 06:41:44 (#jwrd) jfw: those last empty files in musl proved the trickiest for tracking down what was going on, but it's done, and the noeols too; so I now have a mechanically-certified all-well-formed-text Gales tree. (I'm not going so far as to ban CR line endings, though my script does warn of their presence.)
[19:05] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3847 -- I'm open to trying out screen.
[19:05] sourcerer: 2022-04-17 18:04:03 (#jwrd) jfw: this might just be the last straw re tmux; it keeps coming up with these fundamental defects among the superficial ones. It only ever got in over GNU Screen because it was the default on openbsd when I was exploring that, and I came to like its tiled panes.
[19:06] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3871 -- pretty brutal, "nobody" could've predicted.
[19:06] sourcerer: 2022-04-19 20:04:52 (#jwrd) jfw: in amusing reports from a user of the "ledger" "hardware wallet", their software wouldn't allow access to the coin stored thereon until it was allowed to update the device's firmware; and upon allowing such update, its memory was wiped and it now reports a balance of zero. The only option appears to be digging up the 'seed phrase' to recover it.
[21:11] jfw: welcome back dorion :P
[21:11] jfw: how's it going?
[21:15] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3873 - thanks; but I'd have to point out that 8 days was too long to wait for input on that. by about two days, looks like.
[21:15] sourcerer: 2022-04-20 19:02:43 (#jwrd) dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3830 -- I agree with the "nice to have" categorization. I think the approach of separate RPC for "listblockindex" is better than adding to getblockindex.
[21:15] sourcerer: 2022-04-19 05:14:27 (#jwrd) jfw: and a corrected getblockindex vpatch upcoming after final review & signing.
[21:19] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3888 - myeah. the other detail I gleaned was the user already had the software part installed and supposedly ready to go from back when it was all set up; but the 'ledger live' web service it depended on wouldn't talk to it, initiating the chain reaction of forced changes.
[21:19] sourcerer: 2022-04-20 19:06:50 (#jwrd) dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3871 -- pretty brutal, "nobody" could've predicted.
[22:17] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3871 -- okay.
[22:17] sourcerer: 2022-04-19 20:04:52 (#jwrd) jfw: in amusing reports from a user of the "ledger" "hardware wallet", their software wouldn't allow access to the coin stored thereon until it was allowed to update the device's firmware; and upon allowing such update, its memory was wiped and it now reports a balance of zero. The only option appears to be digging up the 'seed phrase' to recover it.
[22:17] dorion: whoops, wrong link.
[22:18] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3891 -- okay.
[22:18] sourcerer: 2022-04-20 21:11:43 (#jwrd) jfw: how's it going?
[22:19] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3892 -- yeah, apologies for the tardiness.
[22:19] sourcerer: 2022-04-20 21:15:06 (#jwrd) jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3873 - thanks; but I'd have to point out that 8 days was too long to wait for input on that. by about two days, looks like.
Day changed to 2022-04-25
[18:03] jfw: so last week we implemented a shiny contact form for jwrd.net; today it's gotten its first spam, of the 'SEO' variety informing us that all we're missing is... 'a QUICK, EASY way to connect with you NOW.'
[18:05] jfw: time to randomize the field names already I guess.
[18:07] jfw: arguably it already beats publishing an email address, since there's the possibility of such unobtrusive-to-humans 'challenge/response' type filtering.
[21:02] jfw: http://fixpoint.welshcomputing.com/2022/php-pains/ << didn't quite fit in a log line.
Day changed to 2022-04-26
[14:26] caai: jfw: Good morning! I have received and installed the DDR3s. I am ready to take the next step.
[14:34] caai: On the other hand, could we please move the Operator training to May 4? Although I have been reviewing the material every day, I am just now beginning the homework from Lesson 9 and I would like to at least near finishing the homework from lesson 9 and 10 before the next training.
[16:41] jfw: caai: the easiest check for the RAM is to run 'free' and see if it's the expected total capacity (units are kB so should read around 8 million).
[16:42] jfw: caai: do you have an updated backup of your wallet.dat? another from before the whole corruption incident would be preferable too.
[16:42] jfw: caai: moving training to next week will be fine; thanks for the heads up.
Day changed to 2022-04-27
[12:12] caai: when I run 'free' the total mem: 8197304
[12:29] caai: Yes, I have a backup of my wallet.dat. Dorion helped me backup the wallet.dat in October of last year. I just made another backup of my current wallet.dat in a separate created directory 'cp .bitcoin/wallet.dat /dev/mnt/Backup', too
[12:30] caai: I will make the change in my calendar for the training to be next week, thanks
[15:24] dorion: caai /dev/mnt/Backup looks a bit awkward.. you did, e.g. mount /dev/sdb1 /dev/mnt ? usually /mnt is the mountpoint used for external drives.
[15:42] caai: dorion: thanks for catching that typo and clarifying. I ran 'mount /dev/sdb1 /mnt', then I backed it up 'cp ~/.bitcoin/wallet.dat /mnt/Backup'
[15:48] dorion: yeah, sounds better.
[17:35] jfw: caai: alright, then the next step is to remove everything except the wallet and config file, so that the database gets rebuilt and re-validated from the genesis block. first though I think we should save the old block files so that we'll have the option of usign them to accelerate the sync from local data rather than the p2p network.
[17:36] jfw: (it's ok if some are corrupt since they'll be re-validated in any case.)
[17:37] jfw: so I'd do a 'cd ~/.bitcoin; mkdir old-blocks; mv blk[0-9]*.dat old-blocks'
[17:38] jfw: then: 'rm -r _* addr.dat blkindex.dat database'
[17:41] jfw: up to you if you want to remove the old debug.log too, or move it for safe keeping... I'd be inclined to just zap it, as it's large and I don't expect to need anything further from it since we're abandoning the old db.
[17:43] jfw: and for completeness, there's also db.log which is usually empty or small.
[17:46] jfw: then start the node back up, obviously.
[17:47] jfw: I'm not entirely sure whether it'll figure out what's going on with the old wallet already present, or whether it has to be added later and "rescanned", but we'll find out soon enough.
Day changed to 2022-04-28
[13:38] caai: Alright. I 'cd ~/.bitcoind; mkdir old-blocks; mv blk[0-9]*.dat old-blocks'
[13:40] caai: Then, I 'rm -r _* addr.dat blkindex.dat database db.log'. I 'mv debug.log ../bcrepairdir' just in case
[13:43] caai: Finally, I ran 'bitcoind'. And 'tail -f debug.log | egrep ACCEPTED' - it is accepting blocks
[13:46] caai: When I run 'bitcoind getinfo' the block height is slowly growing and it does not show an error, however the balance is 0.00
[15:53] caai: Now, just around 2 hours later. The block height is already at 100,000
[15:54] jfw: caai: sound good.
[15:55] jfw: it won't show an accurate balance until it's caught up with the relevant transactions.
[15:56] caai: I thought that might be the case; it is good to have that assumption confirmed by you.
Day changed to 2022-04-29
[20:42] jfw: rodolfo: welcome to the channel. how's it going?
[20:43] rodolfo: Everything is good man, I hope you are doing well too
[20:50] jfw: well, I just had a hearty sandwich and fresh coconut water from the backyard, so can't complain.
[20:51] jfw: dorion got me your new ssh public key so I'm working on importing it.
[20:59] jfw: should be set, give it a try. not sure how much info you got on the login, so just ask me if needed. the host fingerprint, for reference, is:
[20:59] jfw: 4096 SHA256:aFFxkkYI8jTURz8yQ+Uj8WrilrpNcCyMGvY6diRCIU8 root@s7g1 (RSA)
[21:01] rodolfo: Hahaha we had two very different kinds of meals.
[21:01] rodolfo: Perfecto!
[21:02] rodolfo: I'll check the host info
[21:02] rodolfo: I'm using a very simple yet flexible tool called MobaXterm
[21:03] rodolfo: It has similar capabilities compared to Putty
[21:09] jfw: not much of a linux guy I gather?
[21:26] rodolfo: https://www.irccloud.com/pastebin/QocPpaA1
[21:27] rodolfo: I really enjoyed dealing with Debian environments, but nowadays because of work, the majority of my time is spent in Windows
[21:28] jfw: gotcha. what happened with those lines spilling over into a paste link there?
[21:30] jfw: I'll be heading out shortly just fyi.
[21:31] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3947 << how it looked from this end. I guess 'irccloud' is a bouncer/proxy of sorts that's "enhancing" things.
[21:31] sourcerer: 2022-04-29 21:26:20 (#jwrd) rodolfo: https://www.irccloud.com/pastebin/QocPpaA1
[21:35] rodolfo: I think the link is related to the method I'm using to connect to IRC
[21:35] rodolfo: Yeah, it's the application I'm using. Sorry for inconveniences
[21:37] rodolfo: Do you already have a hostname I can connect to?
[21:38] rodolfo: I'll set it up on my side inside a list of known hosts and that way when everything is ready on your side, I'll just attempt to connect
[21:40] rodolfo: This is what I sent that later became a link-based message:
[21:41] rodolfo: https://usercontent.irccloud-cdn.com/file/oo6qDhn3/Screenshot_20220429_163948.jpg
[22:25] rodolfo: Also, I was wondering about the capabilities I would have in the Gales env in terms of, for example, installing py libs via pip or connecting to MSSQL via ODBC (if required). Also, if there are limitations for fetching info from external REST APIs and finally if I could get writing/read privileges in order to manipulate serialized data files (mainly pickle files)
[22:28] rodolfo: This might be to soon to address, the VM is going to be the environment to test the project's logic + rendering (HTML), right?
[22:33] rodolfo: If so, what would be the correct way to debug the functionality from an end user perspective? I understand that there is no UI at the moment, which is fine with me, but it would be interesting to have a look of the interface while reaching substantial progress just to make sure I'm not building a house of cards
[22:37] rodolfo: Maybe a visualization of the design could also help so I can also identify in what part of the architecture I'm standing since what I did previously (the real estate app) was conceive since inception as a set of functions standing on Flask so the input methods where oriented to satisfy the general rules of that framework
Day changed to 2022-04-30
[13:37] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3936 -- welcome rodolfo.
[13:37] sourcerer: 2022-04-29 20:43:17 (#jwrd) rodolfo: Everything is good man, I hope you are doing well too
[13:41] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3961 -- you could start with flask at first, but as we discussed, the plan is to ditch it once Apache and the full LAMP stack is ready and available on Gales. In my mind you'd load the site in your local browser to debug.
[13:41] sourcerer: 2022-04-29 22:33:07 (#jwrd) rodolfo: If so, what would be the correct way to debug the functionality from an end user perspective? I understand that there is no UI at the moment, which is fine with me, but it would be interesting to have a look of the interface while reaching substantial progress just to make sure I'm not building a house of cards
[13:42] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3959 -- mssql won't be available on Gales, mysql is.
[13:42] sourcerer: 2022-04-29 22:25:34 (#jwrd) rodolfo: Also, I was wondering about the capabilities I would have in the Gales env in terms of, for example, installing py libs via pip or connecting to MSSQL via ODBC (if required). Also, if there are limitations for fetching info from external REST APIs and finally if I could get writing/read privileges in order to manipulate serialized data files (mainly pickle files)
[13:42] dorion: I'll let jfw answer the other parts of that question.
[16:56] rodolfo: dorion: understood the debugging and Flask/Apache explanation
[17:01] rodolfo: I'm not proficient in PHP, but I could give it a try
[17:04] rodolfo: dorion: sorry. Same question, but referring to MySQL
[17:46] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3954 - no worries, we can revisit later if it comes up again (though I suspect it will)
[17:46] sourcerer: 2022-04-29 21:35:59 (#jwrd) rodolfo: Yeah, it's the application I'm using. Sorry for inconveniences
[17:47] jfw: I'll send you the IP in a private message, so that it's up to you when you're ready to make it public.
[17:57] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3955 - I'm seeing two facets to this question: what's allowed or possible, and what's the best aligned or supported way to do things in the new environment.
[17:57] sourcerer: 2022-04-29 21:37:13 (#jwrd) rodolfo: Do you already have a hostname I can connect to?
[17:57] jfw: ah, wrong link; that was re http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3959
[17:57] sourcerer: 2022-04-29 22:25:34 (#jwrd) rodolfo: Also, I was wondering about the capabilities I would have in the Gales env in terms of, for example, installing py libs via pip or connecting to MSSQL via ODBC (if required). Also, if there are limitations for fetching info from external REST APIs and finally if I could get writing/read privileges in order to manipulate serialized data files (mainly pickle files)
[18:01] jfw: on the first, you have root access to your VM and in principle you can install & do whatever you need on it, as long as it's not abusive of the host machine or network.
[18:04] jfw: one catch is that that's transitive, in the sense that if the VM gets infected/compromised, it's your problem (though we'll try to help).
[18:08] jfw: on the second, a core principle of Gales along with everything else we do is to exercise ownership of your own stuff. the trouble with installing things through pip is that you're essentially giving write access to your codebase to any number of unknown parties on the internet.
[18:12] jfw: if you accept that you're responsible for your dependencies, rather than trusting that "the community" will maintain them adequately & in alignment of your usage of them, it starts to make more sense to look closer at them, and what they're pulling in, and keep your own copies for reference & deployment.
[18:14] jfw: also note that some python packages include C extensions, which work by building a shared lib that gets dynamically loaded into the interpreter; this won't work on Gales because it doesn't have dynamic linking. in some cases this is just 'acceleration' code with a pure python alternative that will work. otherwise, it's possible to tweak the python build itself if there are actual missing features.
[18:17] jfw: a big one coming to mind there is the 'openssl' module, which is used implicitly eg by urllib for 'https' urls; I had tried to keep the openssl code (with its dismal security track record) out of the python executable and use external helper programs; in practice this didn't work so well, and I expect we'll have to enable it.
[18:30] jfw: for connecting to mysql from python, on CentOS I've used MySQLdb; unclear as yet if that's the best choice on Gales though.
[18:47] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3960 - I anticipate the VM being usable both for test & production usage, though we can also spin up another if needed.
[18:47] sourcerer: 2022-04-29 22:28:40 (#jwrd) rodolfo: This might be to soon to address, the VM is going to be the environment to test the project's logic + rendering (HTML), right?
[18:49] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3961 - I don't follow this; which interface or lack thereof do you mean?
[18:49] sourcerer: 2022-04-29 22:33:07 (#jwrd) rodolfo: If so, what would be the correct way to debug the functionality from an end user perspective? I understand that there is no UI at the moment, which is fine with me, but it would be interesting to have a look of the interface while reaching substantial progress just to make sure I'm not building a house of cards
[18:53] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3962 - seems like we need to clarify a bit the structure of your existing thing and the direction to take it.
[18:53] sourcerer: 2022-04-29 22:37:33 (#jwrd) rodolfo: Maybe a visualization of the design could also help so I can also identify in what part of the architecture I'm standing since what I did previously (the real estate app) was conceive since inception as a set of functions standing on Flask so the input methods where oriented to satisfy the general rules of that framework
[18:54] rodolfo: jfw: yes, I believe architecture in general needs to be clarified in order to have a standard criteria
[18:56] jfw: as I see it, you have 3 main parts: 1) scraping data from the web and importing to a database; 2) analysis, number crunching, machine learning or however you call it on said data; 3) processing end-user requests, visualizing data & returning results.
[18:57] jfw: and I gather that at least 2 and 3 are closely coupled at present, in python/flask
[18:57] rodolfo: jfw: in terms of debugging, I understand Gales does not have a UI that allows me to test the functionalities of a web app. Correct? I could just test it on my local computer, so that's not really an issue at the moment
[18:58] jfw: do you mean just running the code & browsing the site, or some kind of higher level testing framework?
[19:03] rodolfo: jfw: correct, that's in general terms the processing flow. I could attempt scraping data with a headless browser tool as puppeteer, which virtually needs no real browser as Firefox or Chrome. In my opinion, I would skip that part and focus on front end functionalities because if by any chance the current env doesn't support dependencies required by any scrapping tool, that specific process could be hosted else where and from Gales I could just make a
[19:03] rodolfo: request via API. Makes sense?
[19:04] jfw: I'd say running the browser on your local machine is fine but for the backend it'll be best to develop & test in something as close to the production environment as possible. and there isn't really a mainstream distribution that's all that similar to gales, if that's what you're looking for; possibly gentoo, or the BSDs even
[19:04] rodolfo: jfw: Just running the code, getting the app server up and see how it works. Nothing fancy
[19:05] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3996 -- what I meant here is that the code is on the server and you, e.g. load the IP using the browser on your local machine to test/debug.
[19:05] sourcerer: 2022-04-30 18:57:45 (#jwrd) rodolfo: jfw: in terms of debugging, I understand Gales does not have a UI that allows me to test the functionalities of a web app. Correct? I could just test it on my local computer, so that's not really an issue at the moment
[19:05] sourcerer: 2022-04-30 13:41:08 (#jwrd) dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#3961 -- you could start with flask at first, but as we discussed, the plan is to ditch it once Apache and the full LAMP stack is ready and available on Gales. In my mind you'd load the site in your local browser to debug.
[19:07] jfw: rodolfo: how are you doing the scraping part now?
[19:09] rodolfo: dorion: understood (regarding the debugging issue)
[19:13] rodolfo: jfw: let me refresh my memory by taking a look on the original process. I'm almost sure the majority of the process is done by requests (pointing to specific XPaths) which needs no browser interaction at all, but I think there is a small part that uses Selenium (which does depend on a browser/headless browser). Certainly, the Selenium part could be replaced by requests.
[19:15] jfw: yeah, selenium for sure is a heavy requirement
[19:17] jfw: to me it seems that the scraping is a key piece of the whole and doesn't necessarily make sense to push off on some external API rather than getting it to work in Gales
[19:19] jfw: unless such API is from a specialist that can actually ensure it keeps working reliably; not too sure what's out there.
[19:20] jfw: but seems to me that scraping by its very nature is always going to be... kind of scrappy
[19:31] rodolfo: Yeah, I'll omit using Selenium, work fully with requests and throw that thing elsewhere because it is a costing process. Doesn't make sense to put Gales's server resources in such hostile scenario
[19:32] rodolfo: Scraping tends to be very resource-exhausting because of many reasons, and exception handing is basically witchcraft hahaha
[19:34] rodolfo: I'm hands on, just started to reduce some redundant features on the data set and sftp that to the VM
[19:34] rodolfo: =)
[19:37] jfw: it'd be better still if you can use urllib/urllib2, which is in the standard library, rather than requests; I have yet to find a situation it can't handle so not sure why people love 'requests' so much.
[19:39] jfw: I have the whole python 2.7 docs mirrored, fyi.
[19:41] rodolfo: Hahaha because requests is the go-to solution for virtually any kind of API-related problem. The scraping particularly has nothing to do with calling API methods, but for sure it's easier to shrink down the amount of libraries used in a project
[19:42] rodolfo: Right, I'm taking a look on the current modules installed on the VM Python
[19:42] jfw: that seems a bit circular - people go to it because it's the go-to?
[19:43] rodolfo: It's the kind of abstraction people like
[19:43] rodolfo: Hahaha
[19:43] rodolfo: Let's maybe requests is trendy
[19:46] rodolfo: Outside the Python command line, how do you invoke MySQL?
[19:46] rodolfo: From $_
[19:47] rodolfo: Just wondering how to intereact directly with MySQL from the Linux terminal
[19:47] jfw: 'requests' would be the usual convenience-chasing formula, with something like 50% more convenient in the narrow view and 500% less convenient when looking at the whole, seems to me. there's way worse examples, to be sure.
[19:48] jfw: rodolfo: mysql isn't installed there yet, as I haven't fully wrapped up the porting process unfortunately, though it's a priority.
[19:49] jfw: the answer though would be simply 'mysql'
[19:50] jfw: http://fixpoint.welshcomputing.com/category/software/mysql/ is my series documenting the work, as far as it's gotten
[19:51] jfw: for that you could certainly use a different distribution to get acquainted with it.
[19:53] rodolfo: I just learned requests depends on urllib, had no idea.
[19:53] jfw: haha
[19:54] rodolfo: It's OK, I just wanted to explore capabilities hehehe for now plain text files would work fine
[19:55] rodolfo: I was almost sure that requests was a dependency for all the other HTTP-modules. Little did I know
[19:57] jfw: and in case you're not overloaded on links yet, there's http://fixpoint.welshcomputing.com/category/software/gales-linux/ for a bit broader reference on Gales, along with what's scattered around the IRC logs.
[19:58] rodolfo: Hahaha eventually I will finish reading all of it
[20:04] jfw: it's certainly possible at least at the surface level, since I haven't really written all *that* much.
[20:05] jfw: there's quite a lot more that comes in by reference though.
[20:27] rodolfo: I see that most of the content is actually like deep thoughts about Gales hehehe
[22:45] rodolfo: @jfw would you say this list of Python 2 modules is the one available at the VM?
[22:45] rodolfo:
[22:45] rodolfo: https://docs.python.org/2/py-modindex.html
[23:19] rodolfo: Also, I did find native modules (or combination of these) that could replace functionalities found in numpy, pandas, scipy, sklearn, etc. However, I'm wondering if it is worth the time/effort to implement this kind of solution vs, again, deploying the whole logical/regression process on an external server and relying of urllib2 for HTTP calling via API. BTW, I would still need to spend some time setting up urllib2 with the correct socket configuration,
[23:19] rodolfo: according to the module's literature. The socket config does not come out of the box, as it happens with urllib3 or requests
[23:21] rodolfo: Not complaining at all, I find the experience very enriching just wanted to know your thoughts.
[23:34] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Apr-2022/#4040 - well, writing is a good way to help figure things out. were you expecting something different?
[23:34] sourcerer: 2022-04-30 20:27:39 (#jwrd) rodolfo: I see that most of the content is actually like deep thoughts about Gales hehehe
[23:37] rodolfo: jfw: You are right. For a moment I thought it was some kind of documentation/user manual literature
[23:39] jfw: ah. yes, the more technically detailed docs tend to be included with the code, eg in the distribution source tarball
[23:41] jfw: though I expect it's a bit lacking as far as a smooth on-ramp for beginners; thus, feel free to ask things.
[23:42] jfw: the python module list you link looks to be a superset of what's there, as it includes all things which might be available on some system, while python supports many different systems.
[23:42] jfw: I'll put together the exact list for you
[23:47] jfw: rodolfo: http://fixpoint.welshcomputing.com/wp-content/uploads/2022/04/gales-python-modules << this shows how to find the exact list, both the C (builtin) and written-in-python ones.
[23:49] jfw: one catch is that the latter sometimes depend on the former and so will appear to exist but fail to load if the C part is missing; for instance /gales/pkg/python/lib/python2.7/ssl.py does an "import _ssl" which will currently fail.
[23:50] jfw: can you point me to what you're reading re urllib socket config? I recall there's higher-level ways to call it with minimal setup for the common cases
[23:53] jfw: rodolfo: what do you have in mind for that external server? to better evaluate, I suppose we'd have to look at the full list of what's required.