Day changed to 2022-10-02
[01:59] jfw: pointing out this instance where the GNU autoconf people still haven't fixed their error messages, 19 years later.
[02:06] jfw: or their documentation for AC_DEFINE which gives no hint that the 'description' parameter is mandatory.
[16:04] caai: jfw: well, maybe they are waiting for the 2 decade mark to start working on it. ha! is this affecting a 'missing template problem' task that you are working on?
Day changed to 2022-10-03
[02:56] jfw: caai: the fatlogic here looks more like "if we just don't respond to the person pointing out the flaw then we can safely forget it happened and go on pretending the flaw doesn't exist"
[02:58] jfw: a pretty common response too, because it's the easiest
[03:02] jfw: it was a relatively minor nuisance in what I was working on, just an impressively pristine specimen of the failure.
[03:02] jfw: especially because it would have been such a simple fix!
[03:11] jfw: the slightly larger view of what I was up to was bypassing this nonsense, which involved writing ~16 lines of autoconf code as a stub in place of the original ~293 lines
[03:17] jfw: some of the madness in that first link is worth explicitly calling out too: "GNU libiconv, if installed, is not necessarily already in the search path" - if it's not in the search path, then either it's not installed, or it's not meant to be used by default. otherwise IT WOULD BE IN THE SEARCH PATH.
[03:27] jfw: nothing like dealing with the actual problems created by the "solutions" to hallucinated problems.
[15:03] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Sep-2022/#4920 -- did your bro fare ok ?
[15:03] sourcerer: 2022-09-30 22:07:11 (#jwrd) caai: i am glad to hear that you came out unscathed, minus a sleepless night. i saw some footage from the west coast and it looks like there was substantial damage.
[16:08] jfw: dorion: you mean our Naples - St Petersburg guy? he was out of state at the time. dunno what he'll find when he returns.
[16:48] dorion: I mean caai's blood bro who lives on the left coast.
Day changed to 2022-10-04
[19:40] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4927: yes, too bad they have let it collect dust for so long
[19:40] sourcerer: 2022-10-03 03:02:33 (#jwrd) jfw: especially because it would have been such a simple fix!
[19:41] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4934: he was fortunate and only lost a few branches off a palm trees. he is located about 20 miles from the beach in Sarasota
[19:41] sourcerer: 2022-10-03 16:48:22 (#jwrd) dorion: I mean caai's blood bro who lives on the left coast.
[19:41] caai: off a palm tree*
[20:40] caai: jfw: could you help me with this question? In this class, we are using CentOS. Question: What does your .profile file say is your default PATH? Note: If you do not have a .profile, please examine your .bashrc file in your home directory.
[20:40] caai: # User specific environment
[20:40] caai: PATH="$HOME/.local/bin:$HOME/bin:$PATH"
[20:40] caai: export PATH
[20:40] caai: PS1="Your majesty ? "
[20:40] caai: export PS1
[20:42] caai: my username is is229044, therefore my default PATH should be: is229044/.local/bin
[20:42] caai: Does that seem correct to you?
[20:45] dorion: caai, it's the whole string. and to check what $HOME is, you can issue `echo $HOME`
[20:56] caai: dorion: got it. thanks
[21:22] jfw: this is where I would lose points on the exam by stubbornly pointing out that the question is ill-defined, heh.
[21:23] jfw: the profile is defining the PATH in terms of other variables - including the pre-existing PATH - that are not apparent from the profile itself.
[21:25] jfw: so most literally you could give the exact string ($HOME/.local...) but that's not the same as the value that the variable actually gets i.e. what the default PATH actually *is*.
[21:27] jfw: lolz @ "your majesty" prompt, sounds like "I know I'm important because my dog loves me"
[21:27] jfw: the computer's love is cheaper still, doesn't require daily attention!
[23:20] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4951: eso!
[23:20] sourcerer: 2022-10-04 21:23:51 (#jwrd) jfw: the profile is defining the PATH in terms of other variables - including the pre-existing PATH - that are not apparent from the profile itself.
[23:21] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4952: i will copy and paste this response and report back hahaha (or maybe not)
[23:21] sourcerer: 2022-10-04 21:25:46 (#jwrd) jfw: so most literally you could give the exact string ($HOME/.local...) but that's not the same as the value that the variable actually gets i.e. what the default PATH actually *is*.
[23:35] caai: report back what the teacher's response is*
[23:48] jfw: alright good luck. dorion is also correct that you need the whole thing where you focused in on just the first component of the colon-delimited list.
Day changed to 2022-10-05
[00:12] caai: understood. thanks
[01:54] jfw: to document a bit of weird that came up in Gales: I'm running the ./configure script for a program that includes autoconf code probing for a working -fstack-protector and/or -fstack-protector-strong GCC options. The first at least is supposed to work and is even enabled by default in Gales, yet the probe concludes that it doesn't, and a manual test "gcc -fstack-protector file.c" reports "ld:
[01:54] jfw: cannot find -lssp_nonshared". How can this be?
[01:58] jfw: a hint is found here; basically, musl includes the necessary support function to which gcc generates calls, except there's *another* one that it also generates only on certain platforms, which for some odd reasons can't be implemented directly in libc.
[01:59] jfw: this then is supposed to be found in "libssp_nonshared.a", which is provided by neither gcc nor musl, but the trivial implementation is found in musl-based distros such as Alpine.
[02:01] jfw: in turn, the gcc patch produced by the musl people to tell gcc how to work with musl, causes it to attempt to link this libssp_nonshared when -fstack-protector is in use.
[02:03] jfw: but that means when it's used at link time; so "gcc -fstack-protector -c file.c" (compile-only mode) works as expected, then "gcc file.o" works to link it, because on amd64 the offending code is never generated.
[02:04] jfw: "gcc -fstack-protector file.o" then will reproduce the error as in the all-at-once usage.
[02:06] jfw: the "enabled by default" part works by adding it to the "cc1" flags, in effect making "gcc -c file.c" equivalent to "gcc -fstack-protector -c file.c", but there's no such default configured for link mode, which is why we haven't seen the error before.
[02:08] jfw: so in short, there's no problem at present, but if we port to one of the platforms that needs it - reportedly i386 and (32-bit) powerpc - we'll probably need to 1) import the glue code from Alpine and 2) further tweak the gcc specs to always link -lssp_nonshared.
[02:20] jfw: Relatedly: we decided reluctantly and after some deliberation to add a gport for GNU Libtool; to correct its own description, a generic library support script that papers over the complexity of using shared libraries on totally obscure systems by piling on more complexity behind an obnoxious interface.
[02:23] jfw: Previously I'd been attempting to cut out the usage of libtool wherever it turned up in the autoconf/automake inputs, and this is certainly doable but the exact process varies with each codebase, and it came up so often that I started to see libtool in practice as more of a limb of the larger autotools body, it's simply expected to be there.
[02:28] jfw: The preferred situation is to cut out the autoshit entirely (which we've also done successfully in several cases, but again costing time); to this end, surgical extraction of libtool doesn't really make progress, ie it's purely an additional cost along the way. And on the ugly side, there are some ports where we're still running the auto-generated configure turds as shipped in the tarballs,
[02:28] jfw: because we're unable to regenerate them, having punted on the libtool work.
[02:31] jfw: So the idea (so far validated in practice) is that having it available will make more things "just work" in a more hygienic way than before, letting the sleeping dogs lie and making it easier to replace the whole rotten build system if/when the time comes.
[02:34] jfw: However, we have no intention of *supporting* libtool; the moment it causes problems is the moment it gets thrown out for the particular case.
[02:35] jfw: The same as can probably be said for the rest of all that autoconf, automake, cmake etc etc.
[02:43] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4970 -- ok, that certainly is some weird. glad to hear a) you've grasped it, b) it's not an issue for the present job and c) it's noted should we move to future porting.
[02:43] sourcerer: 2022-10-05 02:08:20 (#jwrd) jfw: so in short, there's no problem at present, but if we port to one of the platforms that needs it - reportedly i386 and (32-bit) powerpc - we'll probably need to 1) import the glue code from Alpine and 2) further tweak the gcc specs to always link -lssp_nonshared.
[02:43] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4971 -- lol, nice correction.
[02:43] sourcerer: 2022-10-05 02:20:06 (#jwrd) jfw: Relatedly: we decided reluctantly and after some deliberation to add a gport for GNU Libtool; to correct its own description, a generic library support script that papers over the complexity of using shared libraries on totally obscure systems by piling on more complexity behind an obnoxious interface.
[02:45] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4976 -- word.
[02:45] sourcerer: 2022-10-05 02:34:14 (#jwrd) jfw: However, we have no intention of *supporting* libtool; the moment it causes problems is the moment it gets thrown out for the particular case.
[02:58] jfw: Interestingly, the Alpine port for musl also includes small implementations of some shell utilities traditionally coming from glibc: getconf, getent, and iconv, which might be worth importing directly.
[02:59] jfw: They're covered in the Linux and/or POSIX man pages, but otherwise missing in Gales since they're not included directly in musl.
[03:11] jfw: I know I've personally used "getent" in shell scripts.
Day changed to 2022-10-08
[17:56] jfw: https://dovecot.org/pipermail/dovecot/2022-October/125452.html << my latest dispatch from the front.
Day changed to 2022-10-09
[14:17] caai: question: when a new linux install is done, upon initial start up, it prompts you to create a 'user'. would you suggest that first user be 'root' and then i create a lower-privileged user for everyday tasks, e.g., 'user'?
[14:26] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4987: :/ not cool
[14:26] sourcerer: 2022-10-08 17:56:07 (#jwrd) jfw: https://dovecot.org/pipermail/dovecot/2022-October/125452.html << my latest dispatch from the front.
[18:29] jfw: caai: some distributions indeed have this "fist boot" configuration wizard thing. The root user will always exist already - typically it would have prompted you to set the root password at install time - so this is for creating that unprivileged user for everyday tasks.
[18:43] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4989 - it's the diplomacy front, really; compatibility problems are pretty much a given when trying to work in the swamps and most likely it falls on me to fix others' broken stuff, but we poke them anyway to see what they're made of.
[18:43] sourcerer: 2022-10-09 14:26:18 (#jwrd) caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4987: :/ not cool
[18:45] jfw: btw, it's better to add a space after URLs like that, since eg : is a valid URL-character so linkifiers will include it in the reference. though here at least the bot can handle it for quoting because it recognizes its fixed prefix pattern.
[18:47] jfw: * "first boot" not "fist boot", lol.
[19:19] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4987 -- the swamp speaks : https://dovecot.org/pipermail/dovecot/2022-October/125458.html
[19:19] sourcerer: 2022-10-08 17:56:07 (#jwrd) jfw: https://dovecot.org/pipermail/dovecot/2022-October/125452.html << my latest dispatch from the front.
[21:07] jfw: then, you do the work and we'll consider merging at some point - a weaker offer even than "we will review"
[21:14] dorion: myeah, so what are you thinking, is it fix their shit or rip out the autoshit entirely ?
[21:20] jfw: leaning toward the latter since evidently it wasn't providing the advertised advantage of "configures to adapt to many kinds of systems" anyway
[21:25] dorion: ok by me, why subsidize them.
[21:26] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4976 after all.
[21:26] sourcerer: 2022-10-05 02:34:14 (#jwrd) jfw: However, we have no intention of *supporting* libtool; the moment it causes problems is the moment it gets thrown out for the particular case.
[21:53] jfw: plus we already know their development direction is removing things that looked useful to us like fts-squat and auth-checkpassword.
Day changed to 2022-10-10
[01:05] jfw: dorion: I think the ripping/fixing in this case will have to involve some study of how the plugin interface works, because it builds many binaries for different sub-components and I'll need to know which ones they need to be linked with.
[01:06] jfw: sorry, confusing wording: dovecot builds many binaries for different sub-components, and I'll need to know which of those the plugins need to be linked into.
[01:07] dorion: jfw, ok. probably won't be your favorite thing to do, but sounds like good time spent wrt getting to know the tool. much better than wrangling the autoshit.
[01:11] jfw: I won't mind the work itself, but the grass does momentarily appear greener on the "imap is easy, you just apt-get something and edit some configs" side
[01:13] jfw: (accompanied necessarily by "and don't forget to drink your security vaccines^W updates because you don't stand a chance to maintain it yourself")
[01:30] dorion: I hear you, tis the draw of the fast food.
[01:33] dorion: the bait and the hook.
[18:45] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4991: is 'fist boot' a synonym for 'glove'? hahaha. unfortunately, this lower priviledged user that is created via the wizard thing (fedora workstation) has superpriviledges via 'sudo' and a simple entering of said user's pw. i will have to change that.
[18:45] sourcerer: 2022-10-09 18:29:19 (#jwrd) jfw: caai: some distributions indeed have this "fist boot" configuration wizard thing. The root user will always exist already - typically it would have prompted you to set the root password at install time - so this is for creating that unprivileged user for everyday tasks.
[18:57] caai: btw, the reason i installed fedora, with all its wizardry, is becuase 'Linux+ and LPIC-1: Guide to Linux Certification', which is a great book, references fedora workstation in all of its examples, and i like to be able to follow along a book verbatim whilst learning
[19:11] jfw: fwiw, I put up with fedora for a couple years, and certainly no harm in seeing more of the world outside the jwrd garden.
Day changed to 2022-10-11
[14:16] caai: good to know. what would you say its advantages vs disadvantages are?
[18:53] jfw: caai: the main advantage would be a large collection of recent versions of software available for easy installation; the accompanying disadvantage is that various things will be buggy or break unpredictably.
[18:54] jfw: with little recourse other than to wait and try the next update, seeing what new things are broken.
Day changed to 2022-10-12
[21:41] caai: i see. then this 'roll it out as fast as possible mentality' exists in the linux world as well ....
[21:47] jfw: perhaps it's more like, following the fashions and calling it progress
[21:50] jfw: or, aiming to be adequate for everyone and thus excellent for no one, ala the supermarket model of shopping.
[22:00] jfw: on the supermarket transformation process
Day changed to 2022-10-13
[04:46] jfw: dorion: I mentioned out-of-band but it's now becoming clearer that the choice isn't so simple as this
[04:46] sourcerer: 2022-10-09 21:14:46 (#jwrd) dorion: myeah, so what are you thinking, is it fix their shit or rip out the autoshit entirely ?
[04:47] dorion: jfw: ok, listening.
[04:47] jfw: it's not just that the build system is broken (or unfit for this environment or suchlike) but the code itself at least in some places is built around dyn linking, specifically the ability to call functions by looking up their names as strings at runtime
[04:48] dorion: oy vey.
[04:49] jfw: a hint of this is in the plugin API doc, but I'm finding that it's more than just those two init/deinit things it covers.
[04:50] jfw: forinstance
[04:52] jfw: which means perhaps I'll need to review the 'modules' api just to find all possible instances
[04:52] dorion: tangentially related, that seems to make the mailing list reply even worse.
[04:53] jfw: myeah, "you come up with a fix [because we certainly don't want to, knowing how difficult we made it]"
[04:54] dorion: so it's still theoretically possible, but lots more work than we expected ?
[04:54] dorion: jfw, how many modules are there ?
[04:55] dorion: have you got a grasp on which ones we want ? at least at a high level, not counting dependencies.
[04:55] jfw: well, that's the other angle which is that the included plugins don't look all that interesting to me with the previously noted exception of fts (full-text search indexing)
[04:56] jfw: approximate list
[04:56] jfw: ("module" seems to mean the same as "plugin" from what I've seen so far)
[04:58] jfw: what I meant by "modules api" was the always-included code that it uses for dealing with plugins, containing things like that offending "module_get_symbol_quiet" along with the docs-mentioned "module_dir_init"
[04:58] dorion: maybe mysql plugin in the future.
[04:59] jfw: ah, that one's a "driver" not a "plugin" - not afflicted by this trouble.
[04:59] dorion: ok.
[05:00] dorion: what does fts dependency graph look like ? I see a few fts there.
[05:02] jfw: from previous reconnaisance, we'd want either fts-squat which they've deprecated, or the new fts-flatcurve which pulls in a "xapian" dependency. I don't think they depend on any of the other plugins if that's what you mean.
[05:03] dorion: other plugins is what I mean.
[05:04] dorion: and from what I recall, we didn't see very good reason why fts-squat was deprecated, apart from boredom.
[05:06] jfw: we didn't find a reason, but that doesn't mean there wasn't one. expanding only, unable to shrink possibly?
[05:08] jfw: confirmed that in both cases only the base "fts" plugin is depended upon.
[05:09] jfw: (granted that still doesn't look like a *good* reason to ditch their own code and run off to some new shiny library)
[05:10] jfw: but good or reasonable or no, deprecated it is so we can be sure of no further help from them.
[05:11] dorion: would that FIXME need to be fixed immediately. or would it be a problem only when the searchable text reaches a certain point ?
[05:12] jfw: afaik it just means the index files won't shrink when you delete messages or move to another mailbox/folder. probably could be worked around by deleting & regenerating.
[05:13] dorion: so even a cron job could be a first step to mitigating ?
[05:14] jfw: or even just an occasional maintenance script for when the disk is getting too full.
[05:15] jfw: (routine reindexing might be a performance drain)
[05:16] dorion: seems like this trouble isn't enough to abandon the gporting and taking a crack at cryus just yet.
[05:18] jfw: you refer to cyrus-imapd, being the main competitor we turned up with comparable features & performance.
[05:19] dorion: right. typo.
[05:19] jfw: (also filling in for the logs.) but what are you suggesting - keep at the investigation of the modules code?
[05:20] dorion: yeah.
[05:20] dorion: at least another day focused on what we're more interested in.
[05:21] dorion: what do you think ?
[05:21] jfw: I tend to agree. the extent of the problem may still turn out to be shallow.
[05:21] dorion: ok, cool. probe some more and lettuce see.
[05:25] jfw: thanks.
[05:25] dorion: thank you. talk w/ you tomorrow.
[19:02] jfw: I was unfortunately mistaken about dovecot's notions of module vs plugin, module is used more generally and includes things like authentication backends and indeed SQL drivers... though it looks like the latter already have provision for being built-in (static).
[19:02] sourcerer: 2022-10-13 04:56:21 (#jwrd) jfw: ("module" seems to mean the same as "plugin" from what I've seen so far)
[23:24] jfw: for a first patch of solid ground to stand on, a file-level map of what calls dovecot's modules API and what executables each one ultimately gets linked into http://welshcomputing.com/paste/myiuuudunx
[23:26] jfw: it made for a strong exercise in learning to decode what all is going on in the Makefile.am files
[23:28] jfw: not to mention getting a closer view of what all is in there - an http client & server ?! here I was mentally knocking cyrus for throwing in such kitchen sinks
[23:41] jfw: l0l... thinking about those mailing list replies, if they're saying they only support their code when used in the ways that they currently use it... well, necessarily there are no bugs in that usage, otherwise they wouldn't be able to use it; therefore, the supported portion of dovecot is necessarily a subset of the bug-free portion; so no bugs can be reported.
[23:53] jfw: the map should provide a complete if not necessarily concise or optimal answer to what the plugins can possibly be called from.
[23:53] sourcerer: 2022-10-10 01:05:17 (#jwrd) jfw: dorion: I think the ripping/fixing in this case will have to involve some study of how the plugin interface works, because it builds many binaries for different sub-components and I'll need to know which ones they need to be linked with.
Day changed to 2022-10-14
[00:00] jfw: which is to say, ~everything ; so most likely the approach is to bundle all desired plugins into the big libdovecot.a, then any executable that invokes them gets the right files linked via the normal workings of the linker.
[00:00] jfw: once the dlsym calls are replaced by static calls, that is.
[00:08] jfw: even their filesystem interface has to be abstracted into modular drivers ?!
[01:18] jfw: a next guiding question: for each call to the functions responsible for loading or initializing modules (module_dir_init, module_dir_load, module_dir_load_missing, module_dir_try_load_missing), can it be statically determined which modules it's trying to load? My guess is that for the more "internal" modules it can, and only the "true plugins" work in the mode of "grab whatever we find in the
[01:18] jfw: filesystem".
[01:19] jfw: the distinction is a bit subjective since of course it's all here in the source tree.
[01:24] jfw: but the idea is to find which ones are best replaced by simple static calls and which ones if any might require actually implementing some kind of dynamic dispatch table.
[03:21] jfw: for the first sample, looking at the pluggable filesystem drivers thing (lib-fs/fs-api.c): modules are loaded based on the filename pattern libfs_<driver>, where <driver> comes from the caller (perhaps ultimatly from a config file); but several of the drivers are builtin, so module loading only happens for plugin-supplied drivers; the in-tree plugins in this category are fs-compress and
[03:21] jfw: mail-crypt.
[03:24] jfw: so for that one I'd think we're safe to remove the module loading (or keep it as a non-functional stub, as is the current situation) and those two plugins.
[03:25] jfw: that's the happy third category I guess, "doesn't need any code changes because the affected modules/plugins are of no possible interest".
[03:40] jfw: the mail-crypt plugin, fwiw; the security model isn't clearly explained but looks like it can derive message encryption keys from the user's password.
[03:45] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5067 -- alright.
[03:45] sourcerer: 2022-10-13 19:02:44 (#jwrd) jfw: I was unfortunately mistaken about dovecot's notions of module vs plugin, module is used more generally and includes things like authentication backends and indeed SQL drivers... though it looks like the latter already have provision for being built-in (static).
[03:45] jfw: this seems to me about the best that can be done under the circumstances, but still a dubious marriage; because the user password is transmitted routinely over the network so should be relatively ephemeral, while an encryption password is for life
[03:48] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5069 -- nice, pretty solid indeed.
[03:48] sourcerer: 2022-10-13 23:24:34 (#jwrd) jfw: for a first patch of solid ground to stand on, a file-level map of what calls dovecot's modules API and what executables each one ultimately gets linked into http://welshcomputing.com/paste/myiuuudunx
[03:49] jfw: why thank you.
[03:50] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5071 -- holy smokes.
[03:50] sourcerer: 2022-10-13 23:28:32 (#jwrd) jfw: not to mention getting a closer view of what all is in there - an http client & server ?! here I was mentally knocking cyrus for throwing in such kitchen sinks
[03:51] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5072 -- or perhaps they've grown numb to the bugs crawling in their neckbeards.
[03:51] sourcerer: 2022-10-13 23:41:16 (#jwrd) jfw: l0l... thinking about those mailing list replies, if they're saying they only support their code when used in the ways that they currently use it... well, necessarily there are no bugs in that usage, otherwise they wouldn't be able to use it; therefore, the supported portion of dovecot is necessarily a subset of the bug-free portion; so no bugs can be reported.
[03:54] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5075 -- alright.
[03:54] sourcerer: 2022-10-14 00:00:02 (#jwrd) jfw: which is to say, ~everything ; so most likely the approach is to bundle all desired plugins into the big libdovecot.a, then any executable that invokes them gets the right files linked via the normal workings of the linker.
[03:55] jfw: perhaps numb, but I'd say I hit them with something sufficiently out of their ordinary diet.
[03:55] dorion: what is this steak ? we only eat bark.
[03:58] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5081 -- I can't say I've heard of a dynamic dispatch table, is this a common technique or are you drawing from something similar ? fwiw, it sounds like a reasonable separation on first pass.
[03:58] sourcerer: 2022-10-14 01:24:36 (#jwrd) jfw: but the idea is to find which ones are best replaced by simple static calls and which ones if any might require actually implementing some kind of dynamic dispatch table.
[03:59] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5084 -- ok.
[03:59] sourcerer: 2022-10-14 03:24:05 (#jwrd) jfw: so for that one I'd think we're safe to remove the module loading (or keep it as a non-functional stub, as is the current situation) and those two plugins.
[04:00] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5085 -- enumerating goodness pays again.
[04:00] sourcerer: 2022-10-14 03:25:29 (#jwrd) jfw: that's the happy third category I guess, "doesn't need any code changes because the affected modules/plugins are of no possible interest".
[04:01] jfw: it's where you have a list of some sort, accessible at runtime, mapping names to functions (pointers really); pretty common thing especially in things that interpret textual commands of some sort. dlopen/dlsym act as such a thing where the list is drawn from the compiler-generated symbol table of the library file, so it's replacing that with a simpler special-purpose one.
[04:01] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5089 -- "what's your password" "the letter a"
[04:01] sourcerer: 2022-10-14 03:45:39 (#jwrd) jfw: this seems to me about the best that can be done under the circumstances, but still a dubious marriage; because the user password is transmitted routinely over the network so should be relatively ephemeral, while an encryption password is for life
[04:02] jfw: still hoping it's not necessary because implementing ~anything is tricky in C.
[04:02] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5107 -- thanks, cool.
[04:02] sourcerer: 2022-10-14 04:01:17 (#jwrd) jfw: it's where you have a list of some sort, accessible at runtime, mapping names to functions (pointers really); pretty common thing especially in things that interpret textual commands of some sort. dlopen/dlsym act as such a thing where the list is drawn from the compiler-generated symbol table of the library file, so it's replacing that with a simpler special-purpose one.
[04:03] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5110 -- word.
[04:03] sourcerer: 2022-10-14 04:02:15 (#jwrd) jfw: still hoping it's not necessary because implementing ~anything is tricky in C.
[04:03] dorion: jfw, need any help deciding on anything at this point ? or good to keep going down this path ?
[04:04] jfw: I'd say I'm good, just need to keep looking at each use of the module system to get to the bottom of it.
[04:04] dorion: cool
[04:10] jfw: actually "dynamic dispatch" is used more broadly than just translating strings to function calls; one way or another it's when a call is resolved at runtime from some set of possibilities. I think the term itself comes from OOP or at least that's where I first encountered it. being a 'dynamic language' like scheme or python means ~every identifier reference is resolved dynamically.
[04:14] jfw: actually pre-Common Lisp would be the better example; scheme prides itself on lexical scoping making for referential transparency.
[04:15] jfw: and in python terms it's mostly the 'dot' where dynamic lookups happen.
[04:18] dorion: cool, thanks for the likbez.
[04:19] jfw: a string-to-function dispatch table for handling user commands ; and a symbol-to-function one for internal use, "message passing"
[04:19] jfw: style akin to the 'dot' operator.
[04:22] jfw: and usage of the latter, where the write-cache symbol is passed to extract the procedure which is then called on a specific path.
[18:58] jfw: I dreamt last night that RSA was known to the ancient Greeks; a king was asking a philosopher to explain its wondrous workings to him. A specially trained cat was involved, which somehow helped with the heavy arithmetic by means of the patterns it scratched in the dirt.
[19:01] jfw: sadly I woke before he finished building up the explanation!
[19:08] jfw: ah, it wasn't the patterns the cat scratched, better than that - the granules of dirt functioned as a natural parallel computer from the way they tumbled over each other; the cat was only needed to get it moving and then the trick was in how to read the results.
[19:09] jfw: pretty much how quantum computing works from what I gather.
[19:13] jfw: the results were read with the euclidean toolkit, of course - compass, straightedge and divider.
[23:43] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5108 - even with a strong password, if the men in black wanted the data, presumably they'd simply compel the service provider to log the inbound passwords at whichever point they're in cleartext, turning the whole thing into a honeypot. nonetheless, this mode is how the btc web wallets survived, those that did, last I heard.
[23:43] sourcerer: 2022-10-14 04:01:35 (#jwrd) dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5089 -- "what's your password" "the letter a"
Day changed to 2022-10-15
[20:59] jfw: awwww, the kiddos thought they'd make the module suffix configurable... so long as it's always three characters, that should be enough for anyone!!
[21:23] jfw: an even more bizarre "encryption" plugin, which gets redundantly compiled and installed as two separate libraries, one at the top-level module path and one for the 'auth' subdir.
[21:46] jfw: anyways, continuing from filesystem drivers, the second use of the modules mechanism is for authentication related code, and in this category there are five possible modules. The only one styled as a 'plugin' is var-expand-crypt noted just above. The rest are... not exactly 'builtin' but included directly in src/auth/, call
[21:46] sourcerer: 2022-10-14 03:21:38 (#jwrd) jfw: for the first sample, looking at the pluggable filesystem drivers thing (lib-fs/fs-api.c): modules are loaded based on the filename pattern libfs_<driver>, where <driver> comes from the caller (perhaps ultimatly from a config file); but several of the drivers are builtin, so module loading only happens for plugin-supplied drivers; the in-tree plugins in this category are fs-compress and
[21:46] jfw: then 'non-plugin modules' I guess.
[21:49] jfw: these four contain authentication code for gssapi, ldap, lua (for custom scripts) and imap (for delegating authentication to another imap server).
[21:51] jfw: the first three are built conditional on the respective feature being enabled & external library being available at configure time, while imap (libauthdb_imap) as well as the plugin (lib20_auth_var_expand_crypt) are always built. in their expected dyn-linked configuration, all installed modules are loaded & initialized at startup.
[21:52] jfw: *call them 'non-plugin modules'
[21:53] jfw: *all installed authentication modules are loaded
[21:56] jfw: gssapi from what I've seen is essentially just a layer on top of kerberos, which, I dunno, at some point we could conceivably want, though it seems pretty dead to me.
[21:59] jfw: the one of most interest to us, sql, is not a module in this case but truly builtin (conditional on the feature being enabled).
[22:00] jfw: why the others are done as modules if they're similarly compile-time conditional and always loaded at runtime, is a mystery to me.
[22:03] jfw: in any case, again it would appear fairly safe to simply rm -rf the module code.
[22:03] sourcerer: 2022-10-14 03:24:05 (#jwrd) jfw: so for that one I'd think we're safe to remove the module loading (or keep it as a non-functional stub, as is the current situation) and those two plugins.
[22:13] jfw: The third use of the modules mechanism is by config/config-parser.c, which loads anything it finds in a 'settings' module subdir. Unless there's something hiding behind further layers of misdirection, there's nothing in the tree that actually installs any modules there, so it's dead code.
[22:16] jfw: I guess the idea is that somebody on the interwebs could have written a plugin that hooks into it.
[22:17] jfw: yet their lack of api stability and documentation clearly discourages such use.
[22:19] jfw: looks like there's somewhere around nine of these still to go.
[22:20] jfw: (instances of module loading)
[22:31] jfw: The fourth is by dict/main.c, which loads anything in a 'dict' module subdir. The only such module is libdict_ldap - but this already has the option of being builtin.
[22:45] jfw: the fifth is by lib-ssl-iostream/iostream-ssl.c which unsurprisingly enough loads teh openssl. my current configuration builds without this, and it's available module-style only.
[22:47] jfw: btw, I've been favoring real-time reporting here due to the uncertainty in how much remains and the value of continuing.
[22:49] dorion: jfw, good idea, getting logged up now.
[23:17] jfw: one encouraging finding is that dynamic symbol lookup is pretty limited compared to module loading generally; in particular, per my earlier map, it's only used by lib-fs/fs-api.c, config/config-parser.c and config/doveadm.c.
[23:17] sourcerer: 2022-10-13 04:47:56 (#jwrd) jfw: it's not just that the build system is broken (or unfit for this environment or suchlike) but the code itself at least in some places is built around dyn linking, specifically the ability to call functions by looking up their names as strings at runtime
[23:19] jfw: * config/doveconf.c not doveadm.
[23:22] jfw: meaning, I've perhaps already dispensed with them under the fs-api and config-parser cases.
[23:22] jfw: https://pigeonhole.dovecot.org/ came up as a possible external plugin of note that we haven't been considering yet.
[23:24] jfw: (provides 'sieve' filtering support, somehow I thought that was builtin by now.)
[23:24] jfw breaks for today
[23:30] dorion: ok. so you're 5/9ths through analysis, at least by count.
[23:30] sourcerer: 2022-10-15 22:19:51 (#jwrd) jfw: looks like there's somewhere around nine of these still to go.
Day changed to 2022-10-16
[00:41] jfw: dorion, correct. at this rate it seems sensible to push through to round out the picture.
[00:43] jfw: process design overview doc that gave me the idea that 'sieve' was included as a first-class component, showing it clear as day.
[00:44] dorion: jfw, I agree to rounding out the picture.
[00:44] jfw: dorion: I reckon whether or not 'sieve' is interesting will depend on whether the various webmail options support it
[00:45] jfw: but we might want to consider if or how mail filtering is to be supported for clients.
[00:47] dorion: jfw, ok. noted to dig into in rounding out my preliminary research.
[00:49] jfw: for background - in the POP days, filtering was necessarily a client side (MUA) consideration. when IMAP came along, the same approach still worked to the extent that you used only the one client, but didn't really fit the model. Various server-side things existed such as procmail which simply ran off a config file in the home directory, following the 'mail user == unix user' model.
[00:52] jfw: meanwhile the open source webmail offerings, at least from what we found so far, were built to rely on an IMAP server for the storage backend.
[00:54] jfw: so somewhere relatively recently, sieve came along as a more integrated way to do server-side filtering and expose its configuration to the client without requiring full shell access.
[00:55] dorion: alright, that's helpful background.
[00:55] jfw: I never used it myself so can't speak to whether it works or works well.
[00:58] jfw: strikes me as a failure of IMAP that they couldn't extend the protocol to support that need and instead cooked up some whole new 'managesieve' protocol, but what do I know.
Day changed to 2022-10-17
[20:12] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5149 -- re-reading, at this point you were at 3 and reported 9 to go. does that mean there are 12 total and I was mistaken ? or is 9 the total ?
[20:12] sourcerer: 2022-10-15 22:19:51 (#jwrd) jfw: looks like there's somewhere around nine of these still to go.
[20:12] sourcerer: 2022-10-15 23:30:31 (#jwrd) dorion: ok. so you're 5/9ths through analysis, at least by count.
[20:58] jfw: dorion: indeed my words were 9 to go.
[21:01] jfw: so, 5/12ths.
Day changed to 2022-10-20
[05:50] jfw: http://fixpoint.welshcomputing.com/2022/busybox-microcom-the-code-review/
[12:07] dorion: jfw, nice, well done.
Day changed to 2022-10-22
[02:58] jfw: here's the earlier paste promoted to permanent link.
[02:58] sourcerer: 2022-10-13 23:24:34 (#jwrd) jfw: for a first patch of solid ground to stand on, a file-level map of what calls dovecot's modules API and what executables each one ultimately gets linked into http://welshcomputing.com/paste/myiuuudunx
[04:00] jfw: our sixth instance of module loading, in the quest to find what modules are loaded by what components, is login-common/main.c; it's yet another special thing in that its module path and module list are configurable, but the default is a "login" subdir of the main modules dir; and I haven't found any modules installing themselves there.
[04:01] jfw: so, similar to the config-parser: possibly just used by out-of-tree plugins.
[04:22] jfw: seventh, old-stats/main.c; it loads from an "old-stats" module subdir; there are two such modules in the tree, and the disorder continues in that one of them is called a plugin and the other not but besides that they seem shaped exactly the same.
[04:25] jfw: these are libold_stats_mail and libstats_auth (because why be consistent even with the "old" renaming?)
[04:30] jfw: there's also an "imap-old-stats" plugin but this would appear NOT to be loaded by the old-stats binary; what does load it I don't yet know.
[04:38] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5183 -- smart.
[04:38] sourcerer: 2022-10-22 02:58:42 (#jwrd) jfw: here's the earlier paste promoted to permanent link.
[15:08] jfw: in demented and undocumented interface design, module_dir_load and family seem to load all modules in a given directory if a null pointer is passed as the module list string, but none of them if an empty string is passed.
[15:30] jfw: eighth up: doveadm/doveadm-mail.c, and now we finally get a bit more interesting. this links into doveadm which is a user-facing command, and it loads the configured "mail_plugins" from the configured "mail_plugin_dir", which defaults to the base module dir. These "mail plugins" seem to be the most numerous category.
[15:34] jfw: FTS is included there, along with any module/plugin not specifying a specific subdir.
[15:36] jfw: ninth is doveadm/doveadm-pw.c, which is another point where all present auth modules are loaded.
[15:36] sourcerer: 2022-10-15 21:46:44 (#jwrd) jfw: anyways, continuing from filesystem drivers, the second use of the modules mechanism is for authentication related code, and in this category there are five possible modules. The only one styled as a 'plugin' is var-expand-crypt noted just above. The rest are... not exactly 'builtin' but included directly in src/auth/, call
[15:37] jfw: tenth is doveadm/doveadm-util.c which loads all available doveadm modules ('doveadm' subdir).
[15:44] jfw: Further accentuating the lack of correspondence between plugins in the sense of subdirs of 'plugins' in the source tree, and the things that actually get loaded, some such 'plugins' include multiple modules in different categories, for instance a mail plugin and a doveadm plugin.
[15:45] jfw: "doveadm which is a user-facing command" - to clarify, that's user-facing as opposed to internally invoked; administrator-facing would be more precise.
[15:53] jfw: now as to the doveadm modules, the ones I'm seeing are all "true" plugins i.e. come from 'plugins' subdirs, namely: acl, quota, fts-lucene (if BUILD_LUCENE), fts, and mail-crypt.
[15:53] sourcerer: 2022-10-14 03:40:51 (#jwrd) jfw: the mail-crypt plugin, fwiw; the security model isn't clearly explained but looks like it can derive message encryption keys from the user's password.
[15:58] jfw: I guess this isn't entirely accurate, since we've already seen the implicit category of filesystem drivers defined by the libfs_ prefix rather than by module subdir.
[15:58] sourcerer: 2022-10-22 15:34:23 (#jwrd) jfw: FTS is included there, along with any module/plugin not specifying a specific subdir.
[15:58] sourcerer: 2022-10-14 03:21:38 (#jwrd) jfw: for the first sample, looking at the pluggable filesystem drivers thing (lib-fs/fs-api.c): modules are loaded based on the filename pattern libfs_<driver>, where <driver> comes from the caller (perhaps ultimatly from a config file); but several of the drivers are builtin, so module loading only happens for plugin-supplied drivers; the in-tree plugins in this category are fs-compress and
[16:27] jfw: eleventh, lib-storage/mail-user.c; this is another possibly more interesting case as, per the map, it's the first one we've seen so far that gets linked into most of the binaries (such as 'imap'), by way of libstorage -> libdovecot-storage.
[16:29] jfw: it loads a specified module from the configured mail_plugin_dir; I haven't got to the bottom of where the module name comes from, but it seems to be considered a "storage driver".
[16:37] jfw: this seems to refer to the mailbox formats which used to be configurable and I suppose were implemented as modules, but are now all builtin to libstorage (not a module).
[16:50] jfw: twelfth, lmtp/lmtp-client.c, similar to #8, loads the configured mail_plugins from the configured mail_plugin_dir; though there seems to be some 'lmtp' context to the settings ie the plugins to load might be configurable independently from that doveadm case or others. this gets linked only into the 'lmtp' exe.
[16:50] sourcerer: 2022-10-22 15:30:38 (#jwrd) jfw: eighth up: doveadm/doveadm-mail.c, and now we finally get a bit more interesting. this links into doveadm which is a user-facing command, and it loads the configured "mail_plugins" from the configured "mail_plugin_dir", which defaults to the base module dir. These "mail plugins" seem to be the most numerous category.
[17:21] jfw: grr, GNUTLS support is broken currently... but does that spare me from slogging through all the overhead of the pretense of interchangeable backends? nooooo
[17:25] jfw: especially weirdly, there's a lib-dcrypt/dcrypt-gnutls.c in git which disappeared in the release tarball without a trace except for a stray comment suggesting dcryp tcould be loaded with either openssl or gnutls.
[17:32] jfw: which brings us to the thirteenth: lib-dcrypt/dcrypt.c, which loads libdcrypt_openssl, which is built if BUILD_DCRYPT_OPENSSL. this is another that's linked into ~everything by way of libdovecot, but the things that actually attempt to initialize it are few: mail-crypt and var-expand-crypt (plugins), passdb_oauth2 (builtin), and some doveadm dumping commands.
[17:33] jfw: libdcrypt_openssl is installed to the top-level module dir, which they've now spelled as "pkglibdir" instead of "moduledir" so as to maximally burden the reader with automake documentation digs.
[17:40] jfw: I've been getting hints that there's a depenendency hell between certain modules, ie order of loading/initialization can matter
[17:54] jfw: fourteenth and finally is lib-storage/mail-storage-service.c ; again similar to #8 it loads mail_plugins from mail_plugin_dir; this time settings context (for lack of a better term) is mail user settings.
[17:54] sourcerer: 2022-10-22 15:30:38 (#jwrd) jfw: eighth up: doveadm/doveadm-mail.c, and now we finally get a bit more interesting. this links into doveadm which is a user-facing command, and it loads the configured "mail_plugins" from the configured "mail_plugin_dir", which defaults to the base module dir. These "mail plugins" seem to be the most numerous category.
[17:59] jfw: like #11 it's linked into many executables by way of libdovecot, and since #11 looked like a vestige, I suppose by process of elimination that this is where the chosen mail plugins get loaded by most executables that do such loading: a main entrance of sorts.
[17:59] sourcerer: 2022-10-22 16:27:16 (#jwrd) jfw: eleventh, lib-storage/mail-user.c; this is another possibly more interesting case as, per the map, it's the first one we've seen so far that gets linked into most of the binaries (such as 'imap'), by way of libstorage -> libdovecot-storage.
[18:08] jfw: a further confirmatory hint of this is that the directly upstack functions mail_storage_service_lookup and mail_storage_service_lookup_next are called from many places including doveadm, imap, indexer, lda, lmtp, pop3, submission, script-login, plugins/mail-crypt and plugins/quota.
[18:11] jfw: the gap between my guess of 12 and the final 14 cases most likely comes from the three doveadm files turning out to be distinct cases.
[18:13] dorion: jfw, quite the swamp. my first impression is it'll probably be more efficient to write the generic replacement for the modules functions that works statically, i.e. the dynamic dispatch table route, vs figuring out one by one which we need and replacing them with the static calls.
[18:13] dorion: what's your outlook ?
[18:15] jfw: dorion: it's not clear to me yet
[18:15] dorion: ok
[18:18] jfw: one question would be, they use this mail_plugins setting to control at runtime what plugins get loaded, and this might be significant i.e. a plugin changes the behavior of something just for its having been loaded. so, do we want to preserve that configurability, or make it all a compile-time decision?
[18:19] jfw: I'm inclined to throw maximum piss and bile at the whole plugin concept, go with the latter and make it truly static as much as feasible
[18:20] jfw: from the whole excavation, what modules came up as having any business being configurable in the first place? not much, it seems to me
[18:22] jfw: supposing some day we want to add quota or acl support or whatever, would do it by tweaking the gport.
[18:23] jfw: changing FTS backend
[18:25] dorion: I tend to agree, but would like to know your thoughts on easy of implementation for each.
[18:26] dorion: ease*
[18:35] jfw: that's what I'm having trouble predicting.
[18:37] jfw: might just need to give it a try, unless there's some more informative causes to act from.
[18:40] dorion: yeah, I can see it being a tough prediction.
[18:41] dorion: alright, you got the green light from me to piss and puke away, with the aim of cleaning up, lolz
[18:41] jfw: there's also still the "do nothing for now" option; now we have a better picture of all the modules that will be nonfunctional and it doesn't look too bad. fts and seive still seem possibly important, but search should work in any case, just slower
[18:44] dorion: hmm, right.
[18:45] dorion: so move onto packaging for now and then deployment and testing ?
[18:51] jfw: the build system is still ugly as sin; it builds & installs all the modules even though they're not usable. it does work though.
[18:52] dorion: well, any way we go is ugly.
[18:55] dorion: if we take the s/do nothing/deploy as is and test/ approach now and later decide to go back to pissing on plugin concept, how difficult do you think it'll be for you to pick back up ? the log and status reports are solid ? do you have other useful notes to help you load it back into head ?
[18:56] jfw: I should at least give it a try on the static modules route, while it's all fresh, I'm thinking.
[18:56] dorion: that second ? was supposed to be a ,
[18:57] jfw: the content is good, and there are some other notes from the dig; still there will be some overhead to resuming
[18:57] dorion: jfw, ok. you've spent a couple weeks w/ the code to get where you're at, any amount of notes still aren't going to be as good as where you're at now.
[18:59] dorion: keep w/ the updates as you see fit here so as to come up for air.
[19:00] dorion: I'm going to step away a few hours, be back later tonight.
[19:00] jfw: and static modules and a clean build system would be the best payoff for the work invested
[19:00] dorion: right.
[21:47] jfw: http://fixpoint.welshcomputing.com/2022/sounding-the-swamps-of-internet-mail-access/
Day changed to 2022-10-23
[04:28] jfw: and I've got the obligatory 600-word introduction down for an article structuring all those findings on the different cases of module loading and types of modules.
[04:29] jfw: my sympathies to the October log readers.
Day changed to 2022-10-26
[23:03] jfw: I'm realizing one lovely bit of self-contamination that's adding to the general confusion in dovecot is that they install internal shared libraries that aren't modules, that is, things that just link directly into various binaries rather than being dlopen-ed, right in the same directory as the modules.
[23:04] jfw: the build system search term for these seems to be pkglib_LTLIBRARIES, and the names start with libdovecot_ .
[23:06] jfw: actually just 'libdovecot'. there's the main libdovecot.la, then libdovecot-x.la for x in: compression, dsync, fts, lda, login, sql, storage.
[23:07] jfw: add ldap, lua, storage-lua if the respective deps are enabled.
[23:56] jfw: http://fixpoint.welshcomputing.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/
Day changed to 2022-10-27
[00:12] jfw: that one was like passing a kidney stone (he says, in blissful ignorance of what that actually feels like) - trying to build some kind of order from a very disorderly pile. hopefully the reading goes down smoother.
[14:47] dorion: jfw, http://fixpoint.welshcomputing.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2233
[17:16] jfw: dorion, replied.
Day changed to 2022-10-28
[02:13] jfw: + commented re SQL drivers mystery solved.
Day changed to 2022-10-29
[04:10] jfw: http://welshcomputing.com/paste/vy3vw52fy5 - preview of what's cooking for static dovecot module support.
[04:13] jfw: http://welshcomputing.com/paste/csg3rfi9m5 - corrected (missed passing a new parameter)
[16:44] dorion: jfw, looks slick.
[19:58] jfw: tenatively done with the conversion for load points 8, 10, 12 and 14 (and didn't even have to kill lmtp)
[19:58] jfw: *tentatively
[20:01] jfw: will need to either revive my previous module_dir_try_load_missing stub or follow through on removing the condemned load points for them to be able to link.
[20:02] jfw: then I'll need to look at the build system and decide whether it's time to flush the autoshit or continue with incremental changes to get the new stuff working.
[01:59] jfw: pointing out this instance where the GNU autoconf people still haven't fixed their error messages, 19 years later.
[02:06] jfw: or their documentation for AC_DEFINE which gives no hint that the 'description' parameter is mandatory.
[16:04] caai: jfw: well, maybe they are waiting for the 2 decade mark to start working on it. ha! is this affecting a 'missing template problem' task that you are working on?
Day changed to 2022-10-03
[02:56] jfw: caai: the fatlogic here looks more like "if we just don't respond to the person pointing out the flaw then we can safely forget it happened and go on pretending the flaw doesn't exist"
[02:58] jfw: a pretty common response too, because it's the easiest
[03:02] jfw: it was a relatively minor nuisance in what I was working on, just an impressively pristine specimen of the failure.
[03:02] jfw: especially because it would have been such a simple fix!
[03:11] jfw: the slightly larger view of what I was up to was bypassing this nonsense, which involved writing ~16 lines of autoconf code as a stub in place of the original ~293 lines
[03:17] jfw: some of the madness in that first link is worth explicitly calling out too: "GNU libiconv, if installed, is not necessarily already in the search path" - if it's not in the search path, then either it's not installed, or it's not meant to be used by default. otherwise IT WOULD BE IN THE SEARCH PATH.
[03:27] jfw: nothing like dealing with the actual problems created by the "solutions" to hallucinated problems.
[15:03] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Sep-2022/#4920 -- did your bro fare ok ?
[15:03] sourcerer: 2022-09-30 22:07:11 (#jwrd) caai: i am glad to hear that you came out unscathed, minus a sleepless night. i saw some footage from the west coast and it looks like there was substantial damage.
[16:08] jfw: dorion: you mean our Naples - St Petersburg guy? he was out of state at the time. dunno what he'll find when he returns.
[16:48] dorion: I mean caai's blood bro who lives on the left coast.
Day changed to 2022-10-04
[19:40] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4927: yes, too bad they have let it collect dust for so long
[19:40] sourcerer: 2022-10-03 03:02:33 (#jwrd) jfw: especially because it would have been such a simple fix!
[19:41] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4934: he was fortunate and only lost a few branches off a palm trees. he is located about 20 miles from the beach in Sarasota
[19:41] sourcerer: 2022-10-03 16:48:22 (#jwrd) dorion: I mean caai's blood bro who lives on the left coast.
[19:41] caai: off a palm tree*
[20:40] caai: jfw: could you help me with this question? In this class, we are using CentOS. Question: What does your .profile file say is your default PATH? Note: If you do not have a .profile, please examine your .bashrc file in your home directory.
[20:40] caai: # User specific environment
[20:40] caai: PATH="$HOME/.local/bin:$HOME/bin:$PATH"
[20:40] caai: export PATH
[20:40] caai: PS1="Your majesty ? "
[20:40] caai: export PS1
[20:42] caai: my username is is229044, therefore my default PATH should be: is229044/.local/bin
[20:42] caai: Does that seem correct to you?
[20:45] dorion: caai, it's the whole string. and to check what $HOME is, you can issue `echo $HOME`
[20:56] caai: dorion: got it. thanks
[21:22] jfw: this is where I would lose points on the exam by stubbornly pointing out that the question is ill-defined, heh.
[21:23] jfw: the profile is defining the PATH in terms of other variables - including the pre-existing PATH - that are not apparent from the profile itself.
[21:25] jfw: so most literally you could give the exact string ($HOME/.local...) but that's not the same as the value that the variable actually gets i.e. what the default PATH actually *is*.
[21:27] jfw: lolz @ "your majesty" prompt, sounds like "I know I'm important because my dog loves me"
[21:27] jfw: the computer's love is cheaper still, doesn't require daily attention!
[23:20] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4951: eso!
[23:20] sourcerer: 2022-10-04 21:23:51 (#jwrd) jfw: the profile is defining the PATH in terms of other variables - including the pre-existing PATH - that are not apparent from the profile itself.
[23:21] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4952: i will copy and paste this response and report back hahaha (or maybe not)
[23:21] sourcerer: 2022-10-04 21:25:46 (#jwrd) jfw: so most literally you could give the exact string ($HOME/.local...) but that's not the same as the value that the variable actually gets i.e. what the default PATH actually *is*.
[23:35] caai: report back what the teacher's response is*
[23:48] jfw: alright good luck. dorion is also correct that you need the whole thing where you focused in on just the first component of the colon-delimited list.
Day changed to 2022-10-05
[00:12] caai: understood. thanks
[01:54] jfw: to document a bit of weird that came up in Gales: I'm running the ./configure script for a program that includes autoconf code probing for a working -fstack-protector and/or -fstack-protector-strong GCC options. The first at least is supposed to work and is even enabled by default in Gales, yet the probe concludes that it doesn't, and a manual test "gcc -fstack-protector file.c" reports "ld:
[01:54] jfw: cannot find -lssp_nonshared". How can this be?
[01:58] jfw: a hint is found here; basically, musl includes the necessary support function to which gcc generates calls, except there's *another* one that it also generates only on certain platforms, which for some odd reasons can't be implemented directly in libc.
[01:59] jfw: this then is supposed to be found in "libssp_nonshared.a", which is provided by neither gcc nor musl, but the trivial implementation is found in musl-based distros such as Alpine.
[02:01] jfw: in turn, the gcc patch produced by the musl people to tell gcc how to work with musl, causes it to attempt to link this libssp_nonshared when -fstack-protector is in use.
[02:03] jfw: but that means when it's used at link time; so "gcc -fstack-protector -c file.c" (compile-only mode) works as expected, then "gcc file.o" works to link it, because on amd64 the offending code is never generated.
[02:04] jfw: "gcc -fstack-protector file.o" then will reproduce the error as in the all-at-once usage.
[02:06] jfw: the "enabled by default" part works by adding it to the "cc1" flags, in effect making "gcc -c file.c" equivalent to "gcc -fstack-protector -c file.c", but there's no such default configured for link mode, which is why we haven't seen the error before.
[02:08] jfw: so in short, there's no problem at present, but if we port to one of the platforms that needs it - reportedly i386 and (32-bit) powerpc - we'll probably need to 1) import the glue code from Alpine and 2) further tweak the gcc specs to always link -lssp_nonshared.
[02:20] jfw: Relatedly: we decided reluctantly and after some deliberation to add a gport for GNU Libtool; to correct its own description, a generic library support script that papers over the complexity of using shared libraries on totally obscure systems by piling on more complexity behind an obnoxious interface.
[02:23] jfw: Previously I'd been attempting to cut out the usage of libtool wherever it turned up in the autoconf/automake inputs, and this is certainly doable but the exact process varies with each codebase, and it came up so often that I started to see libtool in practice as more of a limb of the larger autotools body, it's simply expected to be there.
[02:28] jfw: The preferred situation is to cut out the autoshit entirely (which we've also done successfully in several cases, but again costing time); to this end, surgical extraction of libtool doesn't really make progress, ie it's purely an additional cost along the way. And on the ugly side, there are some ports where we're still running the auto-generated configure turds as shipped in the tarballs,
[02:28] jfw: because we're unable to regenerate them, having punted on the libtool work.
[02:31] jfw: So the idea (so far validated in practice) is that having it available will make more things "just work" in a more hygienic way than before, letting the sleeping dogs lie and making it easier to replace the whole rotten build system if/when the time comes.
[02:34] jfw: However, we have no intention of *supporting* libtool; the moment it causes problems is the moment it gets thrown out for the particular case.
[02:35] jfw: The same as can probably be said for the rest of all that autoconf, automake, cmake etc etc.
[02:43] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4970 -- ok, that certainly is some weird. glad to hear a) you've grasped it, b) it's not an issue for the present job and c) it's noted should we move to future porting.
[02:43] sourcerer: 2022-10-05 02:08:20 (#jwrd) jfw: so in short, there's no problem at present, but if we port to one of the platforms that needs it - reportedly i386 and (32-bit) powerpc - we'll probably need to 1) import the glue code from Alpine and 2) further tweak the gcc specs to always link -lssp_nonshared.
[02:43] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4971 -- lol, nice correction.
[02:43] sourcerer: 2022-10-05 02:20:06 (#jwrd) jfw: Relatedly: we decided reluctantly and after some deliberation to add a gport for GNU Libtool; to correct its own description, a generic library support script that papers over the complexity of using shared libraries on totally obscure systems by piling on more complexity behind an obnoxious interface.
[02:45] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4976 -- word.
[02:45] sourcerer: 2022-10-05 02:34:14 (#jwrd) jfw: However, we have no intention of *supporting* libtool; the moment it causes problems is the moment it gets thrown out for the particular case.
[02:58] jfw: Interestingly, the Alpine port for musl also includes small implementations of some shell utilities traditionally coming from glibc: getconf, getent, and iconv, which might be worth importing directly.
[02:59] jfw: They're covered in the Linux and/or POSIX man pages, but otherwise missing in Gales since they're not included directly in musl.
[03:11] jfw: I know I've personally used "getent" in shell scripts.
Day changed to 2022-10-08
[17:56] jfw: https://dovecot.org/pipermail/dovecot/2022-October/125452.html << my latest dispatch from the front.
Day changed to 2022-10-09
[14:17] caai: question: when a new linux install is done, upon initial start up, it prompts you to create a 'user'. would you suggest that first user be 'root' and then i create a lower-privileged user for everyday tasks, e.g., 'user'?
[14:26] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4987: :/ not cool
[14:26] sourcerer: 2022-10-08 17:56:07 (#jwrd) jfw: https://dovecot.org/pipermail/dovecot/2022-October/125452.html << my latest dispatch from the front.
[18:29] jfw: caai: some distributions indeed have this "fist boot" configuration wizard thing. The root user will always exist already - typically it would have prompted you to set the root password at install time - so this is for creating that unprivileged user for everyday tasks.
[18:43] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4989 - it's the diplomacy front, really; compatibility problems are pretty much a given when trying to work in the swamps and most likely it falls on me to fix others' broken stuff, but we poke them anyway to see what they're made of.
[18:43] sourcerer: 2022-10-09 14:26:18 (#jwrd) caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4987: :/ not cool
[18:45] jfw: btw, it's better to add a space after URLs like that, since eg : is a valid URL-character so linkifiers will include it in the reference. though here at least the bot can handle it for quoting because it recognizes its fixed prefix pattern.
[18:47] jfw: * "first boot" not "fist boot", lol.
[19:19] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4987 -- the swamp speaks : https://dovecot.org/pipermail/dovecot/2022-October/125458.html
[19:19] sourcerer: 2022-10-08 17:56:07 (#jwrd) jfw: https://dovecot.org/pipermail/dovecot/2022-October/125452.html << my latest dispatch from the front.
[21:07] jfw: then, you do the work and we'll consider merging at some point - a weaker offer even than "we will review"
[21:14] dorion: myeah, so what are you thinking, is it fix their shit or rip out the autoshit entirely ?
[21:20] jfw: leaning toward the latter since evidently it wasn't providing the advertised advantage of "configures to adapt to many kinds of systems" anyway
[21:25] dorion: ok by me, why subsidize them.
[21:26] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4976 after all.
[21:26] sourcerer: 2022-10-05 02:34:14 (#jwrd) jfw: However, we have no intention of *supporting* libtool; the moment it causes problems is the moment it gets thrown out for the particular case.
[21:53] jfw: plus we already know their development direction is removing things that looked useful to us like fts-squat and auth-checkpassword.
Day changed to 2022-10-10
[01:05] jfw: dorion: I think the ripping/fixing in this case will have to involve some study of how the plugin interface works, because it builds many binaries for different sub-components and I'll need to know which ones they need to be linked with.
[01:06] jfw: sorry, confusing wording: dovecot builds many binaries for different sub-components, and I'll need to know which of those the plugins need to be linked into.
[01:07] dorion: jfw, ok. probably won't be your favorite thing to do, but sounds like good time spent wrt getting to know the tool. much better than wrangling the autoshit.
[01:11] jfw: I won't mind the work itself, but the grass does momentarily appear greener on the "imap is easy, you just apt-get something and edit some configs" side
[01:13] jfw: (accompanied necessarily by "and don't forget to drink your security vaccines^W updates because you don't stand a chance to maintain it yourself")
[01:30] dorion: I hear you, tis the draw of the fast food.
[01:33] dorion: the bait and the hook.
[18:45] caai: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#4991: is 'fist boot' a synonym for 'glove'? hahaha. unfortunately, this lower priviledged user that is created via the wizard thing (fedora workstation) has superpriviledges via 'sudo' and a simple entering of said user's pw. i will have to change that.
[18:45] sourcerer: 2022-10-09 18:29:19 (#jwrd) jfw: caai: some distributions indeed have this "fist boot" configuration wizard thing. The root user will always exist already - typically it would have prompted you to set the root password at install time - so this is for creating that unprivileged user for everyday tasks.
[18:57] caai: btw, the reason i installed fedora, with all its wizardry, is becuase 'Linux+ and LPIC-1: Guide to Linux Certification', which is a great book, references fedora workstation in all of its examples, and i like to be able to follow along a book verbatim whilst learning
[19:11] jfw: fwiw, I put up with fedora for a couple years, and certainly no harm in seeing more of the world outside the jwrd garden.
Day changed to 2022-10-11
[14:16] caai: good to know. what would you say its advantages vs disadvantages are?
[18:53] jfw: caai: the main advantage would be a large collection of recent versions of software available for easy installation; the accompanying disadvantage is that various things will be buggy or break unpredictably.
[18:54] jfw: with little recourse other than to wait and try the next update, seeing what new things are broken.
Day changed to 2022-10-12
[21:41] caai: i see. then this 'roll it out as fast as possible mentality' exists in the linux world as well ....
[21:47] jfw: perhaps it's more like, following the fashions and calling it progress
[21:50] jfw: or, aiming to be adequate for everyone and thus excellent for no one, ala the supermarket model of shopping.
[22:00] jfw: on the supermarket transformation process
Day changed to 2022-10-13
[04:46] jfw: dorion: I mentioned out-of-band but it's now becoming clearer that the choice isn't so simple as this
[04:46] sourcerer: 2022-10-09 21:14:46 (#jwrd) dorion: myeah, so what are you thinking, is it fix their shit or rip out the autoshit entirely ?
[04:47] dorion: jfw: ok, listening.
[04:47] jfw: it's not just that the build system is broken (or unfit for this environment or suchlike) but the code itself at least in some places is built around dyn linking, specifically the ability to call functions by looking up their names as strings at runtime
[04:48] dorion: oy vey.
[04:49] jfw: a hint of this is in the plugin API doc, but I'm finding that it's more than just those two init/deinit things it covers.
[04:50] jfw: forinstance
[04:52] jfw: which means perhaps I'll need to review the 'modules' api just to find all possible instances
[04:52] dorion: tangentially related, that seems to make the mailing list reply even worse.
[04:53] jfw: myeah, "you come up with a fix [because we certainly don't want to, knowing how difficult we made it]"
[04:54] dorion: so it's still theoretically possible, but lots more work than we expected ?
[04:54] dorion: jfw, how many modules are there ?
[04:55] dorion: have you got a grasp on which ones we want ? at least at a high level, not counting dependencies.
[04:55] jfw: well, that's the other angle which is that the included plugins don't look all that interesting to me with the previously noted exception of fts (full-text search indexing)
[04:56] jfw: approximate list
[04:56] jfw: ("module" seems to mean the same as "plugin" from what I've seen so far)
[04:58] jfw: what I meant by "modules api" was the always-included code that it uses for dealing with plugins, containing things like that offending "module_get_symbol_quiet" along with the docs-mentioned "module_dir_init"
[04:58] dorion: maybe mysql plugin in the future.
[04:59] jfw: ah, that one's a "driver" not a "plugin" - not afflicted by this trouble.
[04:59] dorion: ok.
[05:00] dorion: what does fts dependency graph look like ? I see a few fts there.
[05:02] jfw: from previous reconnaisance, we'd want either fts-squat which they've deprecated, or the new fts-flatcurve which pulls in a "xapian" dependency. I don't think they depend on any of the other plugins if that's what you mean.
[05:03] dorion: other plugins is what I mean.
[05:04] dorion: and from what I recall, we didn't see very good reason why fts-squat was deprecated, apart from boredom.
[05:06] jfw: we didn't find a reason, but that doesn't mean there wasn't one. expanding only, unable to shrink possibly?
[05:08] jfw: confirmed that in both cases only the base "fts" plugin is depended upon.
[05:09] jfw: (granted that still doesn't look like a *good* reason to ditch their own code and run off to some new shiny library)
[05:10] jfw: but good or reasonable or no, deprecated it is so we can be sure of no further help from them.
[05:11] dorion: would that FIXME need to be fixed immediately. or would it be a problem only when the searchable text reaches a certain point ?
[05:12] jfw: afaik it just means the index files won't shrink when you delete messages or move to another mailbox/folder. probably could be worked around by deleting & regenerating.
[05:13] dorion: so even a cron job could be a first step to mitigating ?
[05:14] jfw: or even just an occasional maintenance script for when the disk is getting too full.
[05:15] jfw: (routine reindexing might be a performance drain)
[05:16] dorion: seems like this trouble isn't enough to abandon the gporting and taking a crack at cryus just yet.
[05:18] jfw: you refer to cyrus-imapd, being the main competitor we turned up with comparable features & performance.
[05:19] dorion: right. typo.
[05:19] jfw: (also filling in for the logs.) but what are you suggesting - keep at the investigation of the modules code?
[05:20] dorion: yeah.
[05:20] dorion: at least another day focused on what we're more interested in.
[05:21] dorion: what do you think ?
[05:21] jfw: I tend to agree. the extent of the problem may still turn out to be shallow.
[05:21] dorion: ok, cool. probe some more and lettuce see.
[05:25] jfw: thanks.
[05:25] dorion: thank you. talk w/ you tomorrow.
[19:02] jfw: I was unfortunately mistaken about dovecot's notions of module vs plugin, module is used more generally and includes things like authentication backends and indeed SQL drivers... though it looks like the latter already have provision for being built-in (static).
[19:02] sourcerer: 2022-10-13 04:56:21 (#jwrd) jfw: ("module" seems to mean the same as "plugin" from what I've seen so far)
[23:24] jfw: for a first patch of solid ground to stand on, a file-level map of what calls dovecot's modules API and what executables each one ultimately gets linked into http://welshcomputing.com/paste/myiuuudunx
[23:26] jfw: it made for a strong exercise in learning to decode what all is going on in the Makefile.am files
[23:28] jfw: not to mention getting a closer view of what all is in there - an http client & server ?! here I was mentally knocking cyrus for throwing in such kitchen sinks
[23:41] jfw: l0l... thinking about those mailing list replies, if they're saying they only support their code when used in the ways that they currently use it... well, necessarily there are no bugs in that usage, otherwise they wouldn't be able to use it; therefore, the supported portion of dovecot is necessarily a subset of the bug-free portion; so no bugs can be reported.
[23:53] jfw: the map should provide a complete if not necessarily concise or optimal answer to what the plugins can possibly be called from.
[23:53] sourcerer: 2022-10-10 01:05:17 (#jwrd) jfw: dorion: I think the ripping/fixing in this case will have to involve some study of how the plugin interface works, because it builds many binaries for different sub-components and I'll need to know which ones they need to be linked with.
Day changed to 2022-10-14
[00:00] jfw: which is to say, ~everything ; so most likely the approach is to bundle all desired plugins into the big libdovecot.a, then any executable that invokes them gets the right files linked via the normal workings of the linker.
[00:00] jfw: once the dlsym calls are replaced by static calls, that is.
[00:08] jfw: even their filesystem interface has to be abstracted into modular drivers ?!
[01:18] jfw: a next guiding question: for each call to the functions responsible for loading or initializing modules (module_dir_init, module_dir_load, module_dir_load_missing, module_dir_try_load_missing), can it be statically determined which modules it's trying to load? My guess is that for the more "internal" modules it can, and only the "true plugins" work in the mode of "grab whatever we find in the
[01:18] jfw: filesystem".
[01:19] jfw: the distinction is a bit subjective since of course it's all here in the source tree.
[01:24] jfw: but the idea is to find which ones are best replaced by simple static calls and which ones if any might require actually implementing some kind of dynamic dispatch table.
[03:21] jfw: for the first sample, looking at the pluggable filesystem drivers thing (lib-fs/fs-api.c): modules are loaded based on the filename pattern libfs_<driver>, where <driver> comes from the caller (perhaps ultimatly from a config file); but several of the drivers are builtin, so module loading only happens for plugin-supplied drivers; the in-tree plugins in this category are fs-compress and
[03:21] jfw: mail-crypt.
[03:24] jfw: so for that one I'd think we're safe to remove the module loading (or keep it as a non-functional stub, as is the current situation) and those two plugins.
[03:25] jfw: that's the happy third category I guess, "doesn't need any code changes because the affected modules/plugins are of no possible interest".
[03:40] jfw: the mail-crypt plugin, fwiw; the security model isn't clearly explained but looks like it can derive message encryption keys from the user's password.
[03:45] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5067 -- alright.
[03:45] sourcerer: 2022-10-13 19:02:44 (#jwrd) jfw: I was unfortunately mistaken about dovecot's notions of module vs plugin, module is used more generally and includes things like authentication backends and indeed SQL drivers... though it looks like the latter already have provision for being built-in (static).
[03:45] jfw: this seems to me about the best that can be done under the circumstances, but still a dubious marriage; because the user password is transmitted routinely over the network so should be relatively ephemeral, while an encryption password is for life
[03:48] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5069 -- nice, pretty solid indeed.
[03:48] sourcerer: 2022-10-13 23:24:34 (#jwrd) jfw: for a first patch of solid ground to stand on, a file-level map of what calls dovecot's modules API and what executables each one ultimately gets linked into http://welshcomputing.com/paste/myiuuudunx
[03:49] jfw: why thank you.
[03:50] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5071 -- holy smokes.
[03:50] sourcerer: 2022-10-13 23:28:32 (#jwrd) jfw: not to mention getting a closer view of what all is in there - an http client & server ?! here I was mentally knocking cyrus for throwing in such kitchen sinks
[03:51] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5072 -- or perhaps they've grown numb to the bugs crawling in their neckbeards.
[03:51] sourcerer: 2022-10-13 23:41:16 (#jwrd) jfw: l0l... thinking about those mailing list replies, if they're saying they only support their code when used in the ways that they currently use it... well, necessarily there are no bugs in that usage, otherwise they wouldn't be able to use it; therefore, the supported portion of dovecot is necessarily a subset of the bug-free portion; so no bugs can be reported.
[03:54] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5075 -- alright.
[03:54] sourcerer: 2022-10-14 00:00:02 (#jwrd) jfw: which is to say, ~everything ; so most likely the approach is to bundle all desired plugins into the big libdovecot.a, then any executable that invokes them gets the right files linked via the normal workings of the linker.
[03:55] jfw: perhaps numb, but I'd say I hit them with something sufficiently out of their ordinary diet.
[03:55] dorion: what is this steak ? we only eat bark.
[03:58] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5081 -- I can't say I've heard of a dynamic dispatch table, is this a common technique or are you drawing from something similar ? fwiw, it sounds like a reasonable separation on first pass.
[03:58] sourcerer: 2022-10-14 01:24:36 (#jwrd) jfw: but the idea is to find which ones are best replaced by simple static calls and which ones if any might require actually implementing some kind of dynamic dispatch table.
[03:59] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5084 -- ok.
[03:59] sourcerer: 2022-10-14 03:24:05 (#jwrd) jfw: so for that one I'd think we're safe to remove the module loading (or keep it as a non-functional stub, as is the current situation) and those two plugins.
[04:00] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5085 -- enumerating goodness pays again.
[04:00] sourcerer: 2022-10-14 03:25:29 (#jwrd) jfw: that's the happy third category I guess, "doesn't need any code changes because the affected modules/plugins are of no possible interest".
[04:01] jfw: it's where you have a list of some sort, accessible at runtime, mapping names to functions (pointers really); pretty common thing especially in things that interpret textual commands of some sort. dlopen/dlsym act as such a thing where the list is drawn from the compiler-generated symbol table of the library file, so it's replacing that with a simpler special-purpose one.
[04:01] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5089 -- "what's your password" "the letter a"
[04:01] sourcerer: 2022-10-14 03:45:39 (#jwrd) jfw: this seems to me about the best that can be done under the circumstances, but still a dubious marriage; because the user password is transmitted routinely over the network so should be relatively ephemeral, while an encryption password is for life
[04:02] jfw: still hoping it's not necessary because implementing ~anything is tricky in C.
[04:02] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5107 -- thanks, cool.
[04:02] sourcerer: 2022-10-14 04:01:17 (#jwrd) jfw: it's where you have a list of some sort, accessible at runtime, mapping names to functions (pointers really); pretty common thing especially in things that interpret textual commands of some sort. dlopen/dlsym act as such a thing where the list is drawn from the compiler-generated symbol table of the library file, so it's replacing that with a simpler special-purpose one.
[04:03] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5110 -- word.
[04:03] sourcerer: 2022-10-14 04:02:15 (#jwrd) jfw: still hoping it's not necessary because implementing ~anything is tricky in C.
[04:03] dorion: jfw, need any help deciding on anything at this point ? or good to keep going down this path ?
[04:04] jfw: I'd say I'm good, just need to keep looking at each use of the module system to get to the bottom of it.
[04:04] dorion: cool
[04:10] jfw: actually "dynamic dispatch" is used more broadly than just translating strings to function calls; one way or another it's when a call is resolved at runtime from some set of possibilities. I think the term itself comes from OOP or at least that's where I first encountered it. being a 'dynamic language' like scheme or python means ~every identifier reference is resolved dynamically.
[04:14] jfw: actually pre-Common Lisp would be the better example; scheme prides itself on lexical scoping making for referential transparency.
[04:15] jfw: and in python terms it's mostly the 'dot' where dynamic lookups happen.
[04:18] dorion: cool, thanks for the likbez.
[04:19] jfw: a string-to-function dispatch table for handling user commands ; and a symbol-to-function one for internal use, "message passing"
[04:19] jfw: style akin to the 'dot' operator.
[04:22] jfw: and usage of the latter, where the write-cache symbol is passed to extract the procedure which is then called on a specific path.
[18:58] jfw: I dreamt last night that RSA was known to the ancient Greeks; a king was asking a philosopher to explain its wondrous workings to him. A specially trained cat was involved, which somehow helped with the heavy arithmetic by means of the patterns it scratched in the dirt.
[19:01] jfw: sadly I woke before he finished building up the explanation!
[19:08] jfw: ah, it wasn't the patterns the cat scratched, better than that - the granules of dirt functioned as a natural parallel computer from the way they tumbled over each other; the cat was only needed to get it moving and then the trick was in how to read the results.
[19:09] jfw: pretty much how quantum computing works from what I gather.
[19:13] jfw: the results were read with the euclidean toolkit, of course - compass, straightedge and divider.
[23:43] jfw: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5108 - even with a strong password, if the men in black wanted the data, presumably they'd simply compel the service provider to log the inbound passwords at whichever point they're in cleartext, turning the whole thing into a honeypot. nonetheless, this mode is how the btc web wallets survived, those that did, last I heard.
[23:43] sourcerer: 2022-10-14 04:01:35 (#jwrd) dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5089 -- "what's your password" "the letter a"
Day changed to 2022-10-15
[20:59] jfw: awwww, the kiddos thought they'd make the module suffix configurable... so long as it's always three characters, that should be enough for anyone!!
[21:23] jfw: an even more bizarre "encryption" plugin, which gets redundantly compiled and installed as two separate libraries, one at the top-level module path and one for the 'auth' subdir.
[21:46] jfw: anyways, continuing from filesystem drivers, the second use of the modules mechanism is for authentication related code, and in this category there are five possible modules. The only one styled as a 'plugin' is var-expand-crypt noted just above. The rest are... not exactly 'builtin' but included directly in src/auth/, call
[21:46] sourcerer: 2022-10-14 03:21:38 (#jwrd) jfw: for the first sample, looking at the pluggable filesystem drivers thing (lib-fs/fs-api.c): modules are loaded based on the filename pattern libfs_<driver>, where <driver> comes from the caller (perhaps ultimatly from a config file); but several of the drivers are builtin, so module loading only happens for plugin-supplied drivers; the in-tree plugins in this category are fs-compress and
[21:46] jfw: then 'non-plugin modules' I guess.
[21:49] jfw: these four contain authentication code for gssapi, ldap, lua (for custom scripts) and imap (for delegating authentication to another imap server).
[21:51] jfw: the first three are built conditional on the respective feature being enabled & external library being available at configure time, while imap (libauthdb_imap) as well as the plugin (lib20_auth_var_expand_crypt) are always built. in their expected dyn-linked configuration, all installed modules are loaded & initialized at startup.
[21:52] jfw: *call them 'non-plugin modules'
[21:53] jfw: *all installed authentication modules are loaded
[21:56] jfw: gssapi from what I've seen is essentially just a layer on top of kerberos, which, I dunno, at some point we could conceivably want, though it seems pretty dead to me.
[21:59] jfw: the one of most interest to us, sql, is not a module in this case but truly builtin (conditional on the feature being enabled).
[22:00] jfw: why the others are done as modules if they're similarly compile-time conditional and always loaded at runtime, is a mystery to me.
[22:03] jfw: in any case, again it would appear fairly safe to simply rm -rf the module code.
[22:03] sourcerer: 2022-10-14 03:24:05 (#jwrd) jfw: so for that one I'd think we're safe to remove the module loading (or keep it as a non-functional stub, as is the current situation) and those two plugins.
[22:13] jfw: The third use of the modules mechanism is by config/config-parser.c, which loads anything it finds in a 'settings' module subdir. Unless there's something hiding behind further layers of misdirection, there's nothing in the tree that actually installs any modules there, so it's dead code.
[22:16] jfw: I guess the idea is that somebody on the interwebs could have written a plugin that hooks into it.
[22:17] jfw: yet their lack of api stability and documentation clearly discourages such use.
[22:19] jfw: looks like there's somewhere around nine of these still to go.
[22:20] jfw: (instances of module loading)
[22:31] jfw: The fourth is by dict/main.c, which loads anything in a 'dict' module subdir. The only such module is libdict_ldap - but this already has the option of being builtin.
[22:45] jfw: the fifth is by lib-ssl-iostream/iostream-ssl.c which unsurprisingly enough loads teh openssl. my current configuration builds without this, and it's available module-style only.
[22:47] jfw: btw, I've been favoring real-time reporting here due to the uncertainty in how much remains and the value of continuing.
[22:49] dorion: jfw, good idea, getting logged up now.
[23:17] jfw: one encouraging finding is that dynamic symbol lookup is pretty limited compared to module loading generally; in particular, per my earlier map, it's only used by lib-fs/fs-api.c, config/config-parser.c and config/doveadm.c.
[23:17] sourcerer: 2022-10-13 04:47:56 (#jwrd) jfw: it's not just that the build system is broken (or unfit for this environment or suchlike) but the code itself at least in some places is built around dyn linking, specifically the ability to call functions by looking up their names as strings at runtime
[23:19] jfw: * config/doveconf.c not doveadm.
[23:22] jfw: meaning, I've perhaps already dispensed with them under the fs-api and config-parser cases.
[23:22] jfw: https://pigeonhole.dovecot.org/ came up as a possible external plugin of note that we haven't been considering yet.
[23:24] jfw: (provides 'sieve' filtering support, somehow I thought that was builtin by now.)
[23:24] jfw breaks for today
[23:30] dorion: ok. so you're 5/9ths through analysis, at least by count.
[23:30] sourcerer: 2022-10-15 22:19:51 (#jwrd) jfw: looks like there's somewhere around nine of these still to go.
Day changed to 2022-10-16
[00:41] jfw: dorion, correct. at this rate it seems sensible to push through to round out the picture.
[00:43] jfw: process design overview doc that gave me the idea that 'sieve' was included as a first-class component, showing it clear as day.
[00:44] dorion: jfw, I agree to rounding out the picture.
[00:44] jfw: dorion: I reckon whether or not 'sieve' is interesting will depend on whether the various webmail options support it
[00:45] jfw: but we might want to consider if or how mail filtering is to be supported for clients.
[00:47] dorion: jfw, ok. noted to dig into in rounding out my preliminary research.
[00:49] jfw: for background - in the POP days, filtering was necessarily a client side (MUA) consideration. when IMAP came along, the same approach still worked to the extent that you used only the one client, but didn't really fit the model. Various server-side things existed such as procmail which simply ran off a config file in the home directory, following the 'mail user == unix user' model.
[00:52] jfw: meanwhile the open source webmail offerings, at least from what we found so far, were built to rely on an IMAP server for the storage backend.
[00:54] jfw: so somewhere relatively recently, sieve came along as a more integrated way to do server-side filtering and expose its configuration to the client without requiring full shell access.
[00:55] dorion: alright, that's helpful background.
[00:55] jfw: I never used it myself so can't speak to whether it works or works well.
[00:58] jfw: strikes me as a failure of IMAP that they couldn't extend the protocol to support that need and instead cooked up some whole new 'managesieve' protocol, but what do I know.
Day changed to 2022-10-17
[20:12] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5149 -- re-reading, at this point you were at 3 and reported 9 to go. does that mean there are 12 total and I was mistaken ? or is 9 the total ?
[20:12] sourcerer: 2022-10-15 22:19:51 (#jwrd) jfw: looks like there's somewhere around nine of these still to go.
[20:12] sourcerer: 2022-10-15 23:30:31 (#jwrd) dorion: ok. so you're 5/9ths through analysis, at least by count.
[20:58] jfw: dorion: indeed my words were 9 to go.
[21:01] jfw: so, 5/12ths.
Day changed to 2022-10-20
[05:50] jfw: http://fixpoint.welshcomputing.com/2022/busybox-microcom-the-code-review/
[12:07] dorion: jfw, nice, well done.
Day changed to 2022-10-22
[02:58] jfw: here's the earlier paste promoted to permanent link.
[02:58] sourcerer: 2022-10-13 23:24:34 (#jwrd) jfw: for a first patch of solid ground to stand on, a file-level map of what calls dovecot's modules API and what executables each one ultimately gets linked into http://welshcomputing.com/paste/myiuuudunx
[04:00] jfw: our sixth instance of module loading, in the quest to find what modules are loaded by what components, is login-common/main.c; it's yet another special thing in that its module path and module list are configurable, but the default is a "login" subdir of the main modules dir; and I haven't found any modules installing themselves there.
[04:01] jfw: so, similar to the config-parser: possibly just used by out-of-tree plugins.
[04:22] jfw: seventh, old-stats/main.c; it loads from an "old-stats" module subdir; there are two such modules in the tree, and the disorder continues in that one of them is called a plugin and the other not but besides that they seem shaped exactly the same.
[04:25] jfw: these are libold_stats_mail and libstats_auth (because why be consistent even with the "old" renaming?)
[04:30] jfw: there's also an "imap-old-stats" plugin but this would appear NOT to be loaded by the old-stats binary; what does load it I don't yet know.
[04:38] dorion: http://fixpoint.welshcomputing.com/2022/jwrd-logs-for-Oct-2022/#5183 -- smart.
[04:38] sourcerer: 2022-10-22 02:58:42 (#jwrd) jfw: here's the earlier paste promoted to permanent link.
[15:08] jfw: in demented and undocumented interface design, module_dir_load and family seem to load all modules in a given directory if a null pointer is passed as the module list string, but none of them if an empty string is passed.
[15:30] jfw: eighth up: doveadm/doveadm-mail.c, and now we finally get a bit more interesting. this links into doveadm which is a user-facing command, and it loads the configured "mail_plugins" from the configured "mail_plugin_dir", which defaults to the base module dir. These "mail plugins" seem to be the most numerous category.
[15:34] jfw: FTS is included there, along with any module/plugin not specifying a specific subdir.
[15:36] jfw: ninth is doveadm/doveadm-pw.c, which is another point where all present auth modules are loaded.
[15:36] sourcerer: 2022-10-15 21:46:44 (#jwrd) jfw: anyways, continuing from filesystem drivers, the second use of the modules mechanism is for authentication related code, and in this category there are five possible modules. The only one styled as a 'plugin' is var-expand-crypt noted just above. The rest are... not exactly 'builtin' but included directly in src/auth/, call
[15:37] jfw: tenth is doveadm/doveadm-util.c which loads all available doveadm modules ('doveadm' subdir).
[15:44] jfw: Further accentuating the lack of correspondence between plugins in the sense of subdirs of 'plugins' in the source tree, and the things that actually get loaded, some such 'plugins' include multiple modules in different categories, for instance a mail plugin and a doveadm plugin.
[15:45] jfw: "doveadm which is a user-facing command" - to clarify, that's user-facing as opposed to internally invoked; administrator-facing would be more precise.
[15:53] jfw: now as to the doveadm modules, the ones I'm seeing are all "true" plugins i.e. come from 'plugins' subdirs, namely: acl, quota, fts-lucene (if BUILD_LUCENE), fts, and mail-crypt.
[15:53] sourcerer: 2022-10-14 03:40:51 (#jwrd) jfw: the mail-crypt plugin, fwiw; the security model isn't clearly explained but looks like it can derive message encryption keys from the user's password.
[15:58] jfw: I guess this isn't entirely accurate, since we've already seen the implicit category of filesystem drivers defined by the libfs_ prefix rather than by module subdir.
[15:58] sourcerer: 2022-10-22 15:34:23 (#jwrd) jfw: FTS is included there, along with any module/plugin not specifying a specific subdir.
[15:58] sourcerer: 2022-10-14 03:21:38 (#jwrd) jfw: for the first sample, looking at the pluggable filesystem drivers thing (lib-fs/fs-api.c): modules are loaded based on the filename pattern libfs_<driver>, where <driver> comes from the caller (perhaps ultimatly from a config file); but several of the drivers are builtin, so module loading only happens for plugin-supplied drivers; the in-tree plugins in this category are fs-compress and
[16:27] jfw: eleventh, lib-storage/mail-user.c; this is another possibly more interesting case as, per the map, it's the first one we've seen so far that gets linked into most of the binaries (such as 'imap'), by way of libstorage -> libdovecot-storage.
[16:29] jfw: it loads a specified module from the configured mail_plugin_dir; I haven't got to the bottom of where the module name comes from, but it seems to be considered a "storage driver".
[16:37] jfw: this seems to refer to the mailbox formats which used to be configurable and I suppose were implemented as modules, but are now all builtin to libstorage (not a module).
[16:50] jfw: twelfth, lmtp/lmtp-client.c, similar to #8, loads the configured mail_plugins from the configured mail_plugin_dir; though there seems to be some 'lmtp' context to the settings ie the plugins to load might be configurable independently from that doveadm case or others. this gets linked only into the 'lmtp' exe.
[16:50] sourcerer: 2022-10-22 15:30:38 (#jwrd) jfw: eighth up: doveadm/doveadm-mail.c, and now we finally get a bit more interesting. this links into doveadm which is a user-facing command, and it loads the configured "mail_plugins" from the configured "mail_plugin_dir", which defaults to the base module dir. These "mail plugins" seem to be the most numerous category.
[17:21] jfw: grr, GNUTLS support is broken currently... but does that spare me from slogging through all the overhead of the pretense of interchangeable backends? nooooo
[17:25] jfw: especially weirdly, there's a lib-dcrypt/dcrypt-gnutls.c in git which disappeared in the release tarball without a trace except for a stray comment suggesting dcryp tcould be loaded with either openssl or gnutls.
[17:32] jfw: which brings us to the thirteenth: lib-dcrypt/dcrypt.c, which loads libdcrypt_openssl, which is built if BUILD_DCRYPT_OPENSSL. this is another that's linked into ~everything by way of libdovecot, but the things that actually attempt to initialize it are few: mail-crypt and var-expand-crypt (plugins), passdb_oauth2 (builtin), and some doveadm dumping commands.
[17:33] jfw: libdcrypt_openssl is installed to the top-level module dir, which they've now spelled as "pkglibdir" instead of "moduledir" so as to maximally burden the reader with automake documentation digs.
[17:40] jfw: I've been getting hints that there's a depenendency hell between certain modules, ie order of loading/initialization can matter
[17:54] jfw: fourteenth and finally is lib-storage/mail-storage-service.c ; again similar to #8 it loads mail_plugins from mail_plugin_dir; this time settings context (for lack of a better term) is mail user settings.
[17:54] sourcerer: 2022-10-22 15:30:38 (#jwrd) jfw: eighth up: doveadm/doveadm-mail.c, and now we finally get a bit more interesting. this links into doveadm which is a user-facing command, and it loads the configured "mail_plugins" from the configured "mail_plugin_dir", which defaults to the base module dir. These "mail plugins" seem to be the most numerous category.
[17:59] jfw: like #11 it's linked into many executables by way of libdovecot, and since #11 looked like a vestige, I suppose by process of elimination that this is where the chosen mail plugins get loaded by most executables that do such loading: a main entrance of sorts.
[17:59] sourcerer: 2022-10-22 16:27:16 (#jwrd) jfw: eleventh, lib-storage/mail-user.c; this is another possibly more interesting case as, per the map, it's the first one we've seen so far that gets linked into most of the binaries (such as 'imap'), by way of libstorage -> libdovecot-storage.
[18:08] jfw: a further confirmatory hint of this is that the directly upstack functions mail_storage_service_lookup and mail_storage_service_lookup_next are called from many places including doveadm, imap, indexer, lda, lmtp, pop3, submission, script-login, plugins/mail-crypt and plugins/quota.
[18:11] jfw: the gap between my guess of 12 and the final 14 cases most likely comes from the three doveadm files turning out to be distinct cases.
[18:13] dorion: jfw, quite the swamp. my first impression is it'll probably be more efficient to write the generic replacement for the modules functions that works statically, i.e. the dynamic dispatch table route, vs figuring out one by one which we need and replacing them with the static calls.
[18:13] dorion: what's your outlook ?
[18:15] jfw: dorion: it's not clear to me yet
[18:15] dorion: ok
[18:18] jfw: one question would be, they use this mail_plugins setting to control at runtime what plugins get loaded, and this might be significant i.e. a plugin changes the behavior of something just for its having been loaded. so, do we want to preserve that configurability, or make it all a compile-time decision?
[18:19] jfw: I'm inclined to throw maximum piss and bile at the whole plugin concept, go with the latter and make it truly static as much as feasible
[18:20] jfw: from the whole excavation, what modules came up as having any business being configurable in the first place? not much, it seems to me
[18:22] jfw: supposing some day we want to add quota or acl support or whatever, would do it by tweaking the gport.
[18:23] jfw: changing FTS backend
[18:25] dorion: I tend to agree, but would like to know your thoughts on easy of implementation for each.
[18:26] dorion: ease*
[18:35] jfw: that's what I'm having trouble predicting.
[18:37] jfw: might just need to give it a try, unless there's some more informative causes to act from.
[18:40] dorion: yeah, I can see it being a tough prediction.
[18:41] dorion: alright, you got the green light from me to piss and puke away, with the aim of cleaning up, lolz
[18:41] jfw: there's also still the "do nothing for now" option; now we have a better picture of all the modules that will be nonfunctional and it doesn't look too bad. fts and seive still seem possibly important, but search should work in any case, just slower
[18:44] dorion: hmm, right.
[18:45] dorion: so move onto packaging for now and then deployment and testing ?
[18:51] jfw: the build system is still ugly as sin; it builds & installs all the modules even though they're not usable. it does work though.
[18:52] dorion: well, any way we go is ugly.
[18:55] dorion: if we take the s/do nothing/deploy as is and test/ approach now and later decide to go back to pissing on plugin concept, how difficult do you think it'll be for you to pick back up ? the log and status reports are solid ? do you have other useful notes to help you load it back into head ?
[18:56] jfw: I should at least give it a try on the static modules route, while it's all fresh, I'm thinking.
[18:56] dorion: that second ? was supposed to be a ,
[18:57] jfw: the content is good, and there are some other notes from the dig; still there will be some overhead to resuming
[18:57] dorion: jfw, ok. you've spent a couple weeks w/ the code to get where you're at, any amount of notes still aren't going to be as good as where you're at now.
[18:59] dorion: keep w/ the updates as you see fit here so as to come up for air.
[19:00] dorion: I'm going to step away a few hours, be back later tonight.
[19:00] jfw: and static modules and a clean build system would be the best payoff for the work invested
[19:00] dorion: right.
[21:47] jfw: http://fixpoint.welshcomputing.com/2022/sounding-the-swamps-of-internet-mail-access/
Day changed to 2022-10-23
[04:28] jfw: and I've got the obligatory 600-word introduction down for an article structuring all those findings on the different cases of module loading and types of modules.
[04:29] jfw: my sympathies to the October log readers.
Day changed to 2022-10-26
[23:03] jfw: I'm realizing one lovely bit of self-contamination that's adding to the general confusion in dovecot is that they install internal shared libraries that aren't modules, that is, things that just link directly into various binaries rather than being dlopen-ed, right in the same directory as the modules.
[23:04] jfw: the build system search term for these seems to be pkglib_LTLIBRARIES, and the names start with libdovecot_ .
[23:06] jfw: actually just 'libdovecot'. there's the main libdovecot.la, then libdovecot-x.la for x in: compression, dsync, fts, lda, login, sql, storage.
[23:07] jfw: add ldap, lua, storage-lua if the respective deps are enabled.
[23:56] jfw: http://fixpoint.welshcomputing.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/
Day changed to 2022-10-27
[00:12] jfw: that one was like passing a kidney stone (he says, in blissful ignorance of what that actually feels like) - trying to build some kind of order from a very disorderly pile. hopefully the reading goes down smoother.
[14:47] dorion: jfw, http://fixpoint.welshcomputing.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2233
[17:16] jfw: dorion, replied.
Day changed to 2022-10-28
[02:13] jfw: + commented re SQL drivers mystery solved.
Day changed to 2022-10-29
[04:10] jfw: http://welshcomputing.com/paste/vy3vw52fy5 - preview of what's cooking for static dovecot module support.
[04:13] jfw: http://welshcomputing.com/paste/csg3rfi9m5 - corrected (missed passing a new parameter)
[16:44] dorion: jfw, looks slick.
[19:58] jfw: tenatively done with the conversion for load points 8, 10, 12 and 14 (and didn't even have to kill lmtp)
[19:58] jfw: *tentatively
[20:01] jfw: will need to either revive my previous module_dir_try_load_missing stub or follow through on removing the condemned load points for them to be able to link.
[20:02] jfw: then I'll need to look at the build system and decide whether it's time to flush the autoshit or continue with incremental changes to get the new stuff working.