<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	>
<channel>
	<title>Comments on: Classifying the Dovecot module menagerie by load point</title>
	<atom:link href="http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/feed/" rel="self" type="application/rss+xml" />
	<link>http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/</link>
	<description>The search for invariants</description>
	<pubDate>Sat, 18 Apr 2026 09:52:17 +0000</pubDate>
	<generator>http://polimedia.us</generator>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
		<item>
		<title>By: JWRD Dovecot initial release, aka version 2.4.0 &#171; Fixpoint</title>
		<link>http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2409</link>
		<dc:creator>JWRD Dovecot initial release, aka version 2.4.0 &#171; Fixpoint</dc:creator>
		<pubDate>Mon, 26 Jun 2023 04:40:16 +0000</pubDate>
		<guid isPermaLink="false">http://fixpoint.welshcomputing.com/?p=198#comment-2409</guid>
		<description>[...] Covers 8, 10, 12 and 14 from http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/ [...]</description>
		<content:encoded><![CDATA[<p>[...] Covers 8, 10, 12 and 14 from <a href="http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/" rel="nofollow">http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/</a> [...]</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Grand reopening of #jwrd, the IRC channel &#171; Fixpoint</title>
		<link>http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2386</link>
		<dc:creator>Grand reopening of #jwrd, the IRC channel &#171; Fixpoint</dc:creator>
		<pubDate>Sat, 13 May 2023 00:32:16 +0000</pubDate>
		<guid isPermaLink="false">http://fixpoint.welshcomputing.com/?p=198#comment-2386</guid>
		<description>[...] got it to pass its config test and start up after some adjustments. it does have this suspiciously dovecot-like 'modules' setup where it expects to dynamically load, as yet unclear if the modules will thus work at all or [...]</description>
		<content:encoded><![CDATA[<p>[...] got it to pass its config test and start up after some adjustments. it does have this suspiciously dovecot-like 'modules' setup where it expects to dynamically load, as yet unclear if the modules will thus work at all or [...]</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: The Dovecot reports: how we came to forking a major email server &#171; Fixpoint</title>
		<link>http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2362</link>
		<dc:creator>The Dovecot reports: how we came to forking a major email server &#171; Fixpoint</dc:creator>
		<pubDate>Thu, 06 Apr 2023 23:59:09 +0000</pubDate>
		<guid isPermaLink="false">http://fixpoint.welshcomputing.com/?p=198#comment-2362</guid>
		<description>[...] progressed in porting Dovecot last week. Jacob's first step was producing an article to clearly structure the findings from the prior work. With this firm ground to stand on, we [...]</description>
		<content:encoded><![CDATA[<p>[...] progressed in porting Dovecot last week. Jacob's first step was producing an article to clearly structure the findings from the prior work. With this firm ground to stand on, we [...]</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Jacob Welsh</title>
		<link>http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2245</link>
		<dc:creator>Jacob Welsh</dc:creator>
		<pubDate>Wed, 02 Nov 2022 04:56:11 +0000</pubDate>
		<guid isPermaLink="false">http://fixpoint.welshcomputing.com/?p=198#comment-2245</guid>
		<description>Continuing re #2,

&lt;blockquote&gt;gssapi, ldap and lua modules may be initialized specially via auth/passdb.c (passdbs_init).&lt;/blockquote&gt;

This is approximately correct (to be precise, gssapi is done in auth/mech.c (mech_init)). What this means is that those modules (libauthdb_ldap, libauthdb_lua and libmech_gssapi) are already set up to work builtin, much like libdict_ldap in #4. Thus I'm thinking they can be left in. This leaves libauthdb_imap, which is built unconditionally and has no builtin variant, so I might go ahead and remove it along with the load point; it looks fairly self-contained so this should be easy. Then again, it could still be sustained using the new static module structure.

Re #9,

&lt;blockquote&gt;It would seem that auth modules can add functionality both to the "auth" daemon process (handling authentication requests) and to doveadm (perhaps configuring users &#38; passwords).&lt;/blockquote&gt;

Nonetheless I can't find how doveadm would be at all affected by auth modules, at least the authdb and mech ones on the table. Thus I'd go ahead and remove the load point but leave the auth modules alone except as covered in #2.

5, 7 and 13 are threatening to turn into further time sinks. Besides that, the removals are going smoothly enough.</description>
		<content:encoded><![CDATA[<p>Continuing re #2,</p>
<blockquote><p>gssapi, ldap and lua modules may be initialized specially via auth/passdb.c (passdbs_init).</p></blockquote>
<p>This is approximately correct (to be precise, gssapi is done in auth/mech.c (mech_init)). What this means is that those modules (libauthdb_ldap, libauthdb_lua and libmech_gssapi) are already set up to work builtin, much like libdict_ldap in #4. Thus I'm thinking they can be left in. This leaves libauthdb_imap, which is built unconditionally and has no builtin variant, so I might go ahead and remove it along with the load point; it looks fairly self-contained so this should be easy. Then again, it could still be sustained using the new static module structure.</p>
<p>Re #9,</p>
<blockquote><p>It would seem that auth modules can add functionality both to the "auth" daemon process (handling authentication requests) and to doveadm (perhaps configuring users &amp; passwords).</p></blockquote>
<p>Nonetheless I can't find how doveadm would be at all affected by auth modules, at least the authdb and mech ones on the table. Thus I'd go ahead and remove the load point but leave the auth modules alone except as covered in #2.</p>
<p>5, 7 and 13 are threatening to turn into further time sinks. Besides that, the removals are going smoothly enough.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Jacob Welsh</title>
		<link>http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2239</link>
		<dc:creator>Jacob Welsh</dc:creator>
		<pubDate>Sat, 29 Oct 2022 05:04:59 +0000</pubDate>
		<guid isPermaLink="false">http://fixpoint.welshcomputing.com/?p=198#comment-2239</guid>
		<description>Re #2,

&lt;blockquote&gt;There's another load point at auth_module_load, called by passdb_preinit, userdb_preinit and mech_register_init, which seems entirely redundant with this one, loading specific modules that would be already covered by the blanket loading here.&lt;/blockquote&gt;

what's up there is that the blanket load point is using a filter function passed through a settings struct to exclude any module names starting with authdb_ or mech_ (that is, all of the listed non-plugin modules), stating that those are "lazily loaded" i.e. by the mentioned auth_module_load.

I'd say then that the case is even clearer for removing the blanket load point which in fact covers only the var-expand-crypt plugin, while gssapi/ldap/imap auth options could perhaps still be kept with minimal extra effort given the way things are shaping up anyway.</description>
		<content:encoded><![CDATA[<p>Re #2,</p>
<blockquote><p>There's another load point at auth_module_load, called by passdb_preinit, userdb_preinit and mech_register_init, which seems entirely redundant with this one, loading specific modules that would be already covered by the blanket loading here.</p></blockquote>
<p>what's up there is that the blanket load point is using a filter function passed through a settings struct to exclude any module names starting with authdb_ or mech_ (that is, all of the listed non-plugin modules), stating that those are "lazily loaded" i.e. by the mentioned auth_module_load.</p>
<p>I'd say then that the case is even clearer for removing the blanket load point which in fact covers only the var-expand-crypt plugin, while gssapi/ldap/imap auth options could perhaps still be kept with minimal extra effort given the way things are shaping up anyway.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Jacob Welsh</title>
		<link>http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2238</link>
		<dc:creator>Jacob Welsh</dc:creator>
		<pubDate>Sat, 29 Oct 2022 00:35:09 +0000</pubDate>
		<guid isPermaLink="false">http://fixpoint.welshcomputing.com/?p=198#comment-2238</guid>
		<description>Re #12,

&lt;blockquote&gt;Can one module extend both mail (imap etc) and lmtp processes, or are these properly regarded as distinct module types?&lt;/blockquote&gt;

doc/example-config/conf.d/20-lmtp.conf illustrates a mail_plugins setting within a "protocol lmtp" block, describing it as a "space separated list of plugins to load (default is global mail_plugins)." This strongly suggests that they're the same type of module, just that you can load different ones for the different protocols.</description>
		<content:encoded><![CDATA[<p>Re #12,</p>
<blockquote><p>Can one module extend both mail (imap etc) and lmtp processes, or are these properly regarded as distinct module types?</p></blockquote>
<p>doc/example-config/conf.d/20-lmtp.conf illustrates a mail_plugins setting within a "protocol lmtp" block, describing it as a "space separated list of plugins to load (default is global mail_plugins)." This strongly suggests that they're the same type of module, just that you can load different ones for the different protocols.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Jacob Welsh</title>
		<link>http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2237</link>
		<dc:creator>Jacob Welsh</dc:creator>
		<pubDate>Fri, 28 Oct 2022 23:25:41 +0000</pubDate>
		<guid isPermaLink="false">http://fixpoint.welshcomputing.com/?p=198#comment-2237</guid>
		<description>Indeed from the git log it looks like that preinit hook was specifically added for the old-stats plugin, likely for permissions reasons (privileges being dropped at a later stage so /proc/self/io is no longer readable).</description>
		<content:encoded><![CDATA[<p>Indeed from the git log it looks like that preinit hook was specifically added for the old-stats plugin, likely for permissions reasons (privileges being dropped at a later stage so /proc/self/io is no longer readable).</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Jacob Welsh</title>
		<link>http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2236</link>
		<dc:creator>Jacob Welsh</dc:creator>
		<pubDate>Fri, 28 Oct 2022 23:17:24 +0000</pubDate>
		<guid isPermaLink="false">http://fixpoint.welshcomputing.com/?p=198#comment-2236</guid>
		<description>It's looking like some degree of "static modules API" implementation is going to be required in any case; the &lt;a href="http://jfxpt.com/2022/jwrd-logs-for-Oct-2022/#5225" rel="nofollow"&gt;runtime configurability&lt;/a&gt; of which plugins to load is compounded by the division (entirely spurious IMHO) between loading and initialization hinted at in footnote iv. Case in point, load point #8 stores the loaded module object array at a global "mail_storage_service_modules" pointer; initialization of those modules then is nowhere to be found in the doveadm subtree but possibly done at some other time by lib-storage/mail-storage-service.c. Module objects track their initialization status so that it's done only once, meaning the code may be assuming it can safely call module_dir_init multiple times on the same modules, so it may be unsafe to replace those with direct calls to the specific module initialization functions.

The approach will likely be a hybrid, enumerating the specific modules at the load points as proposed, thereby getting them linked into the necessary binaries, perhaps using a MODULE_LOAD_STATIC macro of some sort to specify the name and init/deinit routines, or a virtual method table structure to define them within each module itself. Then the remaining module object methods would continue to be used as-is.

Yet another complication (undocumented) is that a module can also have a "preinit" routine which if present is called at load time. The old-stats plugin (lib90_old_stats_plugin) has the only instance of it among the plugins; unsure about non-plugin modules so far. Its use there is to open /proc/self/io, for later gathering counts of read/write syscalls and total bytes. I hadn't encountered this interface before so looked it up in proc(5), which reports its origin as Linux kernel 2.6.20 and notes:

&lt;blockquote&gt;In the current implementation, things are a bit racy on 32-bit systems: if process A reads process B's /proc/[pid]/io while process B is updating one of these 64-bit counters, process A could see an intermediate result.&lt;/blockquote&gt;

IOW, it can sometimes report nonsense values. Needless to say, I'm unimpressed and would sooner take it out (which looks quite simple) than implement the preinit mechanism just for this.</description>
		<content:encoded><![CDATA[<p>It's looking like some degree of "static modules API" implementation is going to be required in any case; the <a href="http://jfxpt.com/2022/jwrd-logs-for-Oct-2022/#5225" rel="nofollow">runtime configurability</a> of which plugins to load is compounded by the division (entirely spurious IMHO) between loading and initialization hinted at in footnote iv. Case in point, load point #8 stores the loaded module object array at a global "mail_storage_service_modules" pointer; initialization of those modules then is nowhere to be found in the doveadm subtree but possibly done at some other time by lib-storage/mail-storage-service.c. Module objects track their initialization status so that it's done only once, meaning the code may be assuming it can safely call module_dir_init multiple times on the same modules, so it may be unsafe to replace those with direct calls to the specific module initialization functions.</p>
<p>The approach will likely be a hybrid, enumerating the specific modules at the load points as proposed, thereby getting them linked into the necessary binaries, perhaps using a MODULE_LOAD_STATIC macro of some sort to specify the name and init/deinit routines, or a virtual method table structure to define them within each module itself. Then the remaining module object methods would continue to be used as-is.</p>
<p>Yet another complication (undocumented) is that a module can also have a "preinit" routine which if present is called at load time. The old-stats plugin (lib90_old_stats_plugin) has the only instance of it among the plugins; unsure about non-plugin modules so far. Its use there is to open /proc/self/io, for later gathering counts of read/write syscalls and total bytes. I hadn't encountered this interface before so looked it up in proc(5), which reports its origin as Linux kernel 2.6.20 and notes:</p>
<blockquote><p>In the current implementation, things are a bit racy on 32-bit systems: if process A reads process B's /proc/[pid]/io while process B is updating one of these 64-bit counters, process A could see an intermediate result.</p></blockquote>
<p>IOW, it can sometimes report nonsense values. Needless to say, I'm unimpressed and would sooner take it out (which looks quite simple) than implement the preinit mechanism just for this.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Jacob Welsh</title>
		<link>http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2235</link>
		<dc:creator>Jacob Welsh</dc:creator>
		<pubDate>Fri, 28 Oct 2022 02:07:25 +0000</pubDate>
		<guid isPermaLink="false">http://fixpoint.welshcomputing.com/?p=198#comment-2235</guid>
		<description>SQL drivers mystery solved: while they're installed to the top-level moduledir, they're symlinked into both auth and dict subdirs, thereby falling under the catch-all load points 2, 4 and 9.

However, I do not think it would be accurate to classify them as either Authentication or Dictionary modules, because they do not by themselves implement any new auth method or dict backend. Rather, they implement an internal SQL API (as in lib-sql/sql-api.c et al, with its own virtual method dispatch table and all) which is used in turn by the builtin auth/db-sql.c on the auth side and ...apparently nothing on the dict side.

Thus, despite lacking a load point of their own I think it's fair to consider them a distinct module type: SQL Driver, and I stand by the proposed action: "like #4, to remove the plugin option so they're either builtin or absent".

The bleeding stanched from this hole in my model, I now believe the non-plugin modules have been fully listed, on the strength of a "grep 'module_.*LTLIBRARIES' -r src" not turning up anything new outside of plugins. I will update the article to strike out those cautiously listed as "unknown".</description>
		<content:encoded><![CDATA[<p>SQL drivers mystery solved: while they're installed to the top-level moduledir, they're symlinked into both auth and dict subdirs, thereby falling under the catch-all load points 2, 4 and 9.</p>
<p>However, I do not think it would be accurate to classify them as either Authentication or Dictionary modules, because they do not by themselves implement any new auth method or dict backend. Rather, they implement an internal SQL API (as in lib-sql/sql-api.c et al, with its own virtual method dispatch table and all) which is used in turn by the builtin auth/db-sql.c on the auth side and ...apparently nothing on the dict side.</p>
<p>Thus, despite lacking a load point of their own I think it's fair to consider them a distinct module type: SQL Driver, and I stand by the proposed action: "like #4, to remove the plugin option so they're either builtin or absent".</p>
<p>The bleeding stanched from this hole in my model, I now believe the non-plugin modules have been fully listed, on the strength of a "grep 'module_.*LTLIBRARIES' -r src" not turning up anything new outside of plugins. I will update the article to strike out those cautiously listed as "unknown".</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Jacob Welsh</title>
		<link>http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/#comment-2234</link>
		<dc:creator>Jacob Welsh</dc:creator>
		<pubDate>Thu, 27 Oct 2022 17:14:51 +0000</pubDate>
		<guid isPermaLink="false">http://fixpoint.welshcomputing.com/?p=198#comment-2234</guid>
		<description>You're welcome.

I don't think the removals will generally come up as hard necessities; it's more that the code becomes dead weight, slowing down any codebase-wide operations like searching (and compiling, though at least that's just a matter of machine time), as well as local operations like &lt;a href="http://jfxpt.com/2022/jwrd-logs-for-Oct-2022/#5210" rel="nofollow"&gt;reading&lt;/a&gt;. A compromise would be to remove just the condemned plugins/modules as proposed but leaving the load points, since it would be mostly a file or directory level removal, rather than removing code within a file and possibly cascading up to the callers.

I guess your argument is basically, we did the hard part of finding what can be safely killed and setting it all down, but so far it's not even worth the expenditure of ammunition to follow through. I'll demur on expressing an inclination for now, but can certainly start with the required parts like answering the SQL drivers question and taking the actions for 8, 10 and 14.</description>
		<content:encoded><![CDATA[<p>You're welcome.</p>
<p>I don't think the removals will generally come up as hard necessities; it's more that the code becomes dead weight, slowing down any codebase-wide operations like searching (and compiling, though at least that's just a matter of machine time), as well as local operations like <a href="http://jfxpt.com/2022/jwrd-logs-for-Oct-2022/#5210" rel="nofollow">reading</a>. A compromise would be to remove just the condemned plugins/modules as proposed but leaving the load points, since it would be mostly a file or directory level removal, rather than removing code within a file and possibly cascading up to the callers.</p>
<p>I guess your argument is basically, we did the hard part of finding what can be safely killed and setting it all down, but so far it's not even worth the expenditure of ammunition to follow through. I'll demur on expressing an inclination for now, but can certainly start with the required parts like answering the SQL drivers question and taking the actions for 8, 10 and 14.</p>
]]></content:encoded>
	</item>
</channel>
</rss>
