In honor of our new botly overlords, Fixpoint is presenting its first AI-generated article.
Just kidding, of course! The intelligence as usual is the slow-simmered, old-fashioned human kind, though I did write a script to put the computer to work for what it's good at: data processing and formatting grunt work, as this turns out to be my lengthiest published patch series by far, at least by count.(i)
Our subject as promised is the initial release of JWRD Dovecot, a full-featured, efficiently and statically compiled
The civilized world starts here, you see, and the time to become a somebody is now. Before it's too late, because if you haven't noticed under all the new paint jobs, it quite seems to me that things are continuing to fall apart, all around, day by day.
Anyway, I wrangled the beast in the Git-bound state in which I found it to get to this publishable item in Unix tar format. It's now a simple snapshot of the repository in its preferred form for editing, rather than some kind of second-class generated artifact boiled down and frothed up for luser consumption, as seen in prior practice promulgated by GNU Autobots Autotools. I've chosen to show my work here in plain patch format for the (pre)historical record, but the release will be the only signed item, as it's the first that I'm recommending or indeed even able to use at all.
Download: jwrd-dovecot-2.4.0.tar.gz (4.3 MB), signature.
Table of Patches
In summary, there's around:
- 220 lines of truly new code added ;
- 2`200 lines of new build system (the length of which mostly just reflects the number of files listed in its various file list declarations) ;
- 70`000 lines of Unicode database and kinda-documentation imported, previously auto-downloaded at release time (the better to pretend they don't count, I guess) ;
- 930 lines of previously auto-detected configuration knobs imported, meanwhile reduced to 680 ; and
- 36`000 lines of well-swilled mouthwash sent down the drain, with reasonable confidence of no truly valued teeth or gold fillings in the mix (though the possibility can't be entirely excluded, to be sure).
One pre-existing bugfix was applied, four pre-existing portability failures in the test suite were newly fixed, and three of my own blunders were found and fixed in the process (excluding, of course, the more numerous ones too short-lived to show up in a commit).
Before proceeding to individual patch summaries and commentary as captured in the commit messages, here for the sake of cross-medium bridging is the NEWS file entry for the release, depicting the same subject from yet another angle and level of detail. My interpretation of the file's convention for list item bullet symbols was roughly: - for bugs, + for new features and * for other sorts of changes.
v2.4.0 2022-11-29 Jacob Welsh < http://fixpoint.welshcomputing.com/ >
* First release under new management by JWRD Computing. Rebrand, removing support-email key from IMAP ID command. (Support channels are blog comment, jwrd.net contact form, or in-WoT chat only.)
* Remove dlopen-based plugin/module system, replacing the modules API with simplified, statically linkable code, based on build time fixed lists of the module entry points, one per module category.
* Drop unloved plugins with overly special loading mechanics (fs-compress, mail-crypt, var-expand-crypt, fts-lucene).
* auth, lib-dict-backend, lib-sql, master: Remove module mode for SQL drivers, Lua, GSSAPI and LDAP options so they're either builtin or absent.
* plugins/old-stats, doveadm: Remove stats based on unreliable /proc/self/io, which had brought in its own special undocumented "preinit" extension to the modules API.* Replace Autoconf-Automake build system with much simpler and faster GNU Makefiles and a hand-edited config.h. (Several following changes stem from a config.h audit and work to reduce the burdens of such editing.)
* lib: Improve mremap wrapper portability.
* Remove lib-dcrypt, lib-oauth2, and the features that used them: oauth2 userdb/passdb/SASL mechanism, "Google XOAUTH2 protocol", and doveadm dump subcommands dcrypt-key and dcrypt-file.
* lib: Drop unreliable detection of forward time jumps, which was used only for logging; preserve detection of the more problematic backward jumps.
* lib-imap, lib-mail, lib-storage, lib: Consistently reject negative time_t values (pre-1970) as underflow while tightening date/time parse error handling. May affect searches with invalid date parameter, by properly reporting the error. (Together, these two changes allow removal of the TIME_T_MAX_BITS magic number and resulting Y2038 complications.)
* Clarify or correct various config.h comments.
* Remove TLS Compression code, which used an API that was never standardized and is now long disabled for insecurity.
* Remove code for replacing libssl/libcrypto memory allocation routines, which is a no-op on LibreSSL and had function pointer types sneakily changed on OpenSSL.
* Prune unreferenced config.h macro definitions in bulk.
* Eliminate numerous other config.h macros, by:
Adjusting code to be more portable or consistent in the first place;
Disregarding recent OpenSSL API deprecation noisemaking while requiring a couple of its sufficiently mature functions;
Requiring va_copy (fitting with the overall C99 requirement);
Requiring timegm (non-POSIX but present in GNU, musl and BSDs).
* Initialize some type range and endian related config.h macros that couldn't be removed outright to values that should at least work for all musl/gcc platforms.
* Remove embedding of version control system hash/revision information into built programs.
* Close the remaining gap between VCS and released tarball by importing wiki docs and unicode data files previously downloaded at build or release time. There's no more pre-generating files at release time, so flex, bison and perl are now build requirements.* lib: De-obfuscate fallback and error cases in system RNG interface by separating getrandom from urandom code paths.
* imap-hibernate: Zero pointer to quiet gcc 4.9 warning about possible uninitialized use.
* lib: Remove all but the simplest of the CPU limit tests as they were dreadfully slow and resource intensive.
* lib: Reduce iteration counts on randomized base64 codec tests, as they were fairly slow (and haven't found any problems yet).
+ Enable full execution of the test suite rather than aborting after the first failed program, by adding src/run-tests.sh to handle the summary reporting logic (still invoked from "make check").
- lib-program-client: Trim test that breaks on no-ipv6 kernels and fix function name (net_connect_ip) in error message.
- lib-charset: Drop iconv tests based on UTF-7 source encoding, which isn't implemented in musl and didn't turn up in a search of my own 12 year email collection.
- lib: Fix file_cache_errors test on musl by not demanding specific error strings.
- imap/test-imap-client-hibernate: Use relative path for temp dir to bypass path length limit on unix domain socket addresses.- lib-imap: imap_bodystructure_parse_full(): Reset parts.data on failure to prevent caller confusion (ported from release-2.3 branch).
Fixes: Panic: file message-part-data.c: line 579 (message_part_is_attachment): assertion failed: (data != NULL)
01-always-getrandom.patch
(top - view patch)
m4, configure, lib: always use getrandom(), dropping alternatives and fallback complexity
This requires the kernel support the getrandom syscall. Why *try* to use it if you're not *going* to use it? The result was the worst of both worlds: more code to verify and none of the intended guarantees.
The --with-libbsd configure option disappears, as libbsd was only used for arc4random.
This also changes the internal random_read() function signature not to inline nor unnecessarily convert ssize_t to int.
02-backport-imap_bodystructure_parse-reset.patch
(top - view patch)
lib-imap: imap_bodystructure_parse*() - Reset parts.data on failure
Having the data filled only to some message_parts can confuse the callers,
thinking that all the parts were successfully filled.Fixes:
Panic: file message-part-data.c: line 579 (message_part_is_attachment): assertion failed: (data != NULL)Ported from release-2.3 branch, which diverged from release-2.3.19, by jfw.
03-static-module-infrastructure.patch
(top - view patch)
lib: begin adding true static module support
04-static-module-loading-1.patch
(top - view patch)
doveadm, lib-storage, lmtp: switch to static module loading
Covers 8, 10, 12 and 14 from http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/
05-drop-dynamic-module-infrastructure.patch
(top - view patch)
lib, m4, configure: drop dlopen based module API implementation
06-drop-problem-plugins.patch
(top - view patch)
plugins: drop unloved plugins with overly complex and special loading mechanics (fs-compress, mail-crypt, var-expand-crypt, fts-lucene)
Pursuant to http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/ , these would add unwarranted costs to the static module conversion effort.
07-drop-unused-load-points.patch
(top - view patch)
auth, dict, doveadm, lib-fs, lib-storage, login-common: remove now-unused module load points
Covers 1, 2 (the blanket auth plugin loader part), 4, 6, 9 and 11 from http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/
08-no-module-mode-for-lua-gssapi-ldap.patch
(top - view patch)
configure, m4, auth, lib-dict-backend, master: remove plugin/module mode for lua, gssapi and ldap options so they're either builtin or absent
Completes #4 and addresses comment #8 from http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/
09-no-module-mode-for-sql-drivers.patch
(top - view patch)
configure, m4, lib-sql, master: remove plugin/module mode for SQL drivers so they're either builtin or absent
As with the authdb and mech options.
10-drop-problem-disk-io-stats.patch
(top - view patch)
plugins/old-stats, doveadm: remove stats based on unreliable /proc/self/io
This had brought in its own special undocumented "preinit" extension to the modules API.
11-static-module-loading-2.patch
(top - view patch)
auth, config, lib-dcrypt, lib-ssl-iostream, lib, old-stats: switch to static module loading
Covers 2, 3, 5, 7 and 13 from http://jfxpt.com/2022/classifying-the-dovecot-module-menagerie-by-load-point/
12-prune-dead-dynamic-module-code.patch
(top - view patch)
general pruning of defunct dynamic module system tendrils
removed symbols include:
.login_plugin_dir.mail_plugin_dir.version_ignore.version_stringMASTER_DOVECOT_VERSION_ENVMODULEDIRMODULE_LIBSmaster_service_get_version_stringmaster_service_verify_version_stringmodule_dir_loadmodule_dir_load_missingmodule_get_symbolstruct module_dir_load_settings"moduledir" is still around in the build system.
13-import-config-header.patch
(top - view patch)
config.h: import generated file, preparing for autotools purge
14-prune-dead-config-defs.patch
(top - view patch)
config.h: drop definitions not referenced anywhere in the codebase
15-config-fixups-1.patch
(top - view patch)
fixups from partial config.h readthrough
Import clarifying comments from configure.ac and elsewhere.
CRYPT_USE_XPG6 isn't actually needed - the probe was only for non-breakage.
Remove options for pretend OpenSSL APIs:
ASN1_STRING_get0_dataECDSA_SIG_get0ECDSA_SIG_set0 (adding missing null pointer checks)EC_GROUP_order_bitsEVP_MD_CTX_newRemove option for lacking the well established OpenSSL API: RSA_generate_key_ex. The older RSA_generate_key is inferior because it demands the exponent as a machine integer rather than bignum.
Reformat hard-wrapped comment lines.
Remove broken mremap() feature test macro usage which exploited private glibc internals.
Update webpage link. (They can keep the bugreport email address since they love email so much.)
Add GCC-specific endian detection: best we can do short of implementing a runtime check.
Remove other unused definitions (HAVE_GNUTLS escaped the bulk pruning because it was referenced for the feature listing in master/main.c).
16-drop-dcrypt-and-oauth2-1.patch
(top - view patch)
[1/2] drop all files for dcrypt library and oauth2 which depended on it
17-drop-dcrypt-and-oauth2-2.patch
(top - view patch)
[2/2] cut linkage to dcrypt and oauth2 from remaining files.
18-time-magic-cleanup-1.patch
(top - view patch)
lib: cut two users of the magic TIME_T_MAX_BITS
1. Drop the unreliable and seemingly useless detection of forward time jumps, preserving detection of the more problematic backward jumps.
2. Drop gnarly fallback code for systems lacking timegm.
19-time-magic-cleanup-2.patch
(top - view patch)
lib-imap, lib-mail, lib-storage, lib: cut last uses of the magic TIME_T_MAX_BITS, tightening date/time parse error handling.
The core is that date/time decoding errors in imap_mktime, whether due to time_t overflow or just having a component out of range, are no longer clipped or projected to arbitrary pseudo-maximum values but set to (time_t)-1, which was already an error signal per utc_mktime/timegm. For consistency (and since time_t isn't even necessarily signed), any time before the epoch is now considered underflow.
The rest is adjustment of calling code (imap_mktime is consumed directly only by imap_parse_date/imap_parse_datetime), typically to make it notice the error.
Behavior of the IMAP APPEND command is unchanged: an invalid internal date argument will be ignored and the new message's internal date attribute set to the current time.
Other behavioral changes can't be ruled out - mainly, searches now returning errors for the invalid date parameter instead of proceeding to possibly bogus results.
20-config-fixups-2.patch
(top - view patch)
fixups from remainder of config.h readthrough
Reformat, clarify and correct more comments.
Remove options for pretend OpenSSL APIs:
auto thread deinit (OPENSSL_thread_stop)OPENSSL_cleanupSSL_CIPHER_get_kx_nidSSL_CTX_add0_chain_certSSL_CTX_set1_curves_listSSL_CTX_set_ciphersuites (TLS 1.3)SSL_CTX_set_current_certSSL_CTX_set_min_proto_version (protocol versions can still be selected the old way)Remove option for lacking well established OpenSSL APIs:
SSL_clear_options ( http://jfxpt.com/2022/jwrd-logs-for-Nov-2022/#5336 )SSL_get_servername, SSL_CTX_set_tlsext_servername_callback, SSL_set_tlsext_host_name (SNI)Define HAVE_OPENSSL_ERR_REMOVE_THREAD_STATE and HAVE_X509_CHECK_HOST, since we do have them, but keep them optional since they don't go back very far. Combine options for X509_check_host, X509_check_ip (unused) and X509_check_ip_asc into one, as they're a family introduced at the same time.
Remove TLS Compression code: the API was never standardized and is now long disabled for insecurity ( http://jfxpt.com/2022/jwrd-logs-for-Nov-2022/#5339 ).
Remove code for replacing libssl/libcrypto memory allocation routines ( http://jfxpt.com/2022/jwrd-logs-for-Nov-2022/#5359 ).
Garbage-collect now unused HAVE_TIMEGM and TIME_T_MAX_BITS.
Reduce the integer types insanity, so that one config.h can at least support all Gales platforms, by assuming musl specifics like 64-bit off_t and GCC intrinsics like __SIZEOF_INT__. (TODO: something different for PRIdTIME_T/PRIxTIME_T as current Gales musl doesn't in fact guarantee 64-bit time_t.)
Remove VA_COPY macro wrapping, requiring va_copy support (C99).
21-config-fixups-3.patch
(top - view patch)
replace all PRIdTIME_T and PRIxTIME_T uses by casting the argument to explicit 64-bit type
22-makefiles-and-build-fixes.patch
(top - view patch)
add plain GNU Make build system, with minor required code changes.
Eliminate the generated dovecot-version.h for VCS version string embedding.
Remove dubiously helpful runtime referral to the example config dir.
Fix duplicate global definition of doveadm_settings in test-doveadm-util.c by moving i_strccdascmp from doveadm-util to strfuncs.h and its own C file, reducing spurious link dependencies.
Write out previously generated files lib-dict-backend/dict-drivers-register.c and lib-sql/sql-drivers-register.c, using ifdefs on config.h macros to switch the optional components.
Define single TOP_SRC_DIR macro, replacing several used by test programs for finding data in the build tree: DICT_SRC_DIR, TEST_STOPWORDS_DIR, TEST_BIN_DIR, UDHRDIR.
Corrections to the new static module loading code:
- Include necessary headers in module-dir-load-*.c
- Fix function pointer type syntax in module_load_static signature
- Declare & document module_load_static and module_parse_names in module-dir.hUse explicit paths for unusual header includes to prevent excessive growth of -I flags (which are now global).
Fix duplicate global definitions of stats_metrics and stats_startup_time by moving them to a stats-common.c to mirror their declaration in stats-common.h.
http://jfxpt.com/2022/jwrd-logs-for-Nov-2022/#5421 for further rationale and anecdotes.
23-import-unicode-data.patch
(top - view patch)
lib, lib-fts: import unicode database files from 2.3.19.1 release tarball
Required to generate lib/unicodemap.c, lib-fts/word-boundary-data.c and lib-fts/word-break-data.c, thus there's no escaping that these are part of the source code.
24-makefile-for-docs.patch
(top - view patch)
add doc/Makefile, adjusting & correcting src/Makefile to harmonize
Avoid ".new" suffix for generated temporaries to reduce chance of conflict with humans.
25-import-wiki-docs.patch
(top - view patch)
doc: import wiki files from 2.3.19.1 release tarball
26-drop-autoconf-automake.patch
(top - view patch)
drop autoconf-automake build system
27-fix-make-clean.patch
(top - view patch)
src: 'make clean' needed to descend deeper for .o files
Editor's comment: not sure what's up with the getrandom stub; it looks like a temporary hack for a build error on a test system, which perhaps wasn't noticed earlier due to 'make clean' failing to cause a fully fresh build. In any case, it's redone in the following patches so it's not the entropy-faking disaster that it appears to be!
28-revert-always-getrandom.patch
(top - view patch)
Revert "m4, configure, lib: always use getrandom(), dropping alternatives and fallback complexity"
This reverts commit f51020888d8d5d2d6f776b91439f43bbe3fcde28 (though config.h needs updating as the build system was meanwhile rewritten).
While it would be nice to just use getrandom, it's Linux specific and too new to be counted on.
29-untangle-getrandom-urandom.patch
(top - view patch)
lib: untangle the getrandom vs. urandom code paths and add required config.h settings
Separate the two methods and their error handling, straightening out the error cases, whether in normal or fallback use. Remove unnecessary extra macros and second definition of getrandom_present. Use consistent return type on random_read, removing a spurious integer truncation.
30-report-test-suite-stats.patch
(top - view patch)
offload main test suite loop from Makefile to proper script, to implement aggregate failure reporting
31-fix-program-client-tests.patch
(top - view patch)
lib-program-client: trim test that breaks on no-ipv6 kernels; correct function name in error message
32-drop-broken-utf7-tests.patch
(top - view patch)
lib-charset: drop iconv tests with utf7 source encoding
It's not implemented in musl, and seems unnecessary since this code is used for message decoding & search while utf7 bodies or headers are practically never seen in the wild.
Further, the test code is broken because it proceeds to null pointer dereference on "trans" after the initialization failure.
33-lighten-cpu-spin-tests.patch
(top - view patch)
lib: cut down on the more aggressive cpu limit testing and unsolicited fuzz testing because it's too damn slow
34-fix-unportable-oom-tests.patch
(top - view patch)
lib: don't demand specific error strings in file_cache_errors test
"Cannot allocate memory" is spelled "Out of memory" on musl.
35-version-2.4.0-candidate.patch
(top - view patch)
Update NEWS and version strings for v2.4.0 release, rebranding to JWRD Dovecot.
- Eliminate DOVECOT_VERSION by standardizing on PACKAGE_VERSION.
- Drop now unused PACKAGE_TARNAME and VERSION.
- Drop PACKAGE_BUGREPORT and with it the support-email key in IMAP ID command. (Support channels are blog comment, jwrd.net contact form, or in-WoT chat only.)
- Bump minor version to reflect the deep rework of module and build systems and removal of some features.
36-warnings.patch
(top - view patch)
Fix awkward or hazardous situations found by compiler warnings
- doveadm/doveadm-oldstats.c(cmd_stats_top): unused variable (missed pruning in 9a28a60d61b6e6f6e992cefdc0ee056b721d0c8b)
- imap-hibernate/imap-client.c(imap_client_input_idle_cmd): new_tag is supposed to be initialized by pointer output parameter but it's not obvious that it always is, so zero-initialize it to be on the safe side.
- lib-http/test-http-client-errors.c: include signal.h by standard path (it's not under sys)
- lib-storage/mail-storage-service.c(mail_storage_service_load_modules): control reaches end of non-void function (broken by 088279cc44ec62ae2befb0ce1c0ac9900f2f444a but return value was never used)
- lib/compat.h: change the "unsigned long long" uoff_t case to the explicit uint64_t, for full agreement with its printf specifier PRIu64 and limit UINT64_MAX.Not addressed:
lib-smtp/test-smtp-params.c:30:2: missing initializer for field 'used' of 'struct <anonymous>' [-Wmissing-field-initializers]
lib-smtp/test-smtp-params.c:34:2: missing initializer for field 'used' of 'struct <anonymous>' [-Wmissing-field-initializers]
- False positives from gcc 4.9.Various: passing argument 4 of 'module_load_static' from incompatible pointer type
- Module init functions not conforming to API, previously loaded by dlsym without type checking. Shouldn't be a problem since they simply don't use the supplied parameters.Various unused variables and parameters, especially in perl-generated code.
Various: comparison is always false due to limited range of data type [-Wtype-limits]
- Seems to be an artifact of their COMPILE_ERROR_IF_TRUE implementation.
37-fix-ssl-free-build.patch
(top - view patch)
lib: conditionalize module_dir_load_ssl_iostream based on HAVE_SSL and HAVE_OPENSSL
This snuck by before, presumably because of other conditionals causing the function not to get called, but it failed to compile on a system without openssl/ssl.h.
38-fix-hibernate-test-temp-dir.patch
(top - view patch)
imap/test-imap-client-hibernate: use relative rather than absolute paths for temp dir
In one setting, the full path ran afoul of the path length limit on unix domain socket addresses (108 characters on Linux), and it only seems to complicate the code anyway.
While we're here, stop dot-prefixing the temp dir as there's no good reason to hide it.
39-version-2.4.0-final.patch
(top - view patch)
NEWS: tone down language about new_tag pointer which on closer inspection was not used uninitialized.
- Besides writing, testing and refining the script, much of the "writing" time here went into patch file naming and categorizing, the changes themselves and commit messages being done already. [^]
- As necessary, mind; that is, not just because "wouldn't it be nice if it made coffee and did the dishes too?" as if we didn't have enough work to go around already. [^]
[...] things? Or I dunno, maybe it's not even a secret strictly speaking, but could just as well be since nobody reads code anyway and there's no way to verify it's built from the advertised sources. Which nobody can do anymore [...]
Pingback by That frog prince story as seen from the home castle « Fixpoint — 2023-07-21 @ 21:31
For a first of those necessary refinements, here's a patch I'm currently running which improves user experience by removing spurious slowdowns on first and/or failed login, and by the same stroke removes what might assist an unauthenticated attacker to tie up server resources at minimal cost to himself.
As reported in the #jwrd logs.
I plan to include this or something like it in a 2.4.1 release.
Comment by Jacob Welsh — 2023-09-09 @ 21:54
A second patch fixes parallel 'make' when invoked from the top-level project directory.
Comment by Jacob Welsh — 2023-09-28 @ 15:49
[...] the sake of showing and explaining the work, I've again formatted my git commits as patch files.(ii) They are given as I made them, with mistakes and later [...]
Pingback by Regrinding Busybox archive extraction, fixing directory timestamps, symlink attacks, a buffer overflow and more « Fixpoint — 2024-05-04 @ 03:34