commit 2dc4b8cc5832b3c7f7b4c698b235e11876f5a585 Author: Jacob Welsh AuthorDate: Fri Nov 18 04:15:58 2022 +0000 Commit: Jacob Welsh CommitDate: Fri Nov 18 04:16:21 2022 +0000 Type: reference import doc: import wiki files from 2.3.19.1 release tarball diff --git a/.gitignore b/.gitignore index 45507bdf09..9695e8733c 100644 --- a/.gitignore +++ b/.gitignore @@ -103,7 +103,6 @@ doc/man/dovecot.1 doc/man/dovecot-sysreport.1 doc/example-config/README -doc/wiki/*.txt doc/wiki/Makefile.am src/anvil/anvil src/auth/checkpassword-reply diff --git a/doc/wiki/ACL.txt b/doc/wiki/ACL.txt new file mode 100644 index 0000000000..486d71cf6a --- /dev/null +++ b/doc/wiki/ACL.txt @@ -0,0 +1,258 @@ +Access Control Lists +==================== + +This page talks mainly about how ACLs work, for more general description of how +shared mailboxes work, see . + +Dovecot v1.0 and v1.1 supports administrator-configured ACL files. v1.2+ +supports also IMAP ACL extension, which allows users to change ACLs themselves. +The ACL code was written to allow multiple ACL backends, but currently Dovecot +supports only virtual ACL files. Note that using ACLs doesn't grant mail +processes any extra filesystem permissions that they already don't have. +[SharedMailboxes.Permissions.txt] to be able to access the mailboxes. When +testing you could first try accessing shared/public mailboxes without ACL +plugin even enabled. + +ACLs can be enabled in dovecot.conf with: + +---%<------------------------------------------------------------------------- +mail_plugins = acl +protocol imap { + mail_plugins = $mail_plugins imap_acl +} + +plugin { + # Without global ACLs: + acl = vfile + + # With global ACL files in /etc/dovecot/dovecot-acls file (v2.2.11+): + #acl = vfile:/etc/dovecot/dovecot-acl + + # With global ACLs in /etc/dovecot/acls/ directory (obsolete): + #acl = vfile:/etc/dovecot/acls + + # If enabled, don't try to find dovecot-acl files from mailbox directories. + # This reduces unnecessary disk I/O when only global ACLs are used. +(v2.2.31+) + #acl_globals_only = yes +} +---%<------------------------------------------------------------------------- + +ACL groups support works by returning a comma-separated 'acl_groups' [UserDatabase.ExtraFields.txt] from userdb, which contains all the +groups the user belongs to. User's UNIX groups have no effect on ACLs (you can +"enable" them by using a special [PostLoginScripting.txt]). + +The default ACL for mailboxes is to give the mailbox owner all permissions and +other users none. Mailboxes in public namespaces don't have owners, so by +default no one can access them. + +Master users +------------ + +Note that master users have their own ACLs. They're not the the mailbox owners, +so by default they have no permissions to any of the mailboxes. See +Authentication/MasterUsers#ACLs for more information. + +ACL vfile backend +----------------- + +vfile backend supports per-mailbox ACLs and global ACLs. + +Per-mailbox ACLs are stored in 'dovecot-acl' named file, which exists in: + + * maildir: The Maildir's mail directory (eg. '~/Maildir', + '~/Maildir/.folder/') + * mbox: Control directory. You should explicitly specify ':CONTROL=' in + mail location. + * dbox: dbox's mail directory (eg. '~/dbox/INBOX/dbox-Mails/') + +ACL Inheritance +--------------- + +Every time you create a new mailbox, it gets its ACLs from the parent mailbox. +If you're creating a root-level mailbox, it uses the namespace's default ACLs. +There is no actual inheritance, however: If you modify parent's ACLs, the +child's ACLs stay the same. There is currently no support for ACL inheritance. + + * Maildir: Namespace's default ACLs are read from "dovecot-acl" file in the + namespace's mail root directory (e.g.'/var/public/Maildir'). Note that + currently these default ACLs are used only when creating new mailboxes, they + aren't used for mailboxes without ACLs. + * v2.2.2+: If 'plugin { acl_defaults_from_inbox=yes } ', the default ACLs for + private and shared namespaces (but not public namespaces) are taken from the + INBOX. This means that giving somebody access to your INBOX will give them + access to all your other mailboxes as well, unless the specific mailboxes' + ACLs override the INBOX's. + +*NOTE*: Currently the default ACLs are merged with the mailbox-specific ACLs. +So if a default ACL gives access to "user1" and a per-mailbox ACL gives access +to "user2", the "user1" still has access to that mailbox. + +Global ACLs +----------- + +Global ACLs can be used to apply ACLs globally to all user's specific +mailboxes. They are used mainly for two purposes: + + 1. Removing some permissions from users' personal mailboxes. For example each + user might have an "Invoices" mailbox which will be read-only. + 2. Giving permissions to master user logins. See + [Authentication.MasterUsers.txt] for more + information. + +If a mailbox has both global ACLs and the per-mailbox ACL file, both of them +are read and the ACLs are merged. If there are any conflicts, the global ACL +file overrides per-mailbox ACL file. This is because users can modify their own +per-mailbox ACL files via IMAP ACL extension. Global ACLs can only be modified +by administrator, so users shouldn't be able to override them. + +Global ACL file (v2.2.11+) +-------------------------- + +Global ACL file path is specified as a parameter to vfile backend in 'acl' +setting ('/etc/dovecot/dovecot-acl' in the above example). The file contains +otherwise the same data as regular per-mailbox 'dovecot-acl' files, except each +line is prefixed by the mailbox name pattern. The pattern may contain "*" and +"?" wildcards. For example: + +---%<------------------------------------------------------------------------- +* user=foo lrw +Public user=bar lrwstipekxa +Public/* user=bar lrwstipekxa +---%<------------------------------------------------------------------------- + +Global ACL directory (obsolete) +------------------------------- + +Global ACL directory is specified as a parameter to vfile backend in 'acl' +setting ('/etc/dovecot/acls/' in the above example). They are looked up using +the mailbox's virtual name. For example: + + * INBOX: '/etc/dovecot/acls/INBOX' + * archives.2007: '/etc/dovecot/acls/archives.2007' + * archives/2007: '/etc/dovecot/acls/archives/2007' + +The filenames must start with namespace prefix (if it has one). For example +with namespace 'prefix=INBOX/' containing mailbox "foo" use +'/etc/dovecot/acls/INBOX/foo'. + +There is an extra problem with mailbox formats that use '/' as the separator +(e.g. mbox, dbox): For example if you have mailboxes "foo" and "foo/bar" and +you wish to give ACLs to both of them, you can't create both +'/etc/dovecot/acls/foo' and '/etc/dovecot/acls/foo/bar' files. The 'foo' has to +be either a directory or a file, it can't be both. To solve this problem, you +can instead create a .DEFAULT file for "foo": + + * foo: '/etc/dovecot/acls/foo/.DEFAULT' + * foo/bar: '/etc/dovecot/acls/foo/bar' + +ACL files +--------- + +The files themselves are in format: + +---%<------------------------------------------------------------------------- + [:] +---%<------------------------------------------------------------------------- + +Where *identifier* is one of: + + * group-override=*group name* + * user=*user name* + * owner + * group=*group name* + * authenticated + * anyone (or anonymous, which is alias for anyone) + +The ACLS are processed in the precedence given above, so for example if you +have given read-access to a group, you can still remove that from specific +users inside the group. + +Group-override identifier allows you to override users' ACLs. Probably the most +useful reason to do this is to temporarily disable access for some users. For +example: + +---%<------------------------------------------------------------------------- +user=timo rw +group-override=tempdisabled +---%<------------------------------------------------------------------------- + +Now if /timo/ is in /tempdisabled/ group, he has no access to the mailbox. This +wouldn't be possible with a normal group identifier, because the 'user=timo' +would override it. + +The currently supported ACLs and their corresponding named ACLs are: ++---+---------------+---------------------------------------------------------+ +| l | lookup | Mailbox is visible in mailbox list. Mailbox can be | +| | | subscribed to. | ++---+---------------+---------------------------------------------------------+ +| r | read | Mailbox can be opened for reading. | ++---+---------------+---------------------------------------------------------+ +| w | write | Message flags and keywords can be changed, except \Seen | +| | | and \Deleted | ++---+---------------+---------------------------------------------------------+ +| s | write-seen | \Seen flag can be changed | ++---+---------------+---------------------------------------------------------+ +| t | write-deleted | \Deleted flag can be changed | ++---+---------------+---------------------------------------------------------+ +| i | insert | Messages can be written or copied to the mailbox | ++---+---------------+---------------------------------------------------------+ +| p | post | Messages can be posted to the mailbox by , e.g.| +| | | from [Pigeonhole.Sieve.txt] scripts | ++---+---------------+---------------------------------------------------------+ +| e | expunge | Messages can be expunged | ++---+---------------+---------------------------------------------------------+ +| k | create | Mailboxes can be created (or renamed) directly under | +| | | this mailbox (but not necessarily under its children, | +| | | see ACL Inheritance section above) (renaming also | +| | | requires delete rights) | ++---+---------------+---------------------------------------------------------+ +| x | delete | Mailbox can be deleted | ++---+---------------+---------------------------------------------------------+ +| a | admin | Administration rights to the mailbox (currently: ability| +| | | to change ACLs for mailbox) | ++---+---------------+---------------------------------------------------------+ + +The ACLs are compatible with RFC 4314 (IMAP ACL extension, updated version). + +Unknown ACL letters are complained about, but unknown named ACLs are ignored. +Named ACLs are mostly intended for future extensions. + +Note that the file is rather picky about formatting; using a tab (or multiple +spaces) instead of a space character between fields may not work. If you are +having problems, make sure to check for tabs, extra spaces and other unwanted +characters. + +Examples +-------- + +Mailbox owner has all privileges, "timo" has list-read privileges: + +---%<------------------------------------------------------------------------- +owner lrwstipekxa +user=timo lr +---%<------------------------------------------------------------------------- + +Allow everyone to list and read a public mailbox (public namespace has no +owner): + +---%<------------------------------------------------------------------------- +anyone lr +---%<------------------------------------------------------------------------- + +Prevent all users from deleting their Spam folder (notice no x flag) + +---%<------------------------------------------------------------------------- +INBOX.Spam owner lrwstipeka +---%<------------------------------------------------------------------------- + +List cache +---------- + +'dovecot-acl-list' file lists all mailboxes that have "l" rights assigned. If +you manually add/edit 'dovecot-acl' files, you may need to delete the +'dovecot-acl-list' to get the mailboxes visible. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AixPluginsSupport.txt b/doc/wiki/AixPluginsSupport.txt new file mode 100644 index 0000000000..444c00d7df --- /dev/null +++ b/doc/wiki/AixPluginsSupport.txt @@ -0,0 +1,141 @@ +Plugins Support on AIX +====================== + +/How to build Dovecot with plugins supported on AIX./ + +Tested with: + + * AIX 5.2, VAC 5.0.2 and Dovecot 1.0rc21 to 1.0.13 /(Dovecot 1.1.x does not + build with VAC 5.0.2)/ + * AIX 5.2, VAC 8.0.0 and Dovecot 1.1.1 + +The Problem +----------- + + * When you trying to use plugins on AIX you seen error messages like this: + ---%<---------------------------------------------------------------------- + imap(root): Error: + dlopen(/usr/local/lib/dovecot/imap/lib20_zlib_plugin.so) failed: + rtld: 0712-001 Symbol i_error was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol i_stream_get_data was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol i_stream_skip was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol i_stream_seek was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol i_stream_close was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol i_panic was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + rtld: 0712-001 Symbol pool_get_exp_grown_size was referenced + from module /usr/local/lib/dovecot/imap/lib20_zlib_plugin.so(), + but a runtime definition of the symbol was not found. + Additional errors occurred but are not reported. + ---%<---------------------------------------------------------------------- + + * .. produced by executing + ---%<---------------------------------------------------------------------- + # MAIL_PLUGINS=zlib /usr/local/libexec/dovecot/imap + ---%<---------------------------------------------------------------------- + +Compiler Script +--------------- + + * Create a compiler script to rewrite the /xlc/ command line on the fly: + *dovecot-cc* + + ---%<---------------------------------------------------------------------- + #!/bin/bash + + xlc=/usr/bin/xlc + ar=/bin/ar + sed=/bin/sed + + dest=NOBINARY + + for i in "$@"; do + case "$i" in + '-o') dest=;; + *) if [ -z "$dest" ]; then dest="$i"; break; fi;; + esac + done + + case "$dest" in + imap-login) args1="../lib-charset/libcharset.a ../lib-mail/libmail.a + -liconv";; + imap) args1="../lib-sql/libsql.a";; + pop3) args1="../lib-sql/libsql.a";; + deliver) args1="../lib-sql/libsql.a";; + esac + + for i in "$@" $args1; do + case "$i" in + */*.a) lib="${i##*/}"; obj=`$ar -t $i | $sed "s:^:${i%/*}/:"`;; + *.a) lib="$i"; obj=`$ar -t $i | $sed "s:^:./:"`;; + *) continue;; + esac + test -d .libs || mkdir .libs + > .libs/${lib%.a}.exp + args2="$args2 -bE:.libs/${lib%.a}.exp" + + (set -x ; exec $xlc -qmkshrobj -qexpfile=.libs/${lib%.a}.exp $obj) + 2>/dev/null + done + (set -x ; exec $xlc "$@" $args1 $args2) + ---%<---------------------------------------------------------------------- + +Compiling Dovecot +----------------- + + * Expand Dovecot: + ---%<---------------------------------------------------------------------- + gzip -cd doveccot-1.0.rc21.tar.gz | tar xvf - + ---%<---------------------------------------------------------------------- + + * Setup build environment: + ---%<---------------------------------------------------------------------- + export CC=$PWD/dovecot-cc + export LDFLAGS="-bexpall -brtl" + ---%<---------------------------------------------------------------------- + + * Configure and build Dovecot + ---%<---------------------------------------------------------------------- + cd dovecot-1.0.rc21 + bash configure + make + ---%<---------------------------------------------------------------------- + + * Test a plugin (e.g. zlib) + ---%<---------------------------------------------------------------------- + echo 0 logout | MAIL_PLUGIN_DIR=src/plugins/zlib/.libs/ MAIL_PLUGINS="zlib" + MAIL=maildir:/tmp src/imap/imap + ---%<---------------------------------------------------------------------- + + you should see this: + ---%<---------------------------------------------------------------------- + * PREAUTH [CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND + UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS] Logged in as root + * BYE Logging out + 0 OK Logout completed. + imap(root): Info: Disconnected: Logged out + ---%<---------------------------------------------------------------------- + + * Install Dovecot + ---%<---------------------------------------------------------------------- + make install + ---%<---------------------------------------------------------------------- + +Prebuild Binaries for AIX 5.2 +----------------------------- + +You will find prebuild AIX 5.2 binaries here: +http://www.fh-trier.de/~beckerr/dovecot/ + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AttachmentIndicator.txt b/doc/wiki/AttachmentIndicator.txt new file mode 100644 index 0000000000..1acac6f384 --- /dev/null +++ b/doc/wiki/AttachmentIndicator.txt @@ -0,0 +1,31 @@ +Attachment indicator +==================== + +Since 2.2.34/2.3.1 dovecot has a feature that indicates whether email has an +attachment or not via keywords. These keywords could be used by IMAP clients. +These keywords will likely become standardized - for now the latest information +is in https://www.ietf.org/mail-archive/web/imapext/current/msg05858.html + +Configuration +------------- + +To enable this feature, you can set *mail_attachment_detection_options*. + +It supports following options + + * add-flags-on-save - Enables the feature, attachments are detected and marked + during save + * content-type=type|!type - Include or exclude given content type. Including + will only negate an exclusion (e.g. content-type=!foo/* + content-type=foo/bar). + * exclude-inlined - Do not consider any attachment with disposition inlined. + +Usage +----- + +Once enabled, mails that are saved, are marked with $Has ''Attachment or $Has +''No ''Attachment keyword. These keywords can be cleared, and are not +protected. Since v2.3.3 it is possible to post-process attachments with +'doveadm rebuild attachments' command. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.CheckPassword.txt b/doc/wiki/AuthDatabase.CheckPassword.txt new file mode 100644 index 0000000000..c285868ca9 --- /dev/null +++ b/doc/wiki/AuthDatabase.CheckPassword.txt @@ -0,0 +1,234 @@ +CheckPassword +============= + +Since v2.3.0 You can also use Lua to write your custom authentication, see + + +Checkpassword is an authentication interface originally implemented by qmail +[http://www.qmail.org/]. Checkpassword combines both the +[PasswordDatabase.txt] and [UserDatabase.txt] lookups into a +single checkpassword lookup, which makes the standard implementation unsuitable +for a standalone userdb. With Dovecot extensions it's also possible to use +checkpassword as a userdb. + +Typically you'll use [UserDatabase.Prefetch.txt] as the userdb, but +it's not required that you use the checkpassword script's userdb capabilities. +You can still use for example [UserDatabase.Static.txt] if +you're using only a single UID and GID, and your home directory fits into a +template. + +Security +-------- + +The standard checkpassword design is incompatible with Dovecot's security +model. If the system has local users and the checkpassword script setuid()s +into a local user, the user is able to ptrace into the communication and change +the authentication results. This is of course undesirable, so v2.2.7+ will just +refuse to run in such environments by default. The possibilities to solve this +are: + + 1. If possible, change the checkpassword to return 'userdb_uid' and + 'userdb_gid' extra fields instead of using 'setuid()' and 'setgid()'. This + also improves the performance. + 2. If you can't change the script, you can make Dovecot's + 'checkpassword-reply' binary setuid or setgid (e.g.'chgrp dovecot + /usr/local/libexec/dovecot/checkpassword-reply; chmod g+s + /usr/local/libexec/dovecot/checkpassword-reply') + 3. If you don't have any untrusted local users and you just don't care about + this check, you can set 'INSECURE_SETUID=1' environment e.g. with a wrapper + checkpassword script. + +Deliver +------- + +If your checkpassword script doesn't support Dovecot extensions, you can't use +it as a user database. This means that if you wish to use , you can't +use the '-d' parameter to do userdb lookups. There are two ways to solve this: + + 1. Use another userdb which does the lookup for deliver, for example + [AuthDatabase.SQL.txt] or [UserDatabase.Static.txt]. Add this + userdb after the prefetch userdb. + 2. Use a script to look up the user's home directory and run deliver without + '-d' parameter. For example: + +---%<------------------------------------------------------------------------- +#!/bin/sh + +# <> + +# If users have different UIDs/GIDs, make sure to also change this process's +UID and GID. +# If you want to override any settings, use dovecot-lda's -o parameter +# (e.g. dovecot-lda -o mail_location=maildir:~/Maildir). + +export HOME +exec /usr/local/libexec/dovecot/dovecot-lda +---%<------------------------------------------------------------------------- + +Checkpassword Interface +----------------------- + +The interface is specified in http://cr.yp.to/checkpwd/interface.html. However +here's a quick tutorial for writing a script: + + * Read ' NUL NUL' from fd 3. + * Verify the username and password. + * If the authentication fails, exit with code 1. This makes Dovecot give + "Authentication failed" error to user. + * This error is returned both for password mismatch and also if the user + doesn't exist at all. Internally Dovecot maps this as password + mismatch. + * If you encounter an internal error, exit with code 111. This makes + Dovecot give "Temporary authentication failure" error to user. + * If the authentication succeeds, you'll need to: + * Set user's home directory to '$HOME' environment. This isn't required, + [VirtualUsers.txt]. + * Set '$USER' environment variable. If the user name was changed (eg. if + you lowercased "Username" to "username"), you can tell about it to + Dovecot by setting '$USER' to the changed user name. + * Return the user's [UserIds.txt] using 'userdb_uid' and + 'userdb_gid' environments and add them to the 'EXTRA' environment (see + below for Dovecot extensions). + * This is recommended over actually changing the UID/GID using + setuid()/setgid() as specified by the standard checkpassword + interface, because it's + [AuthDatabase.CheckPassword.txt]. + * Your program received a path to 'checkpassword-reply' binary as the first + parameter. Execute it. + +Qmail-LDAP +---------- + +Note that auth_imap that comes with qmail-ldap is not compatible with this +interface. You can get a patch that adds auth_dovecot functionality to +qmail-ldap here +[http://japc.uncovering.org/dovecot/qmail-ldap-1.03-20060201-dovecot.patch]. Or +you can use auth_pop instead, but you may need to pass /aliasempty/ to let +auth_pop find the Maildir, so it is recommended to write a +/var/qmail/bin/auth_dovecot wrapper (don't forget to chmod +x it) around +auth_pop. + +---%<------------------------------------------------------------------------- +#!/bin/sh +QMAIL="/var/qmail" +if [ -e $QMAIL/control/defaultdelivery ]; then + ALIASEMPTY=`head -n 1 $QMAIL/control/defaultdelivery 2> /dev/null` +else + ALIASEMPTY=`head -n 1 $QMAIL/control/aliasempty 2> /dev/null` +fi +ALIASEMPTY=${ALIASEMPTY:-"./Maildir/"} +exec $QMAIL/bin/auth_pop "$@" $ALIASEMPTY +---%<------------------------------------------------------------------------- + +you can also use this wrapper to pass LOGLEVEL environmental variable to +auth_pop. + +Dovecot Extensions +------------------ + +If you wish to return [PasswordDatabase.ExtraFields.txt] for +Dovecot, set them in environment variables and then list them in EXTRA +environment variable. The [UserDatabase.ExtraFields.txt] +can be returned by prefixing them with 'userdb_'. For example: + +---%<------------------------------------------------------------------------- +userdb_quota_rule=*:storage=10000 +userdb_mail=mbox:$HOME/mboxes +EXTRA=userdb_quota_rule userdb_mail +---%<------------------------------------------------------------------------- + +Dovecot also sets some environment variables that the script may use: + + * 'SERVICE': contains eg. imap, pop3 or smtp + * 'TCPLOCALIP' and 'TCPREMOTEIP': Client socket's IP addresses if available + * 'MASTER_USER': If master login is attempted. This means that the password + contains the master user's password and the normal username contains the + user who master wants to log in as. + * 'AUTH_*': All of the [Variables.txt] are available as + 'AUTH_' extra fields. For example '%{cert}' is in 'AUTH_CERT'. + (v2.0.16+) + +Checkpassword as userdb +----------------------- + +Dovecot calls the script with 'AUTHORIZED=1' environment set when performing a +userdb lookup. The script must acknowledge this by changing the environment to +'AUTHORIZED=2', otherwise the lookup fails. Other than that, the script works +the same way as a passdb checkpassword script. If user doesn't exist, use exit +code 3. + +Checkpassword with passdb lookups (v2.1.2+) +------------------------------------------- + +Normally checkpassword answers to questions "is user X's password Y?" This +doesn't work with non-plaintext auth mechanisms, or when Dovecot wants to do a +non-authenticating passdb lookup (e.g. for LMTP proxy). These passdb +credentials lookups can be implemented the same way as a userdb lookup (i.e. +change the 'AUTHORIZED' environment). + + * 'AUTHORIZED=1' is set, just like for userdb lookup + * When doing a non-plaintext authentication: + * 'CREDENTIALS_LOOKUP=1' environment is set + * The password scheme that Dovecot wants is available in 'SCHEME' + environment (e.g.'SCHEME=CRAM-MD5') + * If a password is returned, it must be returned as + 'password={SCHEME}secret'. + * When doing a passdb lookup, e.g. a proxy which doesn't really want the + password, just the passdb extra fields: + * Neither 'CREDENTIALS_LOOKUP' nor 'SCHEME' is set. + * FIXME: Unfortunately it looks like you currently can't easily + differentiate a passdb lookup from userdb lookup! + * If user doesn't exist, use exit code 3. + * If you get an error about checkpassword exiting with code 0, you didn't + execute the 'checkpassword-reply' binary as you should have (which exits + with code 2 on success) + +Example +------- + +The standard way: + +---%<------------------------------------------------------------------------- +passdb { + driver = checkpassword + args = /usr/bin/checkpassword +} +userdb { + driver = prefetch +} +# If you want to use deliver -d and your users are in SQL: +userdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +Using checkpassword only to verify the password: + +---%<------------------------------------------------------------------------- +passdb { + driver = checkpassword + args = /usr/bin/checkpassword +} +userdb { + driver = static + args = uid=vmail gid=vmail home=/home/%u +} +---%<------------------------------------------------------------------------- + +Performance +----------- + +The backend is not suited for heavy traffic. Especially if +the script spawned has to launch an entire language interpreter. + +If your user database is only accessible with custom code an alternative might +be using the [AuthDatabase.Dict.txt]. + +Specific checkpassword implementations +-------------------------------------- + + * phpBB dovecot checkpassword authentication, written in python: + https://github.com/ser/checkpassword-phpbb + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.Dict.txt b/doc/wiki/AuthDatabase.Dict.txt new file mode 100644 index 0000000000..4a4405a99c --- /dev/null +++ b/doc/wiki/AuthDatabase.Dict.txt @@ -0,0 +1,391 @@ +Key-value authentication database +================================= + +Key-value databases can be used as auth backends. They probably should be used +only for caching in front of e.g. SQL auth backends. Iteration is supported if +the underlying dict provider supports iteration. See for list +of supported databases. + +Auth configuration +------------------ + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +passdb { + driver = dict + args = /etc/dovecot/dovecot-dict-auth.conf +} +userdb { + driver = dict + args = /etc/dovecot/dovecot-dict-auth.conf +} +---%<------------------------------------------------------------------------- + +Dict configuration +------------------ + +---%<------------------------------------------------------------------------- +uri = redis:host=127.0.0.1:port=6379 + +# Dictionary URI +#uri = + +# Default password scheme +default_pass_scheme = MD5 + +# Username iteration prefix. Keys under this are assumed to contain usernames. +iterate_prefix = userdb/ + +# Should iteration be disabled for this userdb? If this userdb acts only as a +# cache there's no reason to try to iterate the (partial & duplicate) users. +#iterate_disable = no + +# The example here shows how to do multiple dict lookups and merge the replies. +# The "passdb" and "userdb" keys are JSON objects containing key/value pairs, +# for example: { "uid": 1000, "gid": 1000, "home": "/home/user" } + +key passdb { + key = passdb/%u + format = json +} +key userdb { + key = userdb/%u + format = json +} +key quota { + key = userdb/%u/quota # or e.g. quota/%{userdb:quota_class} + #format = value + # The default_value is used if the key isn't found. If default_value setting + # isn't specified at all (even as empty), the passdb/userdb lookup fails with + # "user doesn't exist". + default_value = 100M +} + +# Space separated list of keys whose values contain key/value paired objects. +# All the key/value pairs inside the object are added as passdb fields. +# This can only be used for JSON formatted values. +passdb_objects = passdb + +#passdb_fields { +#} + +# Userdb key/value object list. +userdb_objects = userdb + +userdb_fields { + # dict: refers to key names + quota_rule = *:storage=%{dict:quota} + + # dict:. refers to the objkey inside (JSON) object + mail = maildir:%{dict:userdb.home}/Maildir +} +---%<------------------------------------------------------------------------- + +Example values +-------------- + +The value formats are either "value" that contains a direct value, or "json". +For example userdb lookup should return something like: + +---%<------------------------------------------------------------------------- +{ "uid": 123, "gid": 123, "home": "/home/username" } +---%<------------------------------------------------------------------------- + +dict proxying +------------- + +It may be useful to do the lookups via the "dict" or "dict-async" service. For +example: + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +dict { + cassandra-userdb = cassandra:/etc/dovecot/dovecot-dict-userdb-cql.conf.ext +} +---%<------------------------------------------------------------------------- + +'dovecot-dict-auth.conf.ext': + +---%<------------------------------------------------------------------------- +uri = proxy:dict-async:cassandra-userdb +iterate_disable = yes +# The _key and _path suffixes are not necessary, they're just here to help +# understand how to match them between different parts of the configuration. +key email_key { + key = userdb/email_path/%u +} +key displayname_key { + key = userdb/displayname_path/%u +} +userdb_fields { + # these fields will be visible as %{userdb:u_email} and +%{userdb:u_displayname} + u_email = %{dict:email_key} + u_displayname = %{dict:displayname_key} +} +---%<------------------------------------------------------------------------- + +'dovecot-dict-userdb-cql.conf.ext': + +---%<------------------------------------------------------------------------- +driver = cassandra +connect = host=127.0.0.1 dbname=email_users + +# SELECT displayname FROM user_profile WHERE id = %u +map { + # pattern must match the "key" path, except with added shared/ prefix. %u +gets caught into $username + pattern = shared/userdb/displayname_path/$username + table = user_profile + value_field = displayname + value_type = string + fields { + id = $username + } +} + +# SELECT email FROM user_profile WHERE id = %u +map { + pattern = shared/userdb/email_path/$username + table = user_profile + value_field = email + value_type = string + fields { + id = $username + } +} +---%<------------------------------------------------------------------------- + +Complete example for authenticating via the CDB dictionary +---------------------------------------------------------- + +This example uses the CDB dictionary to store the userdb and passdb. + +Auth configuration +------------------ + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +# Access to the CDB has to go through a dict process. +dict { + auth = cdb:/etc/dovecot/auth.cdb +} + +passdb { + driver = dict + args = /etc/dovecot/dovecot-cdb.conf +} + +userdb { + driver = dict + args = /etc/dovecot/dovecot-cdb.conf +} +---%<------------------------------------------------------------------------- + +Dict configuration +------------------ + +The CDB dictionary doesn't support iteration yet. + +'dovecot-cdb.conf': + +---%<------------------------------------------------------------------------- +uri = proxy::auth + +# FIXME: obsolete configuration - should use the key { .. } instead +password_key = passdb/%u +user_key = userdb/%u +# iterate_prefix = userdb/ # no yet supported +iterate_disable = yes + +default_pass_scheme = BLF-CRYPT +---%<------------------------------------------------------------------------- + +Complete example for authenticating via a UNIX socket +----------------------------------------------------- + +The Dict auth backend can be used to query a local UNIX socket for users. This +can be handy for accessing user databases which would otherwise only be +accessible via the [AuthDatabase.CheckPassword.txt] backend and +a scripting language. + +When given a <"proxy:"> [Quota.Dict.txt] URL the Dict backend speaks a simple +protocol over a UNIX socket. The protocol is defined in +'src/lib-dict/dict-client.h' (GitHub +[https://github.com/dovecot/core/blob/master/src/lib-dict/dict-client.h]). + +Auth configuration +------------------ + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +passdb { + driver = dict + args = /etc/dovecot/dovecot-dict-auth.conf +} +userdb { + # optional + driver = prefetch +} +userdb { + driver = dict + args = /etc/dovecot/dovecot-dict-auth.conf +} +---%<------------------------------------------------------------------------- + +Dict configuration +------------------ + +The last "dictionary name" ("somewhere") argument is redundant here. + +'/etc/dovecot/dovecot-dict-auth.conf.ext': + +---%<------------------------------------------------------------------------- +uri = proxy:/var/run/auth_proxy_dovecot/socket:somewhere + +# FIXME: obsolete configuration - should use the key { .. } instead +password_key = passdb/%u +user_key = userdb/%u +iterate_disable = yes +#default_pass_scheme = plain +---%<------------------------------------------------------------------------- + +Server process for answering Dict lookups +----------------------------------------- + +The server process listening on '/var/run/auth_proxy_dovecot/socket' can be +written in any language.Here's an example in Perl: + +---%<------------------------------------------------------------------------- +package AuthProxyDovecot; +use base qw( Net::Server::PreFork ); + +use strict; +use warnings; + +use JSON::XS; + +AuthProxyDovecot->run() or die "Could not initialize"; + +sub default_values +{ + return { + port => '/var/run/auth_proxy_dovecot/socket|unix', + + log_level => 2, + log_file => 'Sys::Syslog', + syslog_logsock => 'unix', + syslog_ident => 'auth_proxy_dovecot', + syslog_facility => 'daemon', + + background => 1, + setsid => 1, + pid_file => '/var/run/auth_proxy_dovecot.pid', + + user => 'root', + group => 'root', + + max_spare_servers => 2, + min_spare_servers => 1, + min_servers => 2, + max_servers => 10, + + }; +} ## end sub default_values + +################################################## + +sub process_request { + my $self = shift; + + my %L_handler = ( + passdb => sub { + my ($arg) = @_; + my $ret = { + password => '$1$JrTuEHAY$gZA1y4ElkLHtnsrWNHT/e.', + userdb_home => "/home/username/", + userdb_uid => 1000, + userdb_gid => 1000, + }; + return $ret; + }, + userdb => sub { + my ($arg) = @_; + my $ret = { + home => "/home/username/", + uid => 1000, + gid => 1000, + }; + return $ret; + }, + ); + + # protocol from src/lib-dict/dict-client.h + my $json = JSON::XS->new; + + eval { + my $ret; + # Dict protocol is multiline... go through the lines. + while () { + $self->log(2, "Got request: $_"); + chomp; + my $cmd = substr($_,0,1); + next if $cmd eq 'H'; # "hello", skip this line, assume it's ok + die "Protocol error: Bad command $cmd" unless ($cmd eq 'L'); + # Process request + + my ($namespace,$type,$arg) = split ('/',substr($_,1),3); + + if ($namespace eq 'shared') { + my $f = $L_handler{$type}; + + if (defined $f && defined $arg) { + $ret = $f->($arg); + } + else { + die 'Protocol error: Bad arg'; + } + } + else { + die 'Protocol error: Bad namespace' + } + last; # Got an "L" , now respond. + } + if ($ret) { + my $json = JSON::XS->new->indent(0)->utf8->encode($ret); + $self->log(3,"O:$json"); + print "O".$json."\n"; + } + else { + $self->log(3,"NOUSER"); + print "N\n"; + } + 1; + } or do { + $self->log(2, "Error: $@"); + print "F\n"; + }; +} + +sub pre_loop_hook { + my $self = shift; + + $self->log(1, 'Starting server'); +} + +sub pre_server_close_hook { + my $self = shift; + + $self->log(1, 'Server is shut down'); +} + +1; + +__END__ +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.LDAP.AuthBinds.txt b/doc/wiki/AuthDatabase.LDAP.AuthBinds.txt new file mode 100644 index 0000000000..a81eef3049 --- /dev/null +++ b/doc/wiki/AuthDatabase.LDAP.AuthBinds.txt @@ -0,0 +1,89 @@ +Passdb LDAP with authentication binds +===================================== + +Advantages over [AuthDatabase.LDAP.PasswordLookups.txt]: + + * LDAP server verifies the password, so Dovecot doesn't need to know what + format the password is stored in. + * A bit more secure, as a security hole in Dovecot doesn't give attacker + access to all the users' password hashes. (And Dovecot admins in general + don't have direct access to them.) + +You can enable authentication binds by setting 'auth_bind=yes'. Next Dovecot +needs to know what DN to use in the binding. There are two ways to configure +this: lookup or template. + +DN lookup +--------- + +DN is looked up by sending a 'pass_filter' LDAP request and getting the DN from +the reply. This is very similar to doing a +[AuthDatabase.LDAP.PasswordLookups.txt]. The only difference is that +userPassword attribute isn't returned. Just as with password lookups, the +'pass_attrs' may contain special +[PasswordDatabase.ExtraFields.txt]. + +Example: + +---%<------------------------------------------------------------------------- +auth_bind = yes +pass_attrs = uid=user +pass_filter = (&(objectClass=posixAccount)(uid=%u)) +---%<------------------------------------------------------------------------- + +DN template +----------- + +The main reason to use DN template is to avoid doing the DN lookup, so that the +authentication consists only of one LDAP request. With IMAP and POP3 logins the +same optimization can be done by using +[UserDatabase.Prefetch.txt] and returning userdb info in the DN lookup (a total +of two LDAP requests per login in both cases). If you're also using Dovecot for +SMTP AUTH, it doesn't do a userdb lookup so the prefetch optimization doesn't +help. + +If you're using DN template, 'pass_attrs' and 'pass_filter' settings are +completely ignored. That means you can't make passdb return any +[PasswordDatabase.ExtraFields.txt]. You should also set 'auth_username_format = +%Lu' in 'dovecot.conf' to normalize the username by lowercasing it. + +Example: + +---%<------------------------------------------------------------------------- +auth_bind = yes +auth_bind_userdn = cn=%u,ou=people,o=org +---%<------------------------------------------------------------------------- + +Connection optimization +----------------------- + +When using + + * auth binds and + * userdb ldap lookups, + +the userdb lookups should use a separate connection to the LDAP server. That +way it can send LDAP requests asynchronously to the server, which improves the +performance. This can be done by specifying different filenames in the LDAP +passdb and userdb args. The second file could be a symlink to the first one. +For example: + +---%<------------------------------------------------------------------------- +passdb { + driver = ldap + args = /etc/dovecot/dovecot-ldap.conf.ext +} +userdb { + driver = ldap + args = /etc/dovecot/dovecot-ldap-userdb.conf.ext +} +---%<------------------------------------------------------------------------- + +And create the symlink: + +---%<------------------------------------------------------------------------- +ln -s /etc/dovecot/dovecot-ldap.conf.ext +/etc/dovecot/dovecot-ldap-userdb.conf.ext +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.LDAP.PasswordLookups.txt b/doc/wiki/AuthDatabase.LDAP.PasswordLookups.txt new file mode 100644 index 0000000000..a9232d55bf --- /dev/null +++ b/doc/wiki/AuthDatabase.LDAP.PasswordLookups.txt @@ -0,0 +1,98 @@ +Passdb LDAP with password lookups +================================= + +Advantages over [AuthDatabase.LDAP.AuthBinds.txt]: + + * Faster, because Dovecot can keep sending multiple LDAP requests + asynchronously to the server. With auth binds Dovecot must wait for each + request to finish before sending the next one. + * Supports non-plaintext + [Authentication.Mechanisms.txt] (if returning plaintext/ [Authentication.PasswordSchemes.txt]). + * When using and static userdb, deliver can check if destination + user exists. With auth binds this check isn't possible. + +LDAP server permissions +----------------------- + +Normally LDAP server doesn't give anyone access to users' passwords, so you'll +need to create an administrator account that has access to the userPassword +field. With OpenLDAP this can be done by modifying '/etc/ldap/slapd.conf': + +---%<------------------------------------------------------------------------- +# there should already be something like this in the file: +access to attribute=userPassword + by dn="" read # just add this line + by anonymous auth + by self write + by * none +---%<------------------------------------------------------------------------- + +Replace with the DN you specified in 'dovecot-ldap.conf's' 'dn' +setting. + +Dovecot configuration +--------------------- + +The two important settings in password lookups are: + + * 'pass_filter' specifies the LDAP filter how user is found from the LDAP. You + can use all the normal [Variables.txt] like '%u' in the filter. + * 'pass_attrs' specifies a comma-separated list of attributes that are + returned from the LDAP. If you set it to empty, all the attributes are + returned. + +Usually the LDAP attribute names aren't the same as [PasswordDatabase.txt]. You must create a mapping +between them to get the wanted results. This is done by listing the fields as +'='. For example: + +---%<------------------------------------------------------------------------- +pass_attrs = uid=user, userPassword=password +---%<------------------------------------------------------------------------- + +This maps the LDAP "uid" attribute to Dovecot's "user" field and LDAP's +"userPassword" attribute to Dovecot's "password" field. These two fields should +always be returned, but it's also possible to return other special [PasswordDatabase.ExtraFields.txt]. + +Password +-------- + +Most importantly the 'pass_attrs' must return a "password" field, which +contains the user's password. The next thing Dovecot needs to know is what +format the password is in. If all the passwords are in same format, you can use +'default_pass_scheme' setting in 'dovecot-ldap.conf' to specify it. Otherwise +each password needs to be prefixed with "{password-scheme}", for example +"{plain}plaintext-password". See for a +list of supported password schemes. + +Username +-------- + +LDAP lookups are case-insensitive. Unless you somehow normalize the username, +it's possible that a user logging in as "user", "User" and "uSer" are treated +differently. The easiest way to handle this is to tell Dovecot to change the +username to the same case as it's in the LDAP database. You can do this by +returning "user" field in the 'pass_attrs', as shown in the above example. + +If you can't normalize the username in LDAP, you can alternatively lowercase +the username in 'dovecot.conf': + +---%<------------------------------------------------------------------------- +auth_username_format = %Lu +---%<------------------------------------------------------------------------- + +Example +------- + +A typical configuration would look like: + +---%<------------------------------------------------------------------------- +auth_bind = no +pass_attrs = uid=user, userPassword=password +pass_filter = (&(objectClass=posixAccount)(uid=%u)) +default_pass_scheme = MD5 +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.LDAP.Userdb.txt b/doc/wiki/AuthDatabase.LDAP.Userdb.txt new file mode 100644 index 0000000000..3371325030 --- /dev/null +++ b/doc/wiki/AuthDatabase.LDAP.Userdb.txt @@ -0,0 +1,181 @@ +Userdb LDAP +=========== + +Usually your LDAP database also contains the +[UserDatabase.txt]. If your home directory can be specified with a template and +you're using only a single [UserIds.txt], you should use [UserDatabase.Static.txt] instead to avoid an unnecessary LDAP lookup. +You can also use [UserDatabase.Prefetch.txt] to avoid the +userdb LDAP lookup. + +Userdb lookups are always done using the default DN ('dn' setting) bind. It's +not possible to do the lookup using the user's DN (remember that e.g. +needs to do userdb lookups without knowing the user's password). + +The userdb lookups are configured in very much the same way as [AuthDatabase.LDAP.PasswordLookups.txt]. Instead of 'pass_attrs' and +'pass_filter', the userdb uses 'user_attrs' and 'user_filter'. Typically +'pass_filter' and 'user_filter' are equivalent. + +If you're using a single UID and GID for all the users, you can specify them +globally with 'mail_uid' and 'mail_gid' settings instead of returning them from +LDAP. + +Example: + +---%<------------------------------------------------------------------------- +user_attrs = \ + =home=%{ldap:homeDirectory}, \ + =uid=%{ldap:uidNumber}, \ + =gid=%{ldap:gidNumber} +user_filter = (&(objectClass=posixAccount)(uid=%u)) + +# For using doveadm -A: +iterate_attrs = =user=%{ldap:uid} +iterate_filter = (objectClass=posixAccount) +---%<------------------------------------------------------------------------- + +Attribute templates (v2.1+) +--------------------------- + +You can mix static text with the value returned from LDAP by using %{ldap:*} +variables, which expand to the named LDAP attribute's value. Some examples: + +Create a "quota_rule" field with value "*:bytes=" where comes from +"quotaBytes" LDAP attribute: + +---%<------------------------------------------------------------------------- +user_attrs = \ + =quota_rule=*:bytes=%{ldap:quotaBytes} +---%<------------------------------------------------------------------------- + +Create a "mail" field with value "maildir:/var/mail//Maildir" where +comes from "sAMAccountName" LDAP attribute: + +---%<------------------------------------------------------------------------- +user_attrs = \ + =mail=maildir:/var/spool/vmail/%{ldap:sAMAccountName}/Maildir +---%<------------------------------------------------------------------------- + +You can add static fields that aren't looked up from LDAP. For example create a +"mail" field with value "maildir:/var/vmail/%d/%n/Maildir": + +---%<------------------------------------------------------------------------- +user_attrs = \ + =quota_rule=*:bytes=%{ldap:quotaBytes}, \ + =mail=maildir:/var/vmail/%d/%n/Maildir +---%<------------------------------------------------------------------------- + +If you don't want a field to exist at all when its LDAP attribute doesn't +exist, you can give the attribute name before the first "=" character. For +example this doesn't return "home" or "mail" fields if "mailboxPath" doesn't +exist: + +---%<------------------------------------------------------------------------- +user_attrs = \ + =quota_rule=*:bytes=%{ldap:quotaBytes}, \ + mailboxPath=home=/home/%{ldap:mailboxPath}, \ + mailboxPath=mail=maildir:~/Maildir +---%<------------------------------------------------------------------------- + +It's also possible to give default values to nonexistent attributes in v2.1.11+ +by using e.g.'%{ldap:userDomain:example.com} ' where if userDomain attribute +doesn't exist, example.com is used instead. + +Subqueries and pointers (v2.2) +------------------------------ + +LDAP values can now have DN pointers to other entries that are queried. + +*Note*: These aren't actually very useful anymore. See the next section for how +to do multiple queries more easily using multiple userdbs. + +Example: + +---%<------------------------------------------------------------------------- +user_attrs = \ + =user=%{ldap:uid}, \ + @mail=%{ldap:mailDN}, \ + =uid=%{ldap:uidNumber@mail}, \ + =gid=%{ldap:gidNumber@mail}, \ + =home=%{ldap:rootPath@mail}/%d/%n +---%<------------------------------------------------------------------------- + +This will do a regular lookup first. Then does another lookup with DN taken +from mailDN's value. The *@mail attributes are assigned from the second +lookup's results. + +---%<------------------------------------------------------------------------- +user_attrs = \ + =user=%{ldap:uid}, \ + =home=%{ldap_ptr:activePath}, \ + !primaryPath, !secondaryPath +---%<------------------------------------------------------------------------- + +The activePath's value can be either "primaryPath" or "secondaryPath". The +home's value will be the contents of that field. The !field tells Dovecot to +fetch the field's value but not to do anything with it otherwise. + +Multiple queries via userdbs (v2.2+) +------------------------------------ + +Example: Give the user a class attribute, which defines the default quota: + +dovecot.conf: + +---%<------------------------------------------------------------------------- +userdb { + driver = ldap + args = /etc/dovecot/dovecot-users-ldap.conf.ext + result_success = continue-ok +} +userdb { + driver = ldap + args = /etc/dovecot/dovecot-class-ldap.conf.ext + skip = notfound +} +---%<------------------------------------------------------------------------- + +/etc/dovecot/dovecot-users-ldap.conf.ext: + +---%<------------------------------------------------------------------------- +# If user has overridden quota, quota_rule is set below. Otherwise it's still +unset. +user_attrs = \ + =class=%{ldap:userClass} + quotaBytes=quota_rule=*:bytes=%{ldap:quotaBytes} +---%<------------------------------------------------------------------------- + +/etc/dovecot/dovecot-class-ldap.conf.ext: + +---%<------------------------------------------------------------------------- +# Do the lookup using the user's class: +user_filter = (&(objectClass=userClass)(class=%{userdb:class})) +# With :protected suffix the quota_rule isn't overridden if it's already set. +user_attrs = \ + classQuotaBytes=quota_rule:protected=*:bytes=%{ldap:classQuotaBytes} +---%<------------------------------------------------------------------------- + +Variables and domains +--------------------- + +User names and domains may be distinguished using the %n and +%d. They split the /previous username/ at the "@" character. The /previous +username/ is: + + * For LMTP, it will be user@hostname, where hostname depends on e.g. the + Postfix configuration. + * For IMAP, it will be whatever the password database has designated as the + username. If the (LDAP) password database has "user_attrs = =user=%n", then + the domain part of the login name will be stripped by the password database. + The UserDB will not see any domain part, i.e. %n and %u are the same thing + for the UserDB. + +The UserDB may set a new username, too, using "user_attrs = =user=...". This +will be used for + + * Logging + * %u and %d variables in other parts of the configuration (e.g. quota file + names) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.LDAP.txt b/doc/wiki/AuthDatabase.LDAP.txt new file mode 100644 index 0000000000..3ad234840f --- /dev/null +++ b/doc/wiki/AuthDatabase.LDAP.txt @@ -0,0 +1,79 @@ +LDAP +==== + +There are two ways to do LDAP authentication: + + * [AuthDatabase.LDAP.PasswordLookups.txt] + * [AuthDatabase.LDAP.AuthBinds.txt] + +Both of these have their own advantages and disadvantages. + + * [AuthDatabase.LDAP.Userdb.txt] and other common LDAP query + settings. + +Configuration common to LDAP passdb and userdb +---------------------------------------------- + +Connecting +---------- + +There are two alternative ways to specify what LDAP server(s) to connect to: + + * 'hosts': A space separated list of LDAP hosts to connect to. You can also + use host:port syntax to use different ports. + * 'uris': A space separated list of LDAP URIs to connect to. This isn't + supported by all LDAP libraries. The URIs are in syntax + 'protocol://host:port'. For example 'ldap://localhost' or + 'ldaps://secure.domain.org' + +If multiple LDAP servers are specified, it's decided by the LDAP library how +the server connections are handled. Typically the first working server is used, +and it's never disconnected from. So there is no load balancing or automatic +reconnecting to the "primary" server. + +SSL/TLS +------- + +You can enable TLS in two alternative ways: + + * Connect to ldaps port (636) by using "ldaps" protocol, e.g. 'uris = + ldaps://secure.domain.org' + * Connect to ldap port (389) and use STARTTLS command. Use 'tls=yes' to enable + this. + +See the tls_* settings in 'dovecot-ldap-example.conf' for how to configure TLS. +(I think they apply to ldaps too?) + +Getting Dovecot to talk to a LDAPS signed against a custom certificate of +authority +----------------------------------------------------------------------------------- + +If you need to connect to ldaps secured against a custom certificate of +authority (CA), you will need to install the custom CA on your system.On Red +Hat Enterprise Linux 6, Dovecot uses the OpenLDAP library. By default, the CA +must be installed under the directory specified in the TLS_CACERTDIR option +found under /etc/openldap/ldap.conf (default value is /etc/openldap/certs). +After copying the CA, you'll need to run "c_rehash ." inside the directory, +this will create a symlink pointing to the CA. + +You can test the CA installation with this: openssl s_client -connect +yourldap.example.org:636 -CApath /etc/openldap/certs -showcerts + +This should report "Verify return code: 0 (ok)". + +SASL binds +---------- + +It's possible to use SASL binds instead of the regular plaintext binds if your +LDAP library supports them. See the sasl_* settings in +'dovecot-ldap-example.conf'. Note that SASL binds are currently incompatible +with authentication binds. + +Active Directory +---------------- + +When connecting to AD, you may need to use port 3268. Then again, not all LDAP +fields are available in port 3268. Use whatever +works.http://technet.microsoft.com/en-us/library/cc978012.aspx + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.Lua.txt b/doc/wiki/AuthDatabase.Lua.txt new file mode 100644 index 0000000000..41caf61f3d --- /dev/null +++ b/doc/wiki/AuthDatabase.Lua.txt @@ -0,0 +1,293 @@ +Lua based authentication +======================== + +Since v2.3.0 you can implement passdb and userdb using Lua +[https://www.lua.org/] script. + +Contents + + + 1. Lua based authentication + + 1. Known bugs + + 2. Lua interface + + 3. Auth request methods + + 4. Password database + + 5. User database + + 6. Examples + +Known bugs +---------- + + * Before 2.3.4 when returning a table with values, the table values are + mistakenly converted into a number if they seem like a number. So if you are + using values like '012345', this would get converted into '12345' + * Before 2.3.4 returning password without scheme would cause a crash. + +Lua interface +------------- + +For details about Dovecot Lua, see . + +When used in authentication, additional module *dovecot.auth* is added, which +contains constants for passdb and userdb. + +List of constants +----------------- + + * dovecot.auth.PASSDB_RESULT_INTERNAL_FAILURE + * dovecot.auth.PASSDB_RESULT_SCHEME_NOT_AVAILABLE - indicates password scheme + that cannot be understood + * dovecot.auth.PASSDB_RESULT_USER_UNKNOWN + * dovecot.auth.PASSDB_RESULT_USER_DISABLED + * dovecot.auth.PASSDB_RESULT_PASS_EXPIRED + * dovecot.auth.PASSDB_RESULT_NEXT - indicates that this passdb did not + authenticate user, next passdb should do it + * dovecot.auth.PASSDB_RESULT_PASSWORD_MISMATCH + * dovecot.auth.PASSDB_RESULT_OK + * dovecot.auth.USERDB_RESULT_INTERNAL_FAILURE + * dovecot.auth.USERDB_RESULT_USER_UNKNOWN + * dovecot.auth.USERDB_RESULT_OK + +Also, it registers object *struct auth_request** which lets access various +parts of the auth request. You should use the loggers associated with +auth_request when possible. + +Auth request methods +-------------------- + +Functions: + + * auth_request#log_debug(text) - logs debug message (if debug is enabled, noop + otherwise) + * auth_request#log_error(text) - logs error message + * auth_request#log_info(text) - logs informational message + * auth_request#log_warning(text) - logs warning message + * auth_request#response_from_template(template) - takes in key=value template + and expands it using var_expand and produces table suitable for passdb + result + * auth_request#var_expand(template) - performs var expansion on the template + using + * auth_request#password_verify(crypted_password, plain_password) - checks if + the plain password matches the crypted or hashed password + * auth_request#event() - Returns child event for the auth request, can be used + for logging and other events. Comes with a prefix. (Since v2.3.6+) + +Subtables: + + * auth_request#passdb + * auth_request#userdb + +Members: + +See for details + + * auth_request#auth_domain + * auth_request#auth_user + * auth_request#auth_username + * auth_request#cert + * auth_request#client_id + * auth_request#domain + * auth_request#domain_first + * auth_request#domain_last + * auth_request#home + * auth_request#lip + * auth_request#local_name + * auth_request#login_domain + * auth_request#login_user + * auth_request#login_username + * auth_request#lport + * auth_request#master_user + * auth_request#mech + * auth_request#orig_domain + * auth_request#orig_user + * auth_request#orig_username + * auth_request#password + * auth_request#pid + * auth_request#real_lip + * auth_request#real_lport + * auth_request#real_rip + * auth_request#real_rport + * auth_request#rip + * auth_request#rport + * auth_request#secured + * auth_request#service + * auth_request#session + * auth_request#session_pid + * auth_request#user + * auth_request#username + +Additionally you can access + + * skip_password_check - Set if the password has already been validated by + another passdb + * passdbs_seen_user_unknown - If some previous passdb has not found this user + * passdbs_seen_internal_failure - If some previous passdb has had internal + failure + * userdbs_seen_internal_failure - If some previous userdb has had internal + failure + +Password database +----------------- + +Lua passdb supports two modes of function. It can behave as lookup database, or +password verification database. + +Lookup function signature is *auth_passdb_lookup(request)* and the password +verification signature is *auth_password_verify(request, password)* + +Both functions must return a tuple, which contains a return code, and also +additionally string or table. Table must be in key-value format, it will be +imported into auth request. The string must be in key=value format, except if +return code indicates internal error, the second parameter can be used as error +string. + +If *auth_verify_password* is found, it's always used. + +To configure passdb in dovecot, use + +---%<------------------------------------------------------------------------- +passdb { + driver = lua + args = file=/path/to/lua blocking=yes # default is yes +} +---%<------------------------------------------------------------------------- + +By default, dovecot runs Lua scripts in auth-worker processes. If you do not +want this, you can disable blocking, and Lua script will be ran in auth +process. This can degrade performance if your script is slow or makes external +lookups. + +User database +------------- + +Lua userdb supports both single user lookup and iteration. Note that iteration +will hold the whole user database in memory during iteration. + +User lookup function signature is *auth_userdb_lookup(request)*. The function +must return a tuple, which contains a return code, and also additionally string +or table. Table must be in key-value format, it will be imported into auth +request. The string must be in key=value format, except if return code +indicates internal error, the second parameter can be used as error string. + +User iteration function signature is *auth_userdb_iterate*, which is expected +to return table of usernames. Key names are ignored. + +To configure userdb in dovecot, use + +---%<------------------------------------------------------------------------- +userdb { + driver = lua + args = file=/path/to/lua blocking=yes # default is yes +} +---%<------------------------------------------------------------------------- + +Examples +-------- + +Skeleton +-------- + +---%<------------------------------------------------------------------------- +function auth_passdb_lookup(req) + if req.user == "testuser1" then + return dovecot.auth.PASSDB_RESULT_OK, "password=pass" + end + return dovecot.auth.PASSDB_RESULT_USER_UNKNOWN, "no such user" +end + +function auth_userdb_lookup(req) + if req.user == "testuser1" then + return dovecot.auth.USERDB_RESULT_OK, "uid=vmail gid=vmail" + end + return dovecot.auth.USERDB_RESULT_USER_UNKNOWN, "no such user" +end + +function script_init() + return 0 +end + +function script_deinit() +end + +function auth_userdb_iterate() + return {"testuser1"} +end +---%<------------------------------------------------------------------------- + +Simple username password database (such as opensmtpd) +----------------------------------------------------- + +The example uses whitespace separated username and password. As a special +caution, the way Lua is used here means you can have multiple user password per +line, instead of just one.This can be extended to more complicated separators +or multiple fields per user. + +If you only want to autenticate users, and don't care about user listing, you +can use + +---%<------------------------------------------------------------------------- +function auth_passdb_lookup(req) + for line in io.lines("/path/to/file") do + for user, pass in string.gmatch(line, "(%w+)%s(.+)") do + if (user == req.username) then + -- you can add additional information here, like userdb_uid + return dovecot.auth.PASSDB_RESULT_OK, "password=" .. pass + end + end + end + return dovecot.auth.PASSDB_RESULT_USER_UNKNOWN, "" +end +---%<------------------------------------------------------------------------- + +If you also want to be able to list users, so that you could use doveadm cmd -A + +---%<------------------------------------------------------------------------- +local database = "/path/to/file" + +function db_lookup(username) + for line in io.lines(database) do + for user, pass in string.gmatch(line, "(%w+)%s(.+)") do + if (user == username) then + return {result=0, password=pass} + end + end + end + return {result=-1} +end + +function auth_passdb_lookup(req) + res = db_lookup(req.username) + if res.result == 0 then + -- you can add additional information here for passdb + return dovecot.auth.PASSDB_RESULT_OK, "password=" .. res.password + end + return dovecot.auth.PASSDB_RESULT_USER_UNKNOWN, "" +end + +function auth_userdb_lookup(req) + res = db_lookup(req.username) + if res.result == 0 then + -- you can add additional information here for userdb, like uid or home + return dovecot.auth.USERDB_RESULT_OK, "uid=vmail gid=vmail" + end + return dovecot.auth.USERDB_RESULT_USER_UNKNOWN, "" +end + +function auth_userdb_iterate() + users = {} + for line in io.lines(database) do + for user in string.gmatch(line, "(%w+)%s.+") do + table.insert(users, user) + end + end + return users +end +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.Passwd.txt b/doc/wiki/AuthDatabase.Passwd.txt new file mode 100644 index 0000000000..d9e1071e89 --- /dev/null +++ b/doc/wiki/AuthDatabase.Passwd.txt @@ -0,0 +1,53 @@ +Passwd +====== + +User is looked up using 'getpwnam()' call, which usually looks into +'/etc/passwd' file, but depending on NSS +[http://en.wikipedia.org/wiki/Name_Service_Switch] configuration it may also +look up the user from eg. LDAP database. + +Most commonly used as a user database. + +The lookup is by default done in the auth worker processes. If you have only a +small local passwd file, you can avoid having extra auth worker processes by +disabling it: + +---%<------------------------------------------------------------------------- +userdb { + driver = passwd + args = blocking=no +} +---%<------------------------------------------------------------------------- + +Field overriding and extra fields (obsolete in v2.1+) +----------------------------------------------------- + +It's possible to override fields from passwd and add +[UserDatabase.ExtraFields.txt] with templates, but in v2.1+ it's done in a +better way by using override_fields. For example: + +---%<------------------------------------------------------------------------- +userdb { + driver = passwd + # Pre-v2.1: + #args = home=/var/mail/%u mail=maildir:/var/mail/%u/Maildir + # v2.1+: + override_fields = home=/var/mail/%u mail=maildir:/var/mail/%u/Maildir +} +---%<------------------------------------------------------------------------- + +This uses the UID and GID fields from passwd, but home directory is overridden. +Also the default [MailLocation.txt] setting is overridden. + +Passwd as a password database +----------------------------- + +Many systems use shadow passwords nowadays so passwd doesn't usually work as a +password database. BSDs are an exception to this, they still set the password +field even with shadow passwords. + +With FreeBSD, passwd doesn't work as a password database because the password +field is replaced by a '*'. But you can use +[AuthDatabase.PasswdFile.txt]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.PasswdFile.txt b/doc/wiki/AuthDatabase.PasswdFile.txt new file mode 100644 index 0000000000..aa25b6d48a --- /dev/null +++ b/doc/wiki/AuthDatabase.PasswdFile.txt @@ -0,0 +1,150 @@ +Passwd-file +=========== + +This file is compatible with a normal '/etc/passwd' file, and a password file +used by libpam-pwdfile [PasswordDatabase.PAM.txt] plugin. It's in the +following format: + +---%<------------------------------------------------------------------------- +user:password:uid:gid:(gecos):home:(shell):extra_fields +---%<------------------------------------------------------------------------- + +For a password database it's enough to have only the user and password fields. +For a user database, you need to set also uid, gid and preferably also home +(see ). (gecos) and (shell) fields are unused by Dovecot. + +The password field can be in four formats: + + * 'password': Assume CRYPT + [Authentication.PasswordSchemes.txt]. + * '{SCHEME}password': The password is in the given + [Authentication.PasswordSchemes.txt]. + * 'password[13]': libpam-passwd file compatible format for CRYPT + [Authentication.PasswordSchemes.txt]. + * 'password[34]': libpam-passwd file compatible format for MD5 + [Authentication.PasswordSchemes.txt]. + +extra_fields is a space-separated list of key=value pairs which can be used to +set various [PasswordDatabase.ExtraFields.txt] and [UserDatabase.ExtraFields.txt]. Keys which begin with a 'userdb_' +prefix are used for userdb, others are used for passdb. So for example if you +wish to override [MailLocation.txt] setting for one user, use +'userdb_mail=mbox:~/mail'. [Variables.txt] expansion is done for +extra_fields. + +Empty lines and lines beginning with '#' character are ignored. + +Multiple passwd files +--------------------- + +You can use all the [Variables.txt] in the passwd-file filenames, +for example: + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + # Each domain has a separate passwd-file: + args = /etc/auth/%d/passwd +} +---%<------------------------------------------------------------------------- + +Passwd-file args +---------------- + + * *scheme=*: Allows you to specify the default + [Authentication.PasswordSchemes.txt]. The default is CRYPT. This is + available only for passdb. + * *username_format=*: Look up usernames using this format instead of the + full username ('%u'). If you want to enable user@domain logins but have only + "user" in the file, set this to '%n'. + +Examples +-------- + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + args = scheme=plain-md5 username_format=%n /etc/imap.passwd +} +userdb { + driver = passwd-file + args = username_format=%n /etc/imap.passwd + default_fields = uid=vmail gid=vmail home=/home/vmail/%u +} +---%<------------------------------------------------------------------------- + + * The default_fields is explained in + [UserDatabase.txt] They can be used to provide default userdb fields based + on templates in case they're not specified for everyone in the passwd file. + If you leave any of the standard userdb fields (uid, gid, home) empty, these + defaults will be used. + +This file can be used as a passdb: + +---%<------------------------------------------------------------------------- +user:{plain}password +user2:{plain}password2 +---%<------------------------------------------------------------------------- + +passdb with extra fields: + +---%<------------------------------------------------------------------------- +user:{plain}password::::::allow_nets=192.168.0.0/24 +---%<------------------------------------------------------------------------- + +This file can be used as both a passwd and a userdb: + +---%<------------------------------------------------------------------------- +user:{plain}pass:1000:1000::/home/user::userdb_mail=maildir:~/Maildir +allow_nets=192.168.0.0/24 +user2:{plain}pass2:1001:1001::/home/user2 +---%<------------------------------------------------------------------------- + +FreeBSD /etc/master.passwd as passdb and userdb +----------------------------------------------- + +On FreeBSD, '/etc/passwd' doesn't work as a password database because the +password field is replaced by a '*'. '/etc/master.passwd' can be converted into +a format usable by passwd-file. As can access the +system-wide credentials on FreeBSD, what follows is generally needed only if +the mail accounts are different from the system accounts. + +If only using the result for 'name:password:uid:gid' and not using + you may be able to use the extract directly. +However, the Linux-style passwd file has fewer fields than that used by FreeBSD +and it will need to be edited if any fields past the first four are needed. In +particular, it will fail if used directly as a 'userdb' as the field used for +'home' is not in the same place as expected by the Dovecot parser. The +':class:change:expire' stanza in each line should be removed to be consistent +with the Linux-style format. While that stanza often is '::0:0' use of 'cut' is +likely much safer than 'sed' or other blind substitution. + +In '/etc/master.passwd', a password of '*' indicates that password +authentication is disabled for that user and the token '*LOCKED*' prevents all +login authentication, so you might as well exclude those: + +---%<------------------------------------------------------------------------- +# fgrep -v '*' /etc/master.passwd | cut -d : -f 1-4,8-10 > +/path/to/file-with-encrypted-passwords +# chmod 640 /path/to/file-with-encrypted-passwords +# chown root:dovecot /path/to/file-with-encrypted-passwords +---%<------------------------------------------------------------------------- + +or permissions and ownership that may be more appropriate for your install and +security needs. + +The following will work in many situations, after disabling the inclusion of +other 'userdb' and 'passdb' sections + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + args = username_format=%n /path/to/file-with-encrypted-passwords +} +userdb { + driver = passwd-file + args = username_format=%n /path/to/file-with-encrypted-passwords +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.SQL.txt b/doc/wiki/AuthDatabase.SQL.txt new file mode 100644 index 0000000000..6176be7f8b --- /dev/null +++ b/doc/wiki/AuthDatabase.SQL.txt @@ -0,0 +1,243 @@ +SQL +=== + +SQL can be used for both passdb and userdb lookups. If the args parameter in +passdb sql and userdb sql contain the exact same filename, only one SQL +connection is used for both passdb and userdb lookups. + +Contents + + + 1. SQL + + 1. Dovecot configuration + + 2. Password database lookups + + 3. Password verification by SQL server + + 4. User database lookups + + 5. User iteration + + 6. Prefetching + + 7. High availability + + 8. Examples + +Dovecot configuration +--------------------- + +'dovecot.conf': + +---%<------------------------------------------------------------------------- +passdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +---%<------------------------------------------------------------------------- + +Password database lookups +------------------------- + +'password_query' setting contains the SQL query to look up the password. It +must return a field named "password". If you have it by any other name in the +database, you can use the SQL's "AS" keyword ('SELECT pw AS password ..'). You +can use all the normal [Variables.txt] such as '%u' in the SQL +query. + +If all the passwords are in same format, you can use 'default_pass_scheme' to +specify it. Otherwise each password needs to be prefixed with +"{password-scheme}", for example "{plain}plaintext-password". See + for a list of supported password schemes. + +By default MySQL does case-insensitive string comparisons, so you may have a +problem if your users are logging with different as "user", "User" and "uSer". +To fix this, you can make the SQL database return a <"user" field> +[PasswordDatabase.ExtraFields.User.txt], which makes Dovecot modify the +username to the returned value. Note that if you're using separate user and +domain fields, a common problem is that you're returning only the "user" field +from the database.*This drops out the domain from the username*. So make sure +you're returning a concatenated user@domain string or username/domain fields +separately. See the examples below. + +The query can also return other +[PasswordDatabase.ExtraFields.txt] which have special meaning. + +You can't use multiple statements in one query, but you could use a stored +procedure. If you want something like a last login update, use + instead. + +Password verification by SQL server +----------------------------------- + +If the passwords are in some special format in the SQL server that Dovecot +doesn't recognize, it's still possible to use them. Change the SQL query to +return NULL as the password and return the row only if the password matches. +You'll also need to return a non-NULL "nopassword" field. The password is in +'%w' variable. For example: + +---%<------------------------------------------------------------------------- +password_query = SELECT NULL AS password, 'Y' as nopassword, userid AS user \ + FROM users WHERE userid = '%u' AND mysql_pass = password('%w') +---%<------------------------------------------------------------------------- + +This of course makes the verbose logging a bit wrong, since password mismatches +are also logged as "unknown user". + +User database lookups +--------------------- + +Usually your SQL database contains also the userdb information. This means +user's UID, GID and home directory. If you're using only static UID and GID, +and your home directory can be specified with a template, you could use [UserDatabase.Static.txt] instead. It is also a bit faster since it +avoids doing the userdb SQL query. + +'user_query' setting contains the SQL query to look up the userdb information. +The commonly returned userdb fields are uid, gid, home and mail. See + for more information about these and other +fields that can be returned. + +If you're using a single UID and GID for all users, you can set them in +dovecot.conf with: + +---%<------------------------------------------------------------------------- +mail_uid = vmail +mail_gid = vmail +---%<------------------------------------------------------------------------- + +User iteration +-------------- + +Some commands, such as 'doveadm -A' need to get a list of users. With SQL +userdb this is done with 'iterate_query' setting. You can either return + + * "user" field containing either user or user@domain style usernames, or + * "username" and "domain" fields + +Any other fields are ignored. + +Prefetching +----------- + +If you want to avoid doing two SQL queries when logging in with IMAP/POP3, you +can make the 'password_query' return all the necessary userdb fields and use +prefetch userdb to use those fields. If you're using Dovecot's deliver you'll +still need to have the 'user_query' working. + +See for example configuration + +High availability +----------------- + +You can add multiple "host" parameters to the SQL connect string. Dovecot will +do round robin load balancing between them. If one of them goes down, the +others will handle the traffic. + +Examples +-------- + +Note that "user" can have a special meaning in some SQL databases, so we're +using "userid" instead. + +SQL table creation command: + +---%<------------------------------------------------------------------------- +CREATE TABLE users ( + userid VARCHAR(128) NOT NULL, + domain VARCHAR(128) NOT NULL, + password VARCHAR(64) NOT NULL, + home VARCHAR(255) NOT NULL, + uid INTEGER NOT NULL, + gid INTEGER NOT NULL +); +---%<------------------------------------------------------------------------- + +MySQL +----- + +Add to your 'dovecot-sql.conf' file: + +---%<------------------------------------------------------------------------- +driver = mysql +# The mysqld.sock socket may be in different locations in different systems. +# Use "host= ... pass=foo#bar" with double-quotes if your password has '#' +character. +# If you need SSL connection, you can add ssl_ca or ssl_ca_path +# You can also use ssl_cert/ssl_key, ssl_cipher, ssl_verify_server_cert +# or provide option_file and option_group +connect = host=/var/run/mysqld/mysqld.sock dbname=mails user=admin +password=pass +# Alternatively you can connect to localhost as well: +#connect = host=localhost dbname=mails user=admin password=pass # port=3306 + +password_query = SELECT userid AS username, domain, password \ + FROM users WHERE userid = '%n' AND domain = '%d' +user_query = SELECT home, uid, gid FROM users WHERE userid = '%n' AND domain = +'%d' + +# For using doveadm -A: +iterate_query = SELECT userid AS username, domain FROM users +---%<------------------------------------------------------------------------- + +PostgreSQL +---------- + +Add to your 'dovecot-sql.conf' file: + +---%<------------------------------------------------------------------------- +# You can also set up non-password authentication by modifying PostgreSQL's +pg_hba.conf +driver = pgsql +# Use "host= ... pass=foo#bar" if your password has '#' character +connect = host=localhost dbname=mails user=admin password=pass + +password_query = SELECT userid AS username, domain, password \ + FROM users WHERE userid = '%n' AND domain = '%d' +user_query = SELECT home, uid, gid FROM users WHERE userid = '%n' AND domain = +'%d' + +# For using doveadm -A: +iterate_query = SELECT userid AS username, domain FROM users +---%<------------------------------------------------------------------------- + +SQLite +------ + +Add to your 'dovecot-sql.conf' file: + +---%<------------------------------------------------------------------------- +driver = sqlite +connect = /path/to/sqlite.db + +password_query = SELECT userid AS username, domain, password \ + FROM users WHERE userid = '%n' AND domain = '%d' +user_query = SELECT home, uid, gid FROM users WHERE userid = '%n' AND domain = +'%d' + +# For using doveadm -A: +iterate_query = SELECT userid AS username, domain FROM users +---%<------------------------------------------------------------------------- + +PostgreSQL/Horde +---------------- + +I used the following in devocot-sql.conf file to authenticate directly against +the Horde user/password database (with static userdb) on PostgreSQL: + +---%<------------------------------------------------------------------------- +driver = pgsql +connect = host=localhost dbname=horde user=dovecot password= +default_pass_scheme = MD5-CRYPT +password_query = SELECT user_uid AS username, user_pass AS password \ + FROM horde_users WHERE user_uid = '%u' +iterate_query = SELECT user_uid AS username FROM users +---%<------------------------------------------------------------------------- + +Note that you will have to change the password encryption in Horde to +MD5-CRYPT. Also, the example above requires a 'dovecot' user in PostgreSQL with +read (SELECT) privileges on the 'horde_users' table. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.VPopMail.txt b/doc/wiki/AuthDatabase.VPopMail.txt new file mode 100644 index 0000000000..30e71447a3 --- /dev/null +++ b/doc/wiki/AuthDatabase.VPopMail.txt @@ -0,0 +1,167 @@ +VPopMail +======== + +Dovecot supports authenticating against external VPopMail +[http://www.inter7.com/index.php?page=vpopmail] virtual domain manager. Dovecot +must have been configured with '--with-vpopmail' to enable this. You can check +this with 'dovecot --build-options'. See also [HowTo.VMailMgr.txt] +for another similar virtual domain manager. + +If the vpopmail database contains plaintext passwords, it can be used for +non-plaintext authentication as well. + +passdb parameters: + + * cache_key: If set, you can use 'auth_cache' with VPopMail. See + [PasswordDatabase.PAM.txt] for more information about it. + * webmail=IP: If IP address is specified, connections from it are assumed to + come from webmail and VPopMail's webmail usage restrictions apply. + +userdb parameters: + + * cache_key: Like in passdb. + * quota_template=TEMPLATE: Template to specify quota rule, %q in value expands + to Maildir++ quota. + +Example +------- + +---%<------------------------------------------------------------------------- +passdb { + driver = vpopmail + args = webmail=127.0.0.1 +} +userdb { + driver = vpopmail + args = quota_template=quota_rule=*:backend=%q +} +---%<------------------------------------------------------------------------- + +VPopMail + MySQL +================ + +Alternatively, you can use the SQL backend with the following configuration: + +---%<------------------------------------------------------------------------- +driver = mysql +connect = host=/var/run/mysqld/mysqld.sock user=vpopmail +password=YOURPASSWORDHERE dbname=vpopmail + +default_pass_scheme = PLAIN +password_query = SELECT CONCAT(pw_name, '@', pw_domain) AS user, \ + pw_clear_passwd AS password \ + FROM vpopmail \ + WHERE pw_name = '%n' AND pw_domain = '%d' +user_query = SELECT pw_dir as home, \ + 89 AS uid, 89 AS gid \ + FROM vpopmail \ + WHERE pw_name = '%n' AND pw_domain = '%d' +---%<------------------------------------------------------------------------- + +VPopMail + MySQL + pw_gid (disable_imap, disable_webmail) and vlimits support +============================================================================= + +The above example doesn't support vpopmail's abilities to disable access to +services like IMAP, webmail etc. which is controlled by vmoduser and +vmoddomlimits. + +VPopMail uses pw_gid column in the database to store this information. It has a +binary format and every bit of the number stored in this column is responsible +for a different access limit. + +As defined in the vpopmail.h: + +---%<------------------------------------------------------------------------- +/* gid flags */ +#define NO_PASSWD_CHNG 0x01 +#define NO_POP 0x02 +#define NO_WEBMAIL 0x04 +#define NO_IMAP 0x08 +#define BOUNCE_MAIL 0x10 +#define NO_RELAY 0x20 +#define NO_DIALUP 0x40 +#define V_USER0 0x080 +#define V_USER1 0x100 +#define V_USER2 0x200 +#define V_USER3 0x400 +#define NO_SMTP 0x800 +#define QA_ADMIN 0x1000 +#define V_OVERRIDE 0x2000 +---%<------------------------------------------------------------------------- + ++ if vpopmail has been compiled with domain limits (--enable-mysql-limits) +domain wise limits will be defined in a table called "limits" where there are +fields like disable_imap and disable_webmail which values by default are NULL +and 1 if option is set. The use of NULLs in limits table is a bit problematic +because in order to properly handle this situation we're going to have replace +NULL with a numeric value. Of course we're going to join vpopmail table (the +table holding users) with limits table using LEFT JOIN. + +Here's the config taken directly from my install: + +---%<------------------------------------------------------------------------- +# +user_query = SELECT pw_name,89 as uid, 89 as gid, pw_dir as home FROM vpopmail +WHERE pw_name = '%n' AND pw_domain = '%d' +#The below passes all users and doesn't care for vpopmail limits (pw_gid column +or vlimits table) +#password_query = SELECT pw_passwd as password FROM vpopmail WHERE pw_name = +'%n' AND pw_domain = '%d' +# +#A little bit more complicated query to support vpopmail pw_gid flags and +vlimits for domain +#explanation: +#We're using bitwise operations on pw_gid. +#as defined in vpopmail.h: +#- 0x04 - disable webmail flag +#- 0x08 - disable imap flag +# +# !(pw_gid & 8) means - if 8th bit of pw_gid is not set +# !(pw_gid & 4) means - if 4th bit of pw_gid is not set +# (pw_gid & 8192) means - if 14th bit of pw_gid is set (ignore vlimits) +# +# additionally because we're using LEFT JOIN we have to take care of NULLs for +rows that don't return any records from the right table hence the use of +COALESCE() function +# !(pw_gid & 4) (disable webmail flag) is used in conjuntion with +'%r'!="127.0.0.1" which means that it will only apply to connections +originating from hosts other than localhost +# +# So the below query supports pw_gid and vlimits settings for user account and +domains but no domain limit overrides +# +#password_query = select pw_passwd as password FROM vpopmail LEFT JOIN limits +ON vpopmail.pw_domain=limits.domain WHERE pw_name='%n' and pw_domain='%d' and ( +!(pw_gid & 8) and ('%r'!='127.0.0.1' or !(pw_gid & 4)) and ( '%r'!='127.0.0.1' +or COALESCE(disable_webmail,0)!=1) and COALESCE(disable_imap,0)!=1); +# +# The below adds support for vlimits override on user account (vmoduser -o) +# +#logically this means: show password for user=%n at domain=%d when imap on the +account is not disabled and connection is not comming from localhost when +webmail access on the account is not disabled and if imap for the domain is not +disabled and (connection is not comming from localhost when webmail access for +the domain is not disabled) when vlimits are not overriden on the account +# +password_query = select pw_passwd as password FROM vpopmail LEFT JOIN limits ON +vpopmail.pw_domain=limits.domain WHERE pw_name='%n' and pw_domain='%d' and +!(pw_gid & 8) and ('%r'!='127.0.0.1' or !(pw_gid & 4)) and ( ('%r'!='127.0.0.1' +or COALESCE(disable_webmail,0)!=1) and COALESCE(disable_imap,0)!=1 or (pw_gid & +8192) ); +---%<------------------------------------------------------------------------- + +Please be aware that disable_webmail is strictly binded to the IP address hard +coded in the query. In this example webmail connections come from the same +machine that the IMAP server is running on using 127.0.0.1 IP address. So the +webmail client is configured with something like eg. $IMAP_SERVER="127.0.0.1". +If your webmail client is on a different machine you need to change 127.0.0.1 +to your webmail's server IP. + +Also - be aware that dovecot caches SQL results (configurable) so if you're +testing the above config on an account that has logged on succesfully within +the cache timeout period and you changed the settings on it using eg. vmoduser +-i test@example.com account which effectively disabled IMAP access for this +account dovecot can still log this user on because the result of the password +query has been stored in cache and used. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/AuthDatabase.txt b/doc/wiki/AuthDatabase.txt new file mode 100644 index 0000000000..ccfc896f2d --- /dev/null +++ b/doc/wiki/AuthDatabase.txt @@ -0,0 +1,19 @@ +Authentication Databases +======================== + +These databases can be used as both [PasswordDatabase.txt] +and [UserDatabase.txt]: + + * [AuthDatabase.Passwd.txt]: System users (NSS, '/etc/passwd', or + similiar) + * [AuthDatabase.PasswdFile.txt]: '/etc/passwd'-like file in + specified location + * [AuthDatabase.LDAP.txt]: Lightweight Directory Access Protocol + * [AuthDatabase.SQL.txt]: SQL database (PostgreSQL, MySQL, SQLite) + * [AuthDatabase.Dict.txt]: Dict key-value database (Redis, memcached, + etc.) + * [AuthDatabase.VPopMail.txt]: External software used to handle + virtual domains + * [AuthDatabase.Lua.txt]: Lua script for authentication (v2.3.0+) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Caching.txt b/doc/wiki/Authentication.Caching.txt new file mode 100644 index 0000000000..4f696d3ecf --- /dev/null +++ b/doc/wiki/Authentication.Caching.txt @@ -0,0 +1,107 @@ +Caching of authentication results +================================= + +Dovecot supports caching the results of password and user database lookups. The +following rules apply to using the authentication cache: + + * Data is used from the cache if it's not expired ('auth_cache_ttl' setting) + * If authentication fails this time, but it didn't fail last time, it's + assumed that the password has changed and a database lookup is done. + * If a database lookup fails because of some internal error, but data still + exists in the cache (even if expired), the cached data is used. This allows + Dovecot to log in some users even if the database is temporarily down. + +The authentication cache can be flushed by sending a SIGHUP to dovecot-auth. + +Sending SIGUSR2 to dovecot-auth makes it log the number of cache hits and +misses. You can use that information for tuning the cache size and TTL. + +Settings +-------- + +The settings related to the authentication cache are: + + * 'auth_cache_size': Authentication cache size, 0 disables caching (default). + A typical passdb cache entry is around 50 bytes and a typical userdb cache + entry is around 100-200 bytes, depending on the amount of information your + user and password database lookups return. + * 'auth_cache_ttl': Time to live in seconds for cache entries. A cache entry + is no longer used (except for internal failures) if it was created more than + this many seconds ago. Entries are removed from the cache only when the + cache is full and a new entry is to be added. + * 'auth_cache_negative_ttl': If a passdb or userdb lookup didn't return any + data (i.e. the user doesn't exist), it's also stored in the cache as a + negative entry. This setting allows you to give negative entries a different + TTL. 0 disables negative caching completely. + * 'auth_cache_verify_password_with_worker': Password hash verifications are + done by the auth master process by default. Setting this to "yes" moves the + verification to auth-worker processes. This allows distributing the hash + calculations to multiple CPU cores, which could make sense if strong hashes + are used. (v2.2.34+) + +It should be pretty safe to set very high TTLs, because the only field that +usually can change is the user's password, and Dovecot attempts to catch those +cases (see the rules above). + +Cache keys +---------- + +Usually only the username uniquely identifies a user, but in some setups you +may need something more, for example the remote IP address. For SQL and LDAP +lookups Dovecot figures this out automatically by using all the used +<%variables> [Variables.txt] as the cache key. For example if your SQL query +contains %s, %u and %r the cache entry is used only if all of them (service +name, username and remote IP) match for the new lookup. + +With other databases Dovecot doesn't know what could affect caching, so you +have to tell Dovecot manually. The following databases require specifying the +cache key: + + * vpopmail + * pam + * bsdauth + +For example if the PAM lookup depends on username and service, you can use: + +---%<------------------------------------------------------------------------- +passdb { + driver = pam + args = cache_key=%s%u * +} +---%<------------------------------------------------------------------------- + +Password changing scenarios +--------------------------- + +Normal scenario: + + 1. User logs in with password X. The password X is added to cache and login + succeeds. + 2. Password is changed to Y. + 3. User logs in with password Y. The cached password X doesn't match Y, but + since the previous authentication was successful Dovecot does another + backend passdb lookup to see if the password changed. It did, so the + password Y is cached and login succeeds. + +Using old cached password scenario: + + 1. User logs in with password X. The password X is added to cache and login + succeeds. + 2. Password is changed to Y. + 3. User logs in with password X. The cached password X matches X, so login + succeeds. + +Early change scenario: + + 1. User logs in with password X. The password X is added to cache and login + succeeds. + 2. User logs in with password Y. The cached password X doesn't match Y, but + since the previous authentication was successful Dovecot does another + backend passdb lookup to see if the password changed. It didn't, so the + login fails. + 3. Password is changed to Y. + 4. User logs in with password Y. The cached password X doesn't match Y and the + previous authentication was unsuccessful, so Dovecot doesn't bother doing + another backend passdb lookup (until cache TTL expires). The login fails. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Kerberos.txt b/doc/wiki/Authentication.Kerberos.txt new file mode 100644 index 0000000000..d37dc3c892 --- /dev/null +++ b/doc/wiki/Authentication.Kerberos.txt @@ -0,0 +1,229 @@ +Kerberos +======== + +Dovecot supports Kerberos 5 using GSSAPI. The Kerberos authentication mechanism +doesn't require having a [PasswordDatabase.txt], but you do need a + [UserDatabase.txt] so Dovecot can lookup user-specific information, +such as where their mailboxes are stored. With centralized systems, such as +Microsoft Active Directory, LDAP is pretty good choice. + +*Note:* If you only wish to authenticate clients using their Kerberos +/passphrase/ (as opposed to ticket authentication), you will probably want to +use [PasswordDatabase.PAM.txt] authentication with 'pam_krb5.so' instead. + +Pre-requisites +-------------- + +This document assumes that you already have a Kerberos Realm up and functioning +correctly at your site, and that each host in your realm also has a host +/keytab/ installed in the appropriate location. + +For Dovecot, you will need to install the appropriate /service/ keys on your +server. By default, Dovecot will look for these in the host's keytab file, +typically '/etc/krb5.keytab', but you can specify an alternate path using the +'auth_krb5_keytab' configuration entry in dovecot.conf. Anyway specified +keytab file should be readable by user "dovecot" (or whatever user the auth +process is running as). If you wish to provide an IMAP service, you will need +to install a service ticket of the form 'imap/hostname@REALM'. For POP3, you +will need a service ticket of the form 'pop/hostname@REALM'. When using +Dovecot's [Sasl.txt] with MTA, you will need to install service ticket +of the form 'smtp/hostname@REALM'. + +Setting up (samba) +------------------ + +Create symlink for krb5.conf, if you do not have krb5.conf ready + +---%<------------------------------------------------------------------------- +ln -sf /usr/local/samba/private/krb5.conf /etc/krb5.conf +---%<------------------------------------------------------------------------- + +Create dovecot user to your samba instance (choose random password) + +---%<------------------------------------------------------------------------- +$ samba-tool user create dovecot +New Password: +Retype Password: +User 'dovecot' created successfully +---%<------------------------------------------------------------------------- + +Add Service Principal Names (SPNs) and create keytab + +---%<------------------------------------------------------------------------- +$ samba-tool spn add imap/host.domain.com dovecot +$ samba-tool domain exportkeytab --principal imap/host.domain.com +/etc/dovecot/dovecot.keytab +---%<------------------------------------------------------------------------- + +Dovecot needs to be able to read the keytab + +---%<------------------------------------------------------------------------- +$ chgrp dovecot /etc/dovecot/dovecot.keytab +$ chmod g+r /etc/dovecot/dovecot.keytab +---%<------------------------------------------------------------------------- + +Make sure your keytab has entry for imap/host.domain.name@REALM. + +---%<------------------------------------------------------------------------- +$ klist -Kek /etc/dovecot/dovecot.keytab +Keytab name: FILE:/etc/dovecot/dovecot.keytab +KVNO Principal +---- -------------------------------------------------------------------------- + 1 imap/host.domain.name@REALM (des-cbc-crc) + 1 imap/host.domain.name@REALM (des-cbc-md5) + 1 imap/host.domain.name@REALM (arcfour-hmac) +---%<------------------------------------------------------------------------- + +Example dovecot.conf configurations +----------------------------------- + +If you only want to use Kerberos ticket-based authentication: + +---%<------------------------------------------------------------------------- +auth_gssapi_hostname = "$ALL" +auth_mechanisms = gssapi +auth_krb5_keytab = /etc/dovecot/dovecot.keytab + +userdb { + driver = static + args = uid=vmail gid=vmail home=/var/vmail/%u +} +---%<------------------------------------------------------------------------- + +(In this virtual-hosting example, all mail is stored in /var/vmail/$username +with uid and gid set to 'vmail') + +If you also want to support plaintext authentication in addition to +ticket-based authentication, you will need something like: + +---%<------------------------------------------------------------------------- +auth_mechanisms = plain login gssapi +auth_gssapi_hostname = "$ALL" +auth_mechanisms = gssapi +auth_krb5_keytab = /etc/dovecot/dovecot.keytab +passdb { + driver = pam +} +userdb { + driver = passwd +} +---%<------------------------------------------------------------------------- + +(Note that in this example, you will also need to configure PAM to use +whichever authentication backends are appropriate for your site.) + +Enable plaintext authentication to use Kerberos +----------------------------------------------- + +This is needed when some of your clients don't support GSSAPI and you still +want them to authenticate against Kerberos. + +Install pam_krb5 module for PAM, and create '/etc/pam.d/dovecot': + +---%<------------------------------------------------------------------------- +auth sufficient pam_krb5.so +account sufficient pam_krb5.so +---%<------------------------------------------------------------------------- + +Then enable PAM passdb: + +---%<------------------------------------------------------------------------- +passdb { + driver = pam +} +---%<------------------------------------------------------------------------- + +Check '/var/log/auth.log' if you have any problems logging in. The problem +could be that PAM is still trying to use pam_unix.so rather than pam_krb5.so. +Make sure pam_krb5.so is the first module for account or just change +pam_unix.so to sufficient. + +Cross-realm authentication +-------------------------- + +This seems to have all kinds of trouble. Search Dovecot mailing list for +previous threads about it. Some points about it: + + * krb5_kuserok() is used to check if access is allowed. It may try to do the + check by reading ~user/.k5login (good!) or ~dovecot/.k5login (bad!) + * Solaris uses _gss_userok() instead of krb5_kuserok() + * v2.2+ has "k5principals" + [PasswordDatabase.ExtraFields.txt], which is a comma separated list of + usernames that are allowed to log in. If it's set, it bypasses the + krb5_kuserok() check.*NOTE*: for this to work, you need a password database + which supports *credential lookups*. + * With 2.2.13, this excludes LDAP databases using authentication binds + (auth_bind = yes). However, a second LDAP passdb entry without "auth_bind + = yes" may be added for the sole purpose of Kerberos principals mapping. + This passdb doesn't need to return a password attribute (and usually + should'nt). + * UPDATE: code seems to have been reworked in later versions. With 2.2.24, + authentication-bind LDAP databases are able to provide k5principals + lookups if configured with "pass_filter". + +Client support +-------------- + +Mail clients that support Kerberos GSSAPI authentication include: + + * Evolution + * Mozilla Thunderbird + * SeaMonkey + * Mutt + * UW Pine + * Apple Mail + +Test that the server can access the keytab +------------------------------------------ + +This test demonstrates that the server can acquire its private credentials. +First telnet directly to the server + +---%<------------------------------------------------------------------------- +$ telnet localhost 143 +* OK Dovecot ready. +---%<------------------------------------------------------------------------- + +or, if you are using IMAPS then use openssl instead of telnet to connect: + +---%<------------------------------------------------------------------------- +$ openssl s_client -connect localhost:993 +CONNECTED(00000003) +... +* OK Dovecot ready. +---%<------------------------------------------------------------------------- + +Check that GSSAPI appears in the authentication capabilities: + +---%<------------------------------------------------------------------------- +a capability +* CAPABILITY ... AUTH=GSSAPI +---%<------------------------------------------------------------------------- + +Attempt the first round of GSS communication. The '+' indicates that the server +is ready + +---%<------------------------------------------------------------------------- +a authenticate GSSAPI ++ +---%<------------------------------------------------------------------------- + +Abort the telnet session by typing control-] and then 'close' + +---%<------------------------------------------------------------------------- +^] +telnet> close +---%<------------------------------------------------------------------------- + +The test: + + * Setup mutt in /etc/Muttrc to use kerberos using gssapi and imap + configuration + * this is done with 'set imap_authenticators="gssapi"' + * run kinit (type in password for kerb) + * run command mutt + * If you get error No Authentication Method + * run command klist (list all kerberos keys) should show imap/HOSTNAME + * DNS has to function correctly so that kerberos works. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.MasterUsers.txt b/doc/wiki/Authentication.MasterUsers.txt new file mode 100644 index 0000000000..2374cf7c16 --- /dev/null +++ b/doc/wiki/Authentication.MasterUsers.txt @@ -0,0 +1,294 @@ +Master users/passwords +====================== + +It's possible to configure master users who are able to log in as other users. +It's also possible to directly log in as any user using a master password. + +Master users +------------ + +There are two ways for master users to log in as other users: + + 1. Give the login username in the + [Authentication.Mechanisms.txt] authorization ID field. + 2. Specify both the master username and the login username in the same + username field. The usernames are separated by a string configured by the + 'auth_master_user_separator' setting. UW-IMAP uses "*" as the separator, so + that could be a good choice. Using "*" as the separator, the master user + would log in as "login_user*master_user". + +Master users are configured by adding a new [PasswordDatabase.txt] +with 'master=yes' setting. The users in the master passdb cannot log in as +themselves, only as other people. That means they don't need to exist in the + [UserDatabase.txt], because the userdb lookup is done only for the +user they're logging in as. + +You should also add the 'pass=yes' setting to the master passdb if possible. It +means that Dovecot verifies that the login user really exists before allowing +the master user to log in. Without the setting if a nonexistent login username +is given, depending on the configuration, it could either return an internal +login error (the userdb lookup failed) or create a whole new user (with eg. + [UserDatabase.Static.txt]). 'pass=yes' doesn't work with PAM or +LDAP with 'auth_bind=yes', because both of them require knowing the user's +password. + +'pass=yes' is especially useful with a +[PasswordDatabase.CheckPassword.txt] passdb because the script gets both the +login and the master username as environment variables. Other passdbs see only +the login username in '%u'. In the future there will probably be another +setting to make the user verification to be done from userdb. + +If you want master users to be able to log in as themselves, you'll need to +either add the user to the normal passdb or add the passdb to 'dovecot.conf' +twice, with and without 'master=yes'. Note that if the passdbs point to +different locations, the user can have a different password when logging in as +other users than when logging in as himself. This is a good idea since it can +avoid accidentally logging in as someone else. + +Usually it's better to have only a few special master users that are used +*only* to log in as other people. One example could be a special "spam" master +user that trains the users' spam filters by reading the messages from the +user's spam mailbox. + +ACLs +---- + +If plugin is enabled, the Master user is still subject to ACLs just +like any other user, which means that by default the master user has no access +to any mailboxes of the user. The options for handling this are: + + 1. Adding a global for the master user. You can create a "default + ACL", that applies to all mailboxes. See example below. + 2. Set 'plugin { acl_user=%u } ' This preserves the master_user for other + purposes (e.g. %{master_user} variable). + 3. Set 'plugin { master_user=%u } ' This fully hides that master user login is + being used. + +Example configuration +--------------------- + +---%<------------------------------------------------------------------------- +auth_master_user_separator = * +passdb { + driver = passwd-file + args = /etc/dovecot/passwd.masterusers + master = yes + pass = yes +} +passdb { + driver = shadow +} +userdb { + driver = passwd +} +---%<------------------------------------------------------------------------- + +To grant the masteruser access to all Mailboxes, the 'dovecot-acl' file can +contain: + +---%<------------------------------------------------------------------------- +* user=masteruser lr +---%<------------------------------------------------------------------------- + +Where the 'passwd.masterusers' file would contain the master usernames and +passwords: + +---%<------------------------------------------------------------------------- +admin:{SHA1}nU4eI71bcnBGqeO0t9tXvY1u5oQ= +admin2:{SHA1}i+UhJqb95FCnFio2UdWJu1HpV50= +---%<------------------------------------------------------------------------- + +One way to create this master file is to use the htaccess program as follows: + +---%<------------------------------------------------------------------------- +htpasswd -b -c -s passwd.masterusers user password +---%<------------------------------------------------------------------------- + +SQL Example +----------- + +The master passdb doesn't have to be passwd-file, it could be an SQL query as +well: + +---%<------------------------------------------------------------------------- +auth_master_user_separator = * +passdb { + driver = sql + args = /etc/dovecot/dovecot-sql-master.conf.ext + master = yes + pass = yes +} +passdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +userdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +} +---%<------------------------------------------------------------------------- + +'dovecot-sql-master.conf.ext' would contain all the normal connection settings +and a 'password_query': + +---%<------------------------------------------------------------------------- +password_query = SELECT password FROM users WHERE userid = '%u' and master_user += true +---%<------------------------------------------------------------------------- + +Testing +------- + +---%<------------------------------------------------------------------------- +# telnet localhost 143 +* OK Dovecot ready. +1 login loginuser*masteruser masterpass +1 OK Logged in. +---%<------------------------------------------------------------------------- + +If you had any problems, set 'auth_debug=yes' and look at the logs. + +Master passwords +---------------- + +You can configure a passdb which first performs authentication using the master +password. Then it continues to the primary passdb to verify that the user +exists and get other extra fields. + +---%<------------------------------------------------------------------------- +# master password passdb +passdb { + driver = static + default_fields = password=master-password + result_success = continue +} +# primary passdb +passdb { + driver = pam +} +---%<------------------------------------------------------------------------- + +Advanced SQL Examples +--------------------- + +In these example we will create 3 kinds of master users. The first will be +users who can read all email for all domains. The next example will be users +who can read all email for their domain only. The third example will be users +who can read email of domains listed in a separate ownership table. We will use +MySQL and create 2 tables with the following structure. + +---%<------------------------------------------------------------------------- +CREATE TABLE `users` ( + `uid` int(4) NOT NULL AUTO_INCREMENT, + `user_name` varchar(80) NOT NULL, + `domain_name` varchar(80) NOT NULL, + `password` varchar(60) DEFAULT NULL, + `last_login` datetime DEFAULT NULL, + `masteradmin` tinyint(1) NOT NULL DEFAULT '0', + `owns_domain` tinyint(1) NOT NULL DEFAULT '0', + UNIQUE KEY `emaillookup` (`domain_name`,`user_name`), + UNIQUE KEY `uid` (`uid`) +) ENGINE=MyISAM AUTO_INCREMENT=995 DEFAULT CHARSET=latin + +CREATE TABLE `ownership` ( + `login_id` varchar(128) NOT NULL, + `owned_object` varchar(128) NOT NULL, + UNIQUE KEY `login_id_full` (`login_id`,`owned_object`), + KEY `login_id` (`login_id`), + KEY `owned_object` (`owned_object`), + KEY `login_id_index` (`login_id`), + KEY `owned_object_index` (`owned_object`) +) ENGINE=MyISAM DEFAULT CHARSET=latin1 +---%<------------------------------------------------------------------------- + +The dovecot.conf file for all 3 master user configurations will be as follows: + +---%<------------------------------------------------------------------------- +passdb { + driver = sql + args = /etc/dovecot/ownership-sql.conf + master = yes + pass = yes +} + +passdb { + driver = sql + args = /etc/dovecot/domain-owner-sql.conf + master = yes + pass = yes +} + +passdb { + driver = sql + args = /etc/dovecot/masteradmin-sql.conf + master = yes + pass = yes +} +passdb { + args = /etc/dovecot/sql.conf + driver = sql +} +---%<------------------------------------------------------------------------- + +Before we get into the master user tricks, we start with normal email +authentication. The query for that is as follows: + +---%<------------------------------------------------------------------------- +password_query = SELECT user_name, domain_name, password FROM users WHERE +user_name = '%n' AND domain_name = '%d' +---%<------------------------------------------------------------------------- + +In this first example master admin suppose you want to allow a few people to be +master users over all domains. These users will have the "masteradmin" field +set to 1. The query would be: + +---%<------------------------------------------------------------------------- +password_query = SELECT user_name, domain_name, password FROM users WHERE +user_name = '%n' AND domain_name = '%d' AND masteradmin='1' +---%<------------------------------------------------------------------------- + +In the second example suppose you are hosting multiple domains and you want to +allow a few users to become master users of their domain only. + +Your query would be as follows: + +---%<------------------------------------------------------------------------- +password_query = SELECT user_name, domain_name, password FROM users WHERE +user_name = '%n' \ +AND domain_name = '%d' AND owns_domain='1' AND '%d'='%{login_domain}' +---%<------------------------------------------------------------------------- + +This will allow you to log in using the following to read Joe's email if +master@dovecot.org is flagged as the domain_owner. + +---%<------------------------------------------------------------------------- +joe@dovecot.org*master@dovecot.org +---%<------------------------------------------------------------------------- + +In this third example we have a table of owners. There are a list of pairs +between owner email addresses and domains that are owned. That way if a person +controls a lot of domains then they can view all the users in all the domains +they control. The query would be as follows: + +---%<------------------------------------------------------------------------- +password_query = SELECT user_name, domain_name, password FROM users, ownership +WHERE \ +user_name = '%n' AND domain_name = '%d' AND login_id='%u' AND +owned_object='%{login_domain}' +---%<------------------------------------------------------------------------- + +If you really want to get tricky and efficient you can combine all 3 queries +into one giant query that does everything. + +---%<------------------------------------------------------------------------- +password_query = SELECT user_name, domain_name, password FROM users, ownership +WHERE \ +user_name = '%n' AND domain_name = '%d' AND ( \ +(masteradmin='1') OR \ +(owns_domain='1' AND '%d'='%{login_domain}') OR \ +(login_id='%u' and owned_object='%{login_domain}')) \ +group by uid +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Mechanisms.DigestMD5.txt b/doc/wiki/Authentication.Mechanisms.DigestMD5.txt new file mode 100644 index 0000000000..eda617ee6b --- /dev/null +++ b/doc/wiki/Authentication.Mechanisms.DigestMD5.txt @@ -0,0 +1,80 @@ +Digest-MD5 Authentication Mechanism +=================================== + +Digest-MD5 has two things that make it special and which can cause problems: + + * Instead of using user@domain usernames, it supports *realms*. + * User name and realm are part of the MD5 hash that's used for authentication. + +For these and other reasons Digest-MD5 has been obsoleted +[http://tools.ietf.org/html/rfc6331] by SCRAM +[http://tools.ietf.org/html/rfc5802]. + +Realms +------ + +Dovecot v1.0 has problems handling user@domain style usernames with Digest-MD5 +and with passwords stored in plaintext in the password database. + +Instead, user@realm is used. Realms are an integral part of Digest-MD5. You +will need to specify realms you want to advertise to the client in the config +file: + +---%<------------------------------------------------------------------------- +auth_realms = example.com another.example.com foo +---%<------------------------------------------------------------------------- + +The realms don't have to be domains. All listed realms are presented to the +client and it can select to use one of them. Some clients always use the first +realm. Some clients use your domain name, whenever given more than one realm to +choose from. Even if this was NOT one of the choices you provided (KMail, +others?). In both cases the user never sees the advertised realms. + +You can also set an 'auth_default_realm' to use when a client gives an empty +realm. However the client is supposed to use the same realm it tells the server +for the calculation. So when this option helps, it is only for broken clients. + +DIGEST-MD5 scheme +----------------- + +Password must be stored in either plaintext or with DIGEST-MD5 scheme. See +. + +The Digest is the MD5 sum of the string "user:realm:password". So for example +if you want to log in as 'user' with password 'pass' and the realm should be +'example.com' (usually not provided by the user, see above), create the digest +with: + +---%<------------------------------------------------------------------------- +% echo -n "user:example.com:pass" | md5sum +c19c4c6e32f9d8026b26ba77c21fb8eb - +---%<------------------------------------------------------------------------- + +And save it as + +---%<------------------------------------------------------------------------- +user@example.com:c19c4c6e32f9d8026b26ba77c21fb8eb +---%<------------------------------------------------------------------------- + +Note that if you're using DIGEST-MD5 scheme to store the passwords, you can't +change the users' names or realms in any way or the authentication will fail +because the MD5 sums don't match. Also not that this is different from what +Apache does with HTTP AUTH Digest. There it would be +'user:example.com:c19c4c6e32f9d8026b26ba77c21fb8eb' and is created with +'htdigest'. + +Testing +------- + +You can use 'imtest' from Cyrus SASL +[http://asg.web.cmu.edu/sasl/sasl-library.html] library to test an IMAP +connection: + +---%<------------------------------------------------------------------------- +# With realm: +imtest -a user -r example.com +# Without realm: +imtest -a user@example.com +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Mechanisms.NTLM.txt b/doc/wiki/Authentication.Mechanisms.NTLM.txt new file mode 100644 index 0000000000..46b57500bb --- /dev/null +++ b/doc/wiki/Authentication.Mechanisms.NTLM.txt @@ -0,0 +1,34 @@ +NTLM +==== + +There are four authentication submethods inside the NTLM: + + 1. LM: server nonce only, highly vulnerable to MITM and rogue server attacks. + 2. NTLM: different algorithm, almost equally vulnerable as LM today. + 3. NTLM2: server and client nonce, but MITM can force downgrade to NTLM/LM. + 4. NTLMv2: server and client nonce, MITM can't force downgrade. + +NTLM [Authentication.PasswordSchemes.txt] is required for +NTLM, NTLM2 and NTLMv2. + +NTLMv2 can not be negotiated. It must be explicitly enabled on the client side +by setting registry key below to at least 3: + + * Win9x: + 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\LSA\LMCompatibility' + * WinNT: + 'HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\LSA\LMCompatibilityLevel' + +Dovecot's NTLM logic is: + + 1. If we have only LM password scheme, try LM authentication; + 2. If client sends LM response only (some very old clients do it), try LM too; + + 3. If NTLMv2 is guessed (using client response length), try NTLMv2; + 4. If NTLM2 was negotiated, try it; + 5. Otherwise try NTLM. + +For more information about NTLM internals, see http://ubiqx.org/cifs/ and +http://davenport.sourceforge.net/ntlm.html + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Mechanisms.Winbind.txt b/doc/wiki/Authentication.Mechanisms.Winbind.txt new file mode 100644 index 0000000000..256a54d464 --- /dev/null +++ b/doc/wiki/Authentication.Mechanisms.Winbind.txt @@ -0,0 +1,35 @@ +Winbind mechanisms +================== + +Dovecot supports NTLM and GSS-SPNEGO authentication mechanisms using Samba +[http://www.samba.org]'s winbind daemon. It is useful when you need to +authenticate users against a Windows domain (either AD or NT). + +By default NTLM mechanism is handled internally. You can use winbind instead by +setting: + +---%<------------------------------------------------------------------------- +auth_use_winbind = yes +---%<------------------------------------------------------------------------- + +The usernames, returned by winbind, can contain some domain part (either +"DOMAIN\user" or "user@example.com"). Such usernames are always transformed to +the form of "user@domain". To strip domain part (to obtain corresponding local +username, for example), set: + +---%<------------------------------------------------------------------------- +auth_username_format = %n +---%<------------------------------------------------------------------------- + +Dovecot needs path to Samba's 'ntlm_auth' binary to perform the authentication. +You can change the path with: + +---%<------------------------------------------------------------------------- +auth_winbind_helper_path = /usr/bin/ntlm_auth +---%<------------------------------------------------------------------------- + +Dovecot currently does blocking lookups, so if 'ntlm_auth' is slow on +responding (e.g. network problems), Dovecot blocks all other authentication +requests until it's finished. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Mechanisms.txt b/doc/wiki/Authentication.Mechanisms.txt new file mode 100644 index 0000000000..ed3658036f --- /dev/null +++ b/doc/wiki/Authentication.Mechanisms.txt @@ -0,0 +1,82 @@ +Authentication Mechanisms +========================= + +Plaintext authentication +------------------------ + +The simplest authentication mechanism is PLAIN. The client simply sends the +password unencrypted to Dovecot. All clients support the PLAIN mechanism, but +obviously there's the problem that anyone listening on the network can steal +the password. For that reason (and some others) other mechanisms were +implemented. + +Today however many people use [SSL.txt], and there's no problem with +sending unencrypted password inside SSL secured connections. So if you're using +SSL, you probably don't need to bother worrying about anything else than the +PLAIN mechanism. + +Another plaintext mechanism is LOGIN. It's typically used only by SMTP servers +to let Outlook clients perform SMTP authentication. Note that LOGIN mechanism +is not the same as IMAP's LOGIN command. The LOGIN command is internally +handled using PLAIN mechanism. + +Non-plaintext authentication +---------------------------- + +Non-plaintext mechanisms have been designed to be safe to use even without + [SSL.txt] encryption. Because of how they have been designed, they +require access to the plaintext password or their own special hashed version of +it. This means that it's impossible to use non-plaintext mechanisms with +commonly used DES or MD5 password hashes. + +If you want to use more than one non-plaintext mechanism, the passwords must be +stored as plaintext so that Dovecot is able to generate the required special +hashes for all the different mechanisms. If you want to use only one +non-plaintext mechanism, you can store the passwords using the mechanism's own + [Authentication.PasswordSchemes.txt]. + +With [PasswordDatabase.txt] (e.g. PAM) +it's not possible to use non-plaintext mechanisms at all, because they only +support verifying a known plaintext password. + +Dovecot supports the following non-plaintext mechanisms: + + * : Protects the password in transit against eavesdroppers. + Somewhat good support in clients. + * [Authentication.Mechanisms.DigestMD5.txt]: Somewhat stronger + cryptographically than CRAM-MD5, but clients rarely support it. + * SCRAM-SHA-1: Salted Challenge Response Authentication Mechanism (SCRAM) SASL + and GSS-API Mechanisms. Intended as DIGEST-MD5 replacement. + * APOP: This is a POP3-specific authentication. Similar to CRAM-MD5, but + requires storing password in plaintext. + * [Authentication.Mechanisms.NTLM.txt]: Mechanism created by Microsoft + and supported by their clients. + * Optionally supported + [Authentication.Mechanisms.Winbind.txt]. + * GSS-SPNEGO: A wrapper mechanism defined by RFC 4178 + [http://tools.ietf.org/html/rfc4178]. Can be accessed via either GSSAPI or + [Authentication.Mechanisms.Winbind.txt]. + * [Authentication.Kerberos.txt]: Kerberos v5 support. + * RPA: Compuserve RPA authentication mechanism. Similar to DIGEST-MD5, but + client support is rare. + * ANONYMOUS: Support for logging in anonymously. This may be useful if you're + intending to provide publicly accessible IMAP archive. + * OTP and SKEY: One time password mechanisms. + * EXTERNAL: EXTERNAL SASL mechanism. + * OAUTBEARER: OAuth2 bearer authentication + https://tools.ietf.org/html/rfc7628. See + [PasswordDatabase.oauth2.txt] (v2.2.29+) + * XOAUTH2: Google flavor OAUTHBEARER + [https://developers.google.com/gmail/xoauth2_protocol] (v2.2.29+) + +Configuration +------------- + +By default only PLAIN mechanism is enabled. To use more, edit your +'/etc/dovecot/conf.d/10-auth.conf' and set: + +---%<------------------------------------------------------------------------- +auth_mechanisms = plain login cram-md5 +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.MultipleDatabases.txt b/doc/wiki/Authentication.MultipleDatabases.txt new file mode 100644 index 0000000000..8a9869231b --- /dev/null +++ b/doc/wiki/Authentication.MultipleDatabases.txt @@ -0,0 +1,112 @@ +Multiple Authentication Databases +================================= + +Dovecot supports defining multiple authentication databases, so that if the +password doesn't match in the first database, it checks the next one. This can +be useful if you want to easily support having both local system users in +'/etc/passwd' and virtual users. + +Currently the fallback works only with the PLAIN authentication mechanism. + +Often you also want a different mail location for system and virtual users. The +best way to do this would be to always have mails stored below the home +directory ( +[VirtualUsers.Home.txt]): + + * System users' mails: /home/user/Maildir + * Virtual users' mails: /var/vmail/domain/user/Maildir + +This can be done by simply having both system and virtual userdbs return home +directory properly (i.e. virtual users''home=/var/vmail/%d/%n') and then set +'mail_location = maildir:~/Maildir'. + +If it's not possible to have a home directory for virtual users (avoid that if +possible), you can do this by pointing [MailLocation.txt] to +system users' mail location and have the virtual userdb override it by +returning 'mail' [UserDatabase.ExtraFields.txt]. + +Example with home dirs +---------------------- + + * System users' mails: /home/user/Maildir + * Virtual users' mails: /var/vmail/domain/user/Maildir + +dovecot.conf: + +---%<------------------------------------------------------------------------- +# Mail location for both system and virtual users: +mail_location = maildir:~/Maildir + +# try to authenticate using SQL database first +passdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +# fallback to PAM +passdb { + driver = pam +} + +# look up users from SQL first (even if authentication was done using PAM!) +userdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +# if not found, fallback to /etc/passwd +userdb { + driver = passwd +} +---%<------------------------------------------------------------------------- + +dovecot-sql.conf.ext: + +---%<------------------------------------------------------------------------- +password_query = SELECT userid as user, password FROM users WHERE userid = '%u' +user_query = SELECT uid, gid, '/var/vmail/%d/%n' as home FROM users WHERE +userid = '%u' +---%<------------------------------------------------------------------------- + +Example with overriding mail location +------------------------------------- + + * System users' mails: /home/user/Maildir + * Virtual users' mails: /var/vmail/domain/user + +dovecot.conf: + +---%<------------------------------------------------------------------------- +# the default mail location for system users, this will be overridden in userdb +sql. +mail_location = maildir:~/Maildir + +# try to authenticate using SQL database first +passdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +# fallback to PAM +passdb { + driver = pam +} + +# look up users from SQL first (even if authentication was done using PAM!) +userdb { + driver = sql + args = /etc/dovecot/dovecot-sql.conf.ext +} +# if not found, fallback to /etc/passwd +userdb { + driver = passwd +} +---%<------------------------------------------------------------------------- + +dovecot-sql.conf.ext: + +---%<------------------------------------------------------------------------- +password_query = SELECT userid as user, password FROM users WHERE userid = '%u' +# returning mail overrides mail_location setting for SQL users. +user_query = SELECT uid, gid, 'maildir:/var/vmail/%u' as mail FROM users WHERE +userid = '%u' +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.PasswordSchemes.txt b/doc/wiki/Authentication.PasswordSchemes.txt new file mode 100644 index 0000000000..c8516acfc1 --- /dev/null +++ b/doc/wiki/Authentication.PasswordSchemes.txt @@ -0,0 +1,248 @@ +Password Schemes +================ + +Password scheme means the format in which the password is stored in [PasswordDatabase.txt]. The main reason for choosing a scheme other +than *PLAIN* is to prevent someone with access to the password database (such +as a hacker) from stealing users' passwords and using them to access other +services. + +What scheme to use? +------------------- + +You should choose the strongest crypt scheme that's supported by your system. +From strongest to weakest: + + * *ARGON2I/ARGON2ID*: Argon2 [https://en.wikipedia.org/wiki/Argon2] is the + winner of password hashing competition held at July 2015. The password will + start with $argon2i$ or $argon2id$. You can use -r to tune computational + complexity, minimum is 3. ARGON2ID is only available if your libsodium is + recent enough. ARGON2 can require quite a hefty amount of virtual memory, so + we recommend that you set 'service auth { vsz_limit = 2G }' at least, or + more. + * *BLF-CRYPT*: This is the Blowfish crypt (bcrypt) scheme. It is generally + considered to be very secure. The encrypted password will start with '$2y$' + (other generators can generate passwords that have other letters after ' $2 + ', those should work too.) (Note v2.2: bcrypt is not available on most Linux + distributions). Since v2.3.0 this is provided by dovecot. You can tune the + computational cost using -r parameter for doveadm. + * *SHA512-CRYPT*: A strong scheme. The encrypted password will start with + '$6$' + * *SHA256-CRYPT*: A strong scheme. The encrypted password will start with + '$5$' + * *MD5-CRYPT*: A weak but common scheme often used in '/etc/shadow'. The + encrypted password will start with '$1$' + +Note that the above schemes are implemented by the libc's 'crypt()' function. +Using them is especially useful when sharing the same passwords with other +software, because most of them support using 'crypt()' to verify the password. +However, not all libcs (especially older ones) implement all of the above +schemes. See below for other password schemes that are implemented by Dovecot +internally (instead of libc). + +A few articles about why choosing a good password scheme is important: + + * How To Safely Store A Password + [http://codahale.com/how-to-safely-store-a-password/] + * Speed Hashing [http://www.codinghorror.com/blog/2012/04/speed-hashing.html] + +It's not possible to easily switch from one password scheme to another. The +only practical way to do this is to wait until user logs in and change the +password during the login. [HowTo.ConvertPasswordSchemes.txt] +shows one way to do this. + +Generating encrypted passwords +------------------------------ + +You can generate passwords for a particular scheme easily with "doveadm pw" +utility. For example: + +---%<------------------------------------------------------------------------- +doveadm pw +---%<------------------------------------------------------------------------- + +Since v2.3.0+ the scheme defaults to BCRYPT, but you can use -s to override + +---%<------------------------------------------------------------------------- +doveadm pw -s SHA512-CRYPT +---%<------------------------------------------------------------------------- + +To provide password, for scripting purposes, you can use either + +---%<------------------------------------------------------------------------- +doveadm pw -p password +---%<------------------------------------------------------------------------- + +or + +---%<------------------------------------------------------------------------- +printf 'password\npassword\n' | doveadm pw +---%<------------------------------------------------------------------------- + +Default password schemes +------------------------ + +Password databases have a default password scheme: + + * [AuthDatabase.SQL.txt]: See 'default_pass_scheme' setting in + 'dovecot-sql.conf.ext' + * [AuthDatabase.LDAP.txt]: See 'default_pass_scheme' setting in + 'dovecot-ldap.conf.ext' + * [AuthDatabase.PasswdFile.txt]: CRYPT is used by default, but + can be changed with 'scheme' parameter in passdb args. + * [AuthDatabase.Passwd.txt], [PasswordDatabase.Shadow.txt], + [AuthDatabase.VPopMail.txt]: CRYPT is used by default and can't + be changed currently. + * [PasswordDatabase.PAM.txt], [PasswordDatabase.BSDAuth.txt], + [PasswordDatabase.CheckPassword.txt]: Dovecot never even + sees the password with these databases, so Dovecot has nothing to do with + what password scheme is used. + +The password scheme can be overridden for each password by prefixing it with +{SCHEME}, for example:'{PLAIN}pass'. + +Non-plaintext authentication mechanisms +--------------------------------------- + +See for explanation of auth mechanisms. Most +installations use only plaintext mechanisms, so you can skip this section +unless you know you want to use them. + +The problem with non-plaintext auth mechanisms is that the password must be +stored either in plaintext, or using a mechanism-specific scheme that's +incompatible with all other non-plaintext mechanisms. In addition, the +mechanism-specific schemes often offer very little protection. This isn't a +limitation of Dovecot, it's a requirement for the algorithms to even work. + +For example if you're going to use CRAM-MD5 authentication, the password needs +to be stored in either PLAIN or CRAM-MD5 scheme. If you want to allow both +CRAM-MD5 and DIGEST-MD5, the password must be stored in plaintext. + +In future it's possible that Dovecot could support multiple passwords in +different schemes for a single user. + + * *LANMAN*: DES-based encryption. Used sometimes with NTLM mechanism. + * *NTLM*: MD4 sum of the password stored in hex. Used with NTLM mechanism. + * *RPA*: Used with RPA mechanism. + * *CRAM-MD5*: Used with CRAM-MD5 mechanism. + * *DIGEST-MD5*: Used with + [Authentication.Mechanisms.DigestMD5.txt]. The username is included in the + hash, so it's not possible to use the hash for different usernames. + * *SCRAM-SHA-1*: Used with SCRAM-SHA-1 mechanism. (v2.2+) + +Other supported password schemes +-------------------------------- + +Strong schemes and mechanism-specific schemes are listed above. + + * *PLAIN*: Password is in plaintext. + * *CRYPT*: Traditional DES-crypted password in '/etc/passwd' (e.g. "pass" = + 'vpvKh.SaNbR6s') + * Dovecot uses libc's 'crypt()' function, which means that CRYPT is usually + able to recognize MD5-CRYPT and possibly also other password schemes. See + all of the *-CRYPT schemes at the top of this page. + * The traditional DES-crypt scheme only uses the first 8 characters of the + password, the rest are ignored. Other schemes may have other password + length limitations (if they limit the password length at all). + +MD5 based schemes: + + * *PLAIN-MD5*: MD5 sum of the password stored in hex. + * *LDAP-MD5*: MD5 sum of the password stored in base64. + * *SMD5*: Salted MD5 sum of the password stored in base64. + +SHA based schemes (also see below for libc's SHA* support): + + * *SHA*: SHA1 sum of the password stored in base64. + * *SSHA*: Salted SHA1 sum of the password stored in base64. + * *SHA256*: SHA256 sum of the password stored in base64. (v1.1 and later). + * *SSHA256*: Salted SHA256 sum of the password stored in base64. (v1.2 and + later). + * *SHA512*: SHA512 sum of the password stored in base64. (v2.0 and later). + * *SSHA512*: Salted SHA512 sum of the password stored in base64. (v2.0 and + later). + +Other schemes + + * *ARGON2I*: ARGON2i password scheme (v2.3.0+), needs libsodium + * *ARGON2ID*: ARGON2id password scheme (v2.3.0+), needs libsodium + * *PBKDF2*: PKCS5 Password hashing algortihm + +For some schemes (e.g. PLAIN-MD5, SHA) Dovecot is able to detect if the +password hash is base64 or hex encoded, so both can be used.'doveadm pw' anyway +generates the passwords using the encoding mentioned above. + +3rd party password schemes +-------------------------- + +These plugins are provided by community members, we do not provide support or +help with them, please contact the developer(s) directly. Use at your own +discretion. Since v2.3.0 ARGON2 is provided by dovecot itself. + + * *SCRYPT* and *ARGON2*: See + https://github.com/LuckyFellow/dovecot-libsodium-plugin/ + +Encoding +-------- + +The base64 vs. hex encoding that is mentioned above is simply the default +encoding that is used. You can override it for any scheme by adding a ".hex", +".b64" or ".base64" suffix. For example: + + * '{SSHA.b64}986H5cS9JcDYQeJd6wKaITMho4M9CrXM' contains the password encoded + to base64 (just like {SSHA}) + * '{SSHA.HEX}3f5ca6203f8cdaa44d9160575c1ee1d77abcf59ca5f852d1' contains the + password encoded to hex + +This can be especially useful with plaintext passwords to encode characters +that would otherwise be illegal. For example in passwd-file you couldn't use a +":" character in the password without encoding it to base64 or hex. For +example:'{PLAIN}{\}:!"' is the same as '{PLAIN.b64}e1x9OiEiCg=='. + +You can also specify the encoding with doveadm pw. For example: 'doveadm pw -s +plain.b64' + +Salting +------- + +For the SHA512-CRYPT, SHA256-CRYPT and MD5-CRYPT schemes, the salt is stored +before the hash, e.g.:'$6$salt$hash'. For the BLF-CRYPT scheme, bcrypt stores +the salt as part of the hash. + +For most of the other salted password schemes (SMD5, SSHA*) the salt is stored +after the password hash and its length can vary. When hashing the password, +append the salt after the plaintext password, e.g.: SSHA256(pass, salt) = +SHA256(pass + salt) + salt. + +For example with SSHA256 you know that the hash itself is 32 bytes (256 bits/8 +bits per byte). Everything after that 32 bytes is the salt. For example if you +have a password: + +---%<------------------------------------------------------------------------- +{SSHA256}SoR/78T5q0UPFng8UCXWQxOUKhzrJZlwfNtllAupAeUT+kQv +---%<------------------------------------------------------------------------- + +After base64 decoding it you'll see that its length is 36 bytes, so the first +32 bytes are the hash and the following 4 bytes are the salt: + + * length: 'echo SoR/78T5q0UPFng8UCXWQxOUKhzrJZlwfNtllAupAeUT+kQv|base64 -d|wc + -c' -> 36 + * hash: 'echo SoR/78T5q0UPFng8UCXWQxOUKhzrJZlwfNtllAupAeUT+kQv|base64 -d|dd + bs=1 count=32|hexdump -C' -> 4a 84 7f ef c4 f9 ab 45 0f 16 78 3c 50 25 d6 + 43 13 94 2a 1c eb 25 99 70 7c db 65 94 0b a9 01 e5 + * salt: 'echo SoR/78T5q0UPFng8UCXWQxOUKhzrJZlwfNtllAupAeUT+kQv|base64 -d|dd + bs=1 skip=32|hexdump -C' -> 13 fa 44 2f + +Other common hash sizes are: + + * MD5: 16 bytes + * SHA: 20 bytes + * SHA256: 32 bytes + * SHA512: 64 bytes + +The web management gui VBoxAdm [http://developer.gauner.org/vboxadm/] has some +code dealing with creation and verification of salted hashes in Perl. However +not all password schemes provided by dovecotpw are supported. Have a look at +the module VBoxAdm::DovecotPW for more details. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Penalty.txt b/doc/wiki/Authentication.Penalty.txt new file mode 100644 index 0000000000..dcbb3eaf9c --- /dev/null +++ b/doc/wiki/Authentication.Penalty.txt @@ -0,0 +1,45 @@ +Authentication Penalty +====================== + +Dovecot anvil process tracks authentication penalties for different IPs to slow +down brute force login attempts. The algorithm works by: + + * First auth failure reply will be delayed for 2 seconds (this happens even + without auth penalty) + * 'AUTH_PENALTY_INIT_SECS' in 'src/auth/auth-penalty.h' + * The delay will be doubled for 4 -> 8 seconds, and then the upper limit of 15 + seconds is reached. + * 'AUTH_PENALTY_MAX_SECS' and AUTH_PENALTY_MAX_PENALTY in + 'src/auth/auth-penalty.h' + * If the IP is in login_trusted_networks (e.g. webmail), skip any + authentication penalties + * If the username+password combination is the same as one of the last 10 login + attempts, skip increasing authentication penalty. + * 'CHECKSUM_VALUE_PTR_COUNT' in 'src/anvil/penalty.c' + * The idea is that if a user has simply configured the password wrong, it + shouldn't keep increasing the delay. + * The username+password is tracked as the CRC32 of them, so there is a + small possibility of hash collisions + +Problems: + + * It is still possible to do multiple auth lookups from the same IP in + parallel. + * For IPv6 it currently blocks the entire /48 block, which may or may not be + what is wanted. + * PENALTY_IPV6_MASK_BITS in auth-penalty.c + +Authentication penalty tracking can be disabled completely with: + +---%<------------------------------------------------------------------------- +service anvil { + unix_listener anvil-auth-penalty { + mode = 0 + } +} +---%<------------------------------------------------------------------------- + +Also you can have similar functionality with fail2ban +[http://wiki2.dovecot.org/HowTo/Fail2Ban]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.Policy.txt b/doc/wiki/Authentication.Policy.txt new file mode 100644 index 0000000000..b902e4812d --- /dev/null +++ b/doc/wiki/Authentication.Policy.txt @@ -0,0 +1,240 @@ +Authentication policy support +============================= + +Dovecot supports (v2.2.25+) external authentication policy server. This server +can be used to decide whether the connecting user is permitted, tarpitted or +outright rejected. While dovecot can do tarpitting and refusal on its own, this +adds support for making cluster-wide decisions to make it easier to deter and +defeat bruteforce attacks. + +Known issues +------------ + +Prior v2.2.27 request policy was mostly broken. + +Prior v2.2.34, the request attributes contained *orig_username* which was not +correct in all cases, especially with master login. + +Prior v2.3.5.2 / 2.2.36.3, invalid UTF-8 would crash auth server if auth policy +is used. + +Configuration +------------- + +The auth-policy server is a core feature and does not require plugin(s) to +work. To activate this feature, you need to configure it. + + * *auth_policy_server_url*: URL of the policy server, url is appended with + ?command=allow/report unless it ends with '&', at which just + command=allow/report is added. + * /Default/: None (*REQUIRED* configuration) + * Example: 'auth_policy_server_url = http://example.com:4001/' + * *auth_policy_server_api_header*: Header and value to add to request (for API + authentication) + * /Default/: None (No authentication is done) + * See https://en.wikipedia.org/wiki/Basic_access_authentication#Client_side + + * Example: 'Authorization: Basic ' + * *auth_policy_server_timeout_msecs*: Request timeout in milliseconds + * /Default/: 'auth_policy_server_timeout_msecs = 2000' + * *auth_policy_hash_mech*: Hash mechanism to use for password, you can use any + hash mechanism supported by Dovecot (md4,md5,sha1,sha256,sha512) + * /Default/: 'auth_policy_hash_mech = sha256' + * *auth_policy_hash_nonce*: Cluster-wide nonce to add to hash + * /Default/: None (*REQUIRED* configuration) + * Example: 'auth_policy_hash_nonce = localized_random_string' + * *auth_policy_request_attributes*: Request attributes specification (see + attributes section below) + * /Default/: 'auth_policy_request_attributes = login=%{requested_username} + pwhash=%{hashed_password} remote=%{rip} device_id=%{client_id} + protocol=%s' + * *auth_policy_reject_on_fail*: If policy request fails for some reason should + users be rejected + * /Default/: 'auth_policy_reject_on_fail = no' + * *auth_policy_hash_truncate*: How many *bits* to use from password hash. + * /Default/: 'auth_policy_hash_truncate = 12' + * *auth_policy_check_before_auth*: Whether to do policy lookup before + authentication is started + * /Default/: 'auth_policy_check_before_auth = yes' + * *auth_policy_check_after_auth*: Whether to do policy lookup after + authentication is completed + * /Default/: 'auth_policy_check_after_auth = yes' + * *auth_policy_report_after_auth*: Whether to report authentication result + * /Default/: 'auth_policy_report_after_auth = yes' + +Required Minimum Configuration +------------------------------ + +---%<------------------------------------------------------------------------- +auth_policy_server_url = http://example.com:4001/ +auth_policy_hash_nonce = localized_random_string +#auth_policy_server_api_header = Authorization: Basic +#auth_policy_server_timeout_msecs = 2000 +#auth_policy_hash_mech = sha256 +#auth_policy_request_attributes = login=%{requested_username} +pwhash=%{hashed_password} remote=%{rip} device_id=%{client_id} protocol=%s +#auth_policy_reject_on_fail = no +#auth_policy_hash_truncate = 12 +#auth_policy_check_before_auth = yes +#auth_policy_check_after_auth = yes +#auth_policy_report_after_auth = yes +---%<------------------------------------------------------------------------- + +Password hash algorithm +----------------------- + +To generate the hash, you concatenate nonce, login name, nil byte, password and +run it through the hash algorithm once. The hash is truncated when truncation +is set to non-zero. The hash is truncated by first choosing bits from MSB to +byte boundary (rounding up), then right-shifting the remainding bits. + +---%<------------------------------------------------------------------------- +hash = H(nonce||user||'\x00'||password) +bytes = round8(bits*8) +hash = HEX(hash[0:bytes] >> (bytes-bits*8)) +---%<------------------------------------------------------------------------- + +Request attributes +------------------ + +Auth policy server requests are JSON requests. The JSON format can be specified +with auth_policy_request_attributes. The syntax is key=value pairs, and key can +contain one or more / to designate that a JSON object should be made. For +example: + +---%<------------------------------------------------------------------------- +login=%{orig_username} pwhash=%{hashed_password} remote=%{real_rip} +---%<------------------------------------------------------------------------- + +produces + +---%<------------------------------------------------------------------------- +{"login":"john.doe","pwhash":"1234","remote":"127.0.0.1"} +---%<------------------------------------------------------------------------- + +And + +---%<------------------------------------------------------------------------- +login=%{orig_username} pwhash=%{hashed_password} remote=%{real_rip} +attrs/cos=%{userdb:cos} +---%<------------------------------------------------------------------------- + +produces + +---%<------------------------------------------------------------------------- +{"login":"john.doe","pwhash":"1234","remote":"127.0.0.1", +"attrs":{"cos":"premium"}} +---%<------------------------------------------------------------------------- + +Since v2.2.29/v2.3 you can include IMAP ID command result in auth policy +requests, this is achieved with using %{client_id}, which will expand to IMAP +ID command arglist. You must set + +---%<------------------------------------------------------------------------- +imap_id_retain=yes +---%<------------------------------------------------------------------------- + +for this to work. + +Default values for auth_policy_request_attributes +------------------------------------------------- + +Since v2.2.25 + +---%<------------------------------------------------------------------------- +login=%{orig_username} pwhash=%{hashed_password} remote=%{real_rip} +---%<------------------------------------------------------------------------- + +Since v2.2.30 + +---%<------------------------------------------------------------------------- +login=%{orig_username} pwhash=%{hashed_password} remote=%{real_rip} +device_id=%{client_id} protocol=%s +---%<------------------------------------------------------------------------- + +Since v2.2.34 + +---%<------------------------------------------------------------------------- +login=%{requested_username} pwhash=%{hashed_password} remote=%{rip} +device_id=%{client_id} protocol=%s +---%<------------------------------------------------------------------------- + +Since v2.3.0 (note that 2.2 and 2.3 branches have been developed in parallel +for a while) + +---%<------------------------------------------------------------------------- +login=%{orig_username} pwhash=%{hashed_password} remote=%{real_rip} +device_id=%{client_id} protocol=%s +---%<------------------------------------------------------------------------- + +Since v2.3.1 + +---%<------------------------------------------------------------------------- +login=%{requested_username} pwhash=%{hashed_password} remote=%{rip} +device_id=%{client_id} protocol=%s +---%<------------------------------------------------------------------------- + +Since v2.3.2 the request contains 'tls' attribute when TLS has been used. TLS +is also detected if it's offloaded by a load balancer that can provide this +information using HAProxy v2 protocol to dovecot. + +Response +-------- + +---%<------------------------------------------------------------------------- +{"status":-1,"msg":"go away"} +---%<------------------------------------------------------------------------- + +'status' values: see below + +Mode of operation +----------------- + +Auth policy check: Authentication ''before'' userdb/passdb +---------------------------------------------------------- + +First query is done *before* password and user databases are consulted. This +means that any userdb/passdb attributes are left empty. + +The command used here is 'allow' and will appear on the URL as command=allow. + +'status' result values: + + * *-1*: Reject + * *0*: Accept + * *(Any other positive value)*: Tarpit for this number of seconds. + +Auth policy check: Authentication ''after'' successful userdb/passdb lookup +--------------------------------------------------------------------------- + +Second lookup is done *after* authentication succeeds. + +The command used here is 'allow' and will appear on the URL as command=allow. + +'status' result values: + + * *-1*: Authentication fail + * *>= 0*: Authentication succeed + +Auth policy check: Reporting after authentication succeeds +---------------------------------------------------------- + +A report request is sent at end of authentication. + +The command used here is 'report' and will appear on the URL as command=report. + +The JSON request is sent with two additional attributes: + + * *success*: boolean true/false depending on whether the overall + authentication succeeded + * *policy_reject*: boolean true/false whether the failure was due to policy + server + +'status' result value is ignored. + +External Auth Policy Servers +---------------------------- + + * http://oxpedia.org/wiki/index.php?title=AppSuite:Dovecot_Antiabuse_Shield + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.RestrictAccess.txt b/doc/wiki/Authentication.RestrictAccess.txt new file mode 100644 index 0000000000..b24be44f94 --- /dev/null +++ b/doc/wiki/Authentication.RestrictAccess.txt @@ -0,0 +1,85 @@ +Restricting Access +================== + +Restricting IMAP/POP3 access +---------------------------- + +Below examples show how you can give POP3 access to everyone, but IMAP access +only for some people. The exact solution you want depends on what passdb you +use. The solutions can also be modified for other types of IMAP/POP3/SMTP/etc. +access checks. + +PAM +--- + +Set PAM service name to '%s', ie.: + +---%<------------------------------------------------------------------------- +passdb { + driver = pam + args = %s +} +---%<------------------------------------------------------------------------- + +That way PAM uses '/etc/pam.d/imap' for IMAP, and '/etc/pam.d/pop3' for POP3. + +In '/etc/pam.d/imap' you could then use eg. the pam_listfile.so module: + +---%<------------------------------------------------------------------------- +# allow IMAP access only for users in /etc/imapusers file +auth required pam_listfile.so item=user sense=allow +file=/etc/imapusers onerr=fail +---%<------------------------------------------------------------------------- + +SQL +--- + +You can use the '%s' variable which expands to 'imap' or 'pop3' in +'password_query', eg: + +---%<------------------------------------------------------------------------- +password_query = SELECT password FROM users WHERE userid = '%u' and not +(imap_allowed = false and '%s' = 'imap') +---%<------------------------------------------------------------------------- + +LDAP +---- + +Just like with SQL, you can use '%s' in pass_filter, eg.: + +---%<------------------------------------------------------------------------- +pass_filter = (&(objectClass=posixAccount)(uid=%u)(service=%s)) +---%<------------------------------------------------------------------------- + +That would require setting both service=pop3 and service=imap attributes to the +user objects. + +passwd-file +----------- + +You can create a deny passwd-file based on the service: + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + args = /etc/dovecot/deny.%s + deny = yes +} +---%<------------------------------------------------------------------------- + +This makes Dovecot look for '/etc/dovecot/deny.imap' and +'/etc/dovecot/deny.pop3' files. If the user exists in it, the access is denied. +The files don't need to have anything else than one username per line. + +Note that this deny passdb must be before other passdbs. It also means that it +can be used with any other passdb, not just with passwd-file passdbs. + +Restricting IP Access +--------------------- + +It's possible to allow a user to authenticate only from a specific IP or +network. This is especially useful for master users. This can be done by +returning [PasswordDatabase.ExtraFields.AllowNets.txt] extra field +in passdb. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Authentication.txt b/doc/wiki/Authentication.txt new file mode 100644 index 0000000000..d72c5d765e --- /dev/null +++ b/doc/wiki/Authentication.txt @@ -0,0 +1,37 @@ +Authentication +============== + +Authentication is split into four parts: + + 1. [Authentication.Mechanisms.txt] + 2. [Authentication.PasswordSchemes.txt] + 3. [PasswordDatabase.txt] + 4. [UserDatabase.txt] + +See also [Authentication.Penalty.txt] handling for IP +addresses. See also [Authentication.Policy.txt] +for making policy based decisions. + +Authentication mechanisms vs. password schemes +---------------------------------------------- + +Authentication mechanisms and password schemes are often confused, because they +have somewhat similar values. For example there is a PLAIN auth mechanism and +PLAIN password scheme. But they mean completely different things. + + * *Authentication mechanism is a client/server protocol*. It's about how the + client and server talk to each others in order to perform the + authentication. Most people use only PLAIN authentication, which basically + means that the user and password are sent without any kind of encryption to + the server. SSL/TLS can then be used to provide the encryption to make PLAIN + authentication secure. + * *Password scheme is about how the password is hashed in your password + database*. If you use a PLAIN scheme, your passwords are stored in cleartext + without any hashing in the password database. A popular password scheme + MD5-CRYPT (also commonly used in '/etc/shadow') where passwords looks like + "$1$oDMXOrCA$plmv4yuMdGhL9xekM.q.I/". + * Plaintext authentication mechanisms work with ALL password schemes. + * Non-plaintext authentication mechanisms require either PLAIN password scheme + or a mechanism-specific password scheme. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/BasicConfiguration.txt b/doc/wiki/BasicConfiguration.txt new file mode 100644 index 0000000000..a03590b6e7 --- /dev/null +++ b/doc/wiki/BasicConfiguration.txt @@ -0,0 +1,157 @@ +Basic Configuration +=================== + +This page tells you the basics that you'll need to get a working Dovecot +installation. + +Find Dovecot configuration file location using: + +---%<------------------------------------------------------------------------- +doveconf -n | head -n1 +---%<------------------------------------------------------------------------- + +Your configuration file doesn't exist if you installed Dovecot from sources. +The config directory should contain a 'README' file pointing to an example +configuration, which you can use as your basic configuration. For example: + +---%<------------------------------------------------------------------------- +cp -r /usr/share/doc/dovecot/example-config/* /etc/dovecot/ +---%<------------------------------------------------------------------------- + +The default configuration starts from 'dovecot.conf', which contains an +'!include conf.d/*.conf' statement to read the rest of the configuration. This +split of configuration files isn't a requirement to use, and it doesn't really +matter which .conf file you add any particular setting, just as long as it +isn't overridden in another file. You can verify with 'doveconf -n' that +everything looks as you intended. + +Authentication +-------------- + +By default Dovecot is set up to use system user authentication. If you're +planning on using system users, you can simply skip this section and read +[PasswordDatabase.PAM.txt] (or [PasswordDatabase.BSDAuth.txt]) for +configuring it. + +If you're planning on using virtual users, it's easier to first create a simple +passwd-like file to make sure that the authentication will work. Later when you +know Dovecot is working, you can do it differently (see ). + +Run as your own non-root user: + +---%<------------------------------------------------------------------------- +echo "$USER:{PLAIN}password:$UID:$GID::$HOME" > users +sudo mv users /etc/dovecot/ + +# If SELinux is enabled: +restorecon -v /etc/dovecot/users +---%<------------------------------------------------------------------------- + +You can (and should) replace the "password" with whatever password you wish to +use, but don't use any important password here as we'll be logging in with +insecure plaintext authentication until is configured. + +(Remark: $GID is not set per default on systems, replace by 'id +-g') + +If you used the example configuration files, switch to passwd-file by modifying +'conf.d/10-auth.conf': + +---%<------------------------------------------------------------------------- +# Add '#' to comment out the system user login for now: + +# Remove '#' to use passwd-file: +!include auth-passwdfile.conf.ext +---%<------------------------------------------------------------------------- + +In 'conf.d/auth-passwdfile.conf.ext' you should have: + +---%<------------------------------------------------------------------------- +passdb { + driver = passwd-file + args = scheme=CRYPT username_format=%u /etc/dovecot/users +} +userdb { + driver = passwd-file + args = username_format=%u /etc/dovecot/users +} +---%<------------------------------------------------------------------------- + +Verify with 'doveconf -n passdb userdb' that the output looks like above (and +there are no other passdbs or userdbs). + +Plaintext Authentication +------------------------ + +To allow any Authentication without SSL, disable SSL in the +'conf.d/10-ssl.conf' file. This has to be done because Dovecot (now) uses SSL +as default. You probably want to switch this back to "yes" or other options +afterward. + +---%<------------------------------------------------------------------------- +ssl = no +---%<------------------------------------------------------------------------- + +Until SSL is configured, allow plaintext authentication in the +'conf.d/10-auth.conf' file. You probably want to switch this back to "yes" +afterward. + +---%<------------------------------------------------------------------------- +disable_plaintext_auth = no +---%<------------------------------------------------------------------------- + +If you didn't use the temporary passwd-file created above, don't do this if you +don't want your password to be sent in clear to network. Instead get SSL +configuration working and connect to Dovecot only using SSL. + +Mail Location +------------- + +Set the 'mail_location' in 'conf.d/10-mail.conf' as determined by the +instructions in . + +mbox +---- + +If you're using mboxes, it's important to have locking configuration correct. +See for more information. + +If you're using '/var/mail/' or '/var/spool/mail/' directory for INBOXes, you +may need to give Dovecot additional permissions so it can create dotlock files +there. A failure to do so will result in errors like these: + +---%<------------------------------------------------------------------------- +open(/var/mail/.temp.host.1234.abcdefg) failed: Permission denied +file_lock_dotlock() failed with mbox file /var/mail/user: Permission denied +---%<------------------------------------------------------------------------- + +From here on I'm assuming the INBOX directory is '/var/mail'. + +First check what the permissions of '/var/mail' are: + +---%<------------------------------------------------------------------------- +# ls -ld /var/mail +drwxrwxrwt 2 root mail 47 2006-01-07 20:44 /var/mail/ +---%<------------------------------------------------------------------------- + +In this case everyone has write access there and the directory is marked +sticky. This allows Dovecot to create the dotlock files, so you don't need to +do anything. + +---%<------------------------------------------------------------------------- +# ls -ld /var/mail +drwxrwxr-- 2 root mail 47 2006-01-07 20:44 /var/mail/ +---%<------------------------------------------------------------------------- + +In this case only the root and the 'mail' group has write permission to the +directory. You'll need to give Dovecot's mail processes ability to use this +group by changing 'conf.d/10-mail.conf': + +---%<------------------------------------------------------------------------- +mail_privileged_group = mail +---%<------------------------------------------------------------------------- + +Note: Specifying the privileged user must be done as shown. Simply adding +'dovecot' user to the 'mail' group does /*not*/ grant write permission. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Chrooting.txt b/doc/wiki/Chrooting.txt new file mode 100644 index 0000000000..8284dc303f --- /dev/null +++ b/doc/wiki/Chrooting.txt @@ -0,0 +1,65 @@ +Chrooting +========= + +Traditionally chrooting has been done to run the whole server within a single +chroot. This is also possible with Dovecot, but it requires manually setting up +the chroot and it can be a bit tricky. Dovecot however supports internally +running different parts of it in different chroots: + + * Login processes (imap-login, pop3-login) are chrooted by default into an + empty non-writable directory. + * Authentication process (dovecot-auth) can be chrooted by setting + 'chroot=' inside 'service auth' and/or 'service auth-worker' sections. + This could be a good idea to change if you're not using a passdb or userdb + that needs to access files outside of the chroot. Also make sure not to run + the auth process as root then. + * Mail processes (imap, pop3) can be made to chroot in different ways. See + below. + +Security problems +----------------- + +If chrooting is used incorrectly, it allows local users to gain root +privileges. This is possible by hardlinking setuid binaries inside the chroot +jail and tricking them. There are at least two possibilities: + + 1. Hardlink '/bin/su' inside the chroot and create your own + '/etc/passwd'. Then simply run 'su root'. + 2. Create your own '/lib/libc.so' and run any setuid binary. + +Of course both of these require that the setuid binary can be run inside the +chroot. This isn't possible by default. Either user would have to find a +security hole from Dovecot, or the administrator would have had to set up +something special that allows running binaries. + +In any case it's a good idea not to allow users to hardlink setuid binaries +inside the chroots. The safest way to do this is to mount the filesystem with +"nosuid" option. + +Mail process chrooting +---------------------- + +Due to the potential security problem described above, Dovecot won't chroot +mail processes to directories which aren't listed in 'valid_chroot_dirs' +setting. For example if your users may be chrooting under '/var/mail//' +and '/home//', use: + +---%<------------------------------------------------------------------------- +valid_chroot_dirs = /var/mail:/home +---%<------------------------------------------------------------------------- + +You can chroot all users globally into the same directory by using +'mail_chroot' setting. For example: + +---%<------------------------------------------------------------------------- +mail_chroot = /home +---%<------------------------------------------------------------------------- + +You can also make userdb return a chroot. There are two ways to do that: + + 1. Make userdb return 'chroot=' field. + 2. Insert "/./" inside the returned home directory, eg.: 'home=/home/./user' + to chroot into '/home', or 'home=/home/user/./' to chroot into + '/home/user'. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Clients.NegativeUIDs.txt b/doc/wiki/Clients.NegativeUIDs.txt new file mode 100644 index 0000000000..28fc164945 --- /dev/null +++ b/doc/wiki/Clients.NegativeUIDs.txt @@ -0,0 +1,53 @@ +Negative UIDs +============= + +---%<------------------------------------------------------------------------- +Invalid messageset: 1181461470:-1181461446. +---%<------------------------------------------------------------------------- + +IMAP uses unsigned 32bit integers for unique message identifiers. Unfortunately +a lot of IMAP clients use 32bit signed integers, which means that if the UIDs +go higher than 2147483647, they'll wrap to negative integers. This causes +errors such as above. + +However normally the UIDs should never go that high, so it's possible to avoid +this problem. + +mbox +---- + +Earlier Dovecot versions had bugs which could cause X-UID: headers in incoming +messages to grow the UIDs too high. Some spam messages especially contained +these intentionally broken X-UID: headers. + +With newer Dovecot versions these broken X-UID: headers aren't practically ever +used. It happens only if the mail has a valid X-IMAPbase: header, X-UID: header +and the mail is written to an empty mbox file. Note that this can happen only +new mboxes, because expunging all messages in a mailbox causes Dovecot to +create a metadata message at the beginning of the mbox file. + +In any case it's still a good idea to filter out X-UID: and other metadata +headers in your MDA. [LDA.txt] does this internally. See + for a list of headers to filter out. + +Fixing +------ + +Fixing is done by letting Dovecot update UIDVALIDITY value and recreate the +UIDs beginning from one. This means that client's local cache will be +invalidated and the client will be required to download all the messages again. + +mbox +---- + +Delete Dovecot's index files (eg. '.imap/INBOX/') and X-IMAP: and X-IMAPbase: +headers from the mbox file. + +Maildir +------- + +This should really never be a problem with Maildir. If however you have managed +to cause it somehow (by receiving 2 billion mails?), you can recreate the UIDs +by deleting 'dovecot-uidlist' file. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Clients.txt b/doc/wiki/Clients.txt new file mode 100644 index 0000000000..f79d377119 --- /dev/null +++ b/doc/wiki/Clients.txt @@ -0,0 +1,186 @@ +Client issues and configuration +=============================== + +It seems to be quite difficult to implement a working IMAP client. Best +Practices for Implementing an IMAP Client +[http://www.imapwiki.org/ClientImplementation] tries to help you with it. + +Contents + + + 1. Client issues and configuration + + 1. Apple Mail.app + + 2. Outlook + + 3. Outlook Express 6 + + 4. Netscape Mail + + 5. Evolution + + 6. Mulberry + + 7. Claws-mail + + 8. Thunderbird + + 9. Mutt + + 10. Pine + + 11. SquirrelMail + + 12. Horde IMP + + 13. RoundCube Webmail + + 14. @Mail Webmail + + 15. RainLoop Webmail + +Apple Mail.app +-------------- + +On Mac OS X Leopard 10.5 Mail.app appears to support subscribe/unsubscribe by +right clicking on a mailbox, selecting 'Get Account Info' and selecting +'Subscription List' from tabs. This however doesn't really work with any IMAP +server. + +Apple Mail 3.6 (that comes with OS X 10.5 Leopard) supports +subscribing/unsubscribing to folders in the public namespace. + +Outlook +------- + + * You should enable 'outlook-no-nuls' workaround with POP3. + * Outlook 2003 has problems with older Dovecot's default POP3 UIDL format, + which causes it to download the same mails over and over again if "leave + mails to server" option is enabled. See 'pop3_uidl_format' setting. + * Outlook might not hide or purge deleted items by default. Microsoft has a + how-to that shows how to fix this + [http://office.microsoft.com/en-us/outlook/HP100804201033.aspx] (Outlook + 2007, not Outlook 2003). + * If some Outlook users don't see new or sent mails in the appropriate folders + after a migration from UW IMAPd even if they are visible in other clients + (e.g. Roundcube, Thunderbird, or on the disk itself), and you get the error + message "BAD Error in IMAP command UID: Invalid UID messageset" in the log + or rawlog: It helps to remove the problematic IMAP account completely from + Outlook and recreating it again there. It speaks a different IMAP + afterwards, so there are reasons to believe it caches the details of some + server on the first connect and doesn't refresh them even if you change the + server's hostname in the account settings. + +Outlook Express 6 +----------------- + + * Using "Headers only" synchronization is buggy and can cause "Message is no + longer available on this server" error when opening a mail. This isn't + Dovecot specific problem, and I'm not aware of any possible workarounds at + the moment for this in server side. + * You should enable 'delay-newmail' workarounds for IMAP. + * You should enable 'outlook-no-nuls' and 'oe-ns-eoh' workarounds for POP3. + +Netscape Mail +------------- + +I'm not actually sure what version exactly this refers to. + + * You should enable 'oe-ns-eoh' workaround for POP3. + +Evolution +--------- + + * Some versions don't support creating subfolders with mbox format. Evolution + in Ubuntu Gutsy, 2.12.0-0ubuntu5, does support creating subfolders, at least + when the parent folder is empty. + +Mulberry +-------- + +Seems to be OK. + +Claws-mail +---------- + +Everything works perfectly with Dovecot. + +Thunderbird +----------- + + * If you're using [MailboxFormat.mbox.txt], + [MailboxFormat.dbox.txt] or [MailLocation.Maildir.txt] with + ':LAYOUT=fs' , + * You should enable 'tb-extra-mailbox-sep' workaround for IMAP. Bug report + [https://bugzilla.mozilla.org/show_bug.cgi?id=29926]. + * If you're using [MailboxFormat.mbox.txt]: + * If you are not using a technique to allow folders that contain both + sub-folders and messages (e.g. see ) then you will + have to disable "Server supports folders that contain sub-folders and + messages" setting from Thunderbird.Enhancement request + [https://bugzilla.mozilla.org/show_bug.cgi?id=284933]. + * Versions of Thunderbird from at least 17 (possibly earlier) up to 24.0 + display incorrect new mail counts in the New Mail notification box. This is + due to a bug in Thunderbird's handling of the CONDSTORE extension. See Bug + Report [https://bugzilla.mozilla.org/show_bug.cgi?id=885220] for details and + a client-side workaround. + +Mutt +---- + + * New mutt versions supporting IDLE command will hang with Dovecot versions + earlier than v1.0beta3. Upgrade Dovecot or disable IDLE by setting + imap_idle=no in .muttrc. + * [mutt.txt] + +Pine +---- + +Seems to be OK. + +SquirrelMail +------------ + + * Configuration asks IMAP server name for some workarounds. There has been a + Dovecot option since 1.4.6 and 1.5.1. For olderSquirrelMail versions, select + the "other" option and remove the default INBOX-prefix. + +Horde IMP +--------- + +Dovecot namespace detection works automatically with any recent version of IMP +(4.1+). + +Quota support is now integrated into the 'imap' driver (as of horde-groupware +V1.2), an example config of /imp/config/servers.php is: + +---%<------------------------------------------------------------------------- +$servers['imap'] = array( + 'name' => 'IMAP Server', + 'server' => 'localhost', + 'hordeauth' => false, + 'protocol' => 'imap/notls', + 'port' => 143, + 'quota' => array('driver'=>'imap'), +); +---%<------------------------------------------------------------------------- + +RoundCube Webmail +----------------- + +Works fine. + +@Mail Webmail +------------- + +Uses the namespace returned via Dovecot, full support via IMAP/POP3 using @Mail +[http://atmail.com/]. Can also read mailbox quota via the getquotaroot IMAP +command. + +RainLoop Webmail +---------------- + +Works fine. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/CompilingSource.txt b/doc/wiki/CompilingSource.txt new file mode 100644 index 0000000000..49555a3333 --- /dev/null +++ b/doc/wiki/CompilingSource.txt @@ -0,0 +1,369 @@ +Contents + + + 1. Compiling Dovecot From Sources + + 2. Compiling Dovecot From Git + + 3. Compiling Dovecot with rpmbuild (Mandriva, RedHat, etc.) + + 4. SSL/TLS Support + + 1. Solaris and OpenSSL problems + + 5. Notify method + + 1. Linux + + 6. Optional Configure Options + + 1. SQL Driver Options + + 2. Authentication Backend Options + + 7. Dynamic IMAP and POP3 Modules + +Compiling Dovecot From Sources +============================== + +For most people it is enough to do: + +---%<------------------------------------------------------------------------- +./configure +make +sudo make install +---%<------------------------------------------------------------------------- + +That installs Dovecot under the '/usr/local' directory. The configuration file +is in '/usr/local/etc/dovecot.conf'. Logging goes to syslog's mail facility by +default, which typically goes to '/var/log/mail.log' or something similar. If +you are in a hurry, you can then jump to . + + If you have installed some libraries into locations which require special +include or library paths, you can pass them in the 'CPPFLAGS' and 'LDFLAGS' +environment variables. For example: + +---%<------------------------------------------------------------------------- +CPPFLAGS="-I/opt/openssl/include" LDFLAGS="-L/opt/openssl/lib" ./configure +---%<------------------------------------------------------------------------- + +You'll need to create two users for Dovecot's internal use: + + * *dovenull*: Used by untrusted imap-login and pop3-login processes + (default_login_user setting). + * *dovecot*: Used by slightly more trusted Dovecot processes + (default_internal_user setting). + +Both of them should also have their own *dovenull* and *dovecot* groups. See + for more information. + +Compiling Dovecot From Git +========================== + +If you got Dovecot from Git, for instance with + +---%<------------------------------------------------------------------------- +git clone https://github.com/dovecot/core.git dovecot +---%<------------------------------------------------------------------------- + +you will first need to run './autogen.sh' to generate the 'configure' script +and some other files. This requires that you have the following +software/packages installed: + + * 'autoconf' + * 'automake' + * 'libtool' + * 'pkg-config' + * 'gettext' + * 'pandoc' (not strictly required - you can avoid it by using: 'PANDOC=false + ./configure') + * GNU make. + +It is advisable to add '--enable-maintainer-mode' to the 'configure' script. +Thus: + +---%<------------------------------------------------------------------------- +./autogen.sh +./configure --enable-maintainer-mode +make +sudo make install +---%<------------------------------------------------------------------------- + +For later updates, you can use: + +---%<------------------------------------------------------------------------- +git pull +make +sudo make install +---%<------------------------------------------------------------------------- + +Compiling Dovecot with rpmbuild (Mandriva, RedHat, etc.) +======================================================== + +Fetch the source rpm from ftp://ftp.surfnet.nl/ +[ftp://ftp.surfnet.nl/vol/5/mandrakelinux/official/2007.0/SRPMS/contrib/release/] +or any other mirror. At the moment of this writing dovecot-10.rc26.src.rpm can +be found in the cooker subtree. If the current release is newer; updating the +source rpm is not difficult. Unpack the source rpm with 'rpm -ivh +dovecot-10.rc26.src.rpm' to a build environment (/usr/src/rpm...) Copy the +newer tarball from the dovecot site to the SOURCES directory of the build +environment. Change the dovecot.spec file in the SPECS directory to reflect the +new release and the new name of the tarball. The maintainer seems to work with +a bz2 tarball; a tar.gz tarball makes no difference Issue a rpmbuild -ba +dovecot.spec. The resulting rpm will be placed in RPMS/i586 Install with rpm or +urpmi. + +---%<------------------------------------------------------------------------- +rpm -ivh dovecot-1.0.rc26.src.rpm +cd /usr/src/rpm +mv ~/downloads/dovecot-1.0.rc28.tar.gz ./SOURCES +cd SPECS +vi dovecot.spec +...edit release and tarball name. Change default options if needed... +rpmbuild -ba dovecot.spec +cd ../RPMS/i586 +urpmi ./dovecot-1.0.rc28-1mdv2007.0.i586.rpm +---%<------------------------------------------------------------------------- + +During this process missing prerequisites may be detected. Install them and +rerun the build process. The spec file also need updating for the new add-ons +(idxview and logview). + +SSL/TLS Support +=============== + +Dovecot was initially built to support both OpenSSL and GNUTLS. GNUTLS has +however had some problems and nowadays it does not work any more. Patches to +fix it are welcome. + +OpenSSL is used by default now, and it should be automatically detected. If it +is not, you are missing some header files or libraries, or they are just in a +non-standard path. Make sure you have the 'openssl-dev' or a similar package +installed, and if it is not in the standard location, set 'CPPFLAGS' and +'LDFLAGS' as shown in [CompilingSource.txt] + +By default the SSL certificate is read from '/etc/ssl/certs/dovecot.pem' and +the private key from '/etc/ssl/private/dovecot.pem'. The '/etc/ssl' directory +can be changed using the '--with-ssldir=DIR' configure option. Both can of +course be overridden from the configuration file. + +Solaris and OpenSSL problems +---------------------------- + +Solaris 10 includes a bundled OpenSSL that does not function correctly with +Dovecot when attempting to use SSL/TLS with the default dovecot config. This is +because the default setting of ssl_cipher_list in 'dovecot.conf' is HIGH:!ALL; +due to import restrictions in some countries (now apparently relaxed) the high +level routines are part of the unbundled SUNWcry package and are not available +if you don't have this package installed. This confuses the client as dovecot +announces support for high level crypto and then cannot deliver. In any case, +to resolve this you can alternatively (in decreasing order of simplicity): + + 1. Set 'ssl_cipher_list = MEDIUM:!LOW' in 'dovecot.conf' + 2. Find and install the missing SUNWcry package. + 3. Provide an alternate version of the openssl libraries that doesn't have the + high grade routines removed for your protection (sigh). The bundled version + of OpenSSL cannot be removed. Installing a newer OpenSSL from source or + package (for instance, from http://sunfreeware.com/) will enable Dovecot to + work correctly as long as you link against the new OpenSSL. Assuming you + are building with the built-in ld, make and gcc, then your build should go + something like this (notice the -R required by Sun's linker that sets the + runtime linking path in the resulting programs so the OpenSSL libraries + load from '/usr/local/ssl/lib'): + +---%<------------------------------------------------------------------------- +PATH=$PATH:/usr/sfw/bin:/usr/ccs/bin +export PATH +mv /usr/lib/pkgconfig/openssl.pc /usr/lib/pkgconfig/openssl.pc.orig +CPPFLAGS=-I/usr/local/ssl/include \ + LDFLAGS='-L/usr/local/ssl/lib -R/usr/local/ssl/lib' \ + ./configure --with-ssl=openssl +make +make install +---%<------------------------------------------------------------------------- + +Notify method +============= + +Linux +----- + +Note that current 'inotify' is in the Linux kernel since version 2.6.13 and it +is preferred over 'dnotify'. If your distribution does not have the required +'inotify' header file, you can get it from the inotify maintainer (this example +requires cURL [http://curl.haxx.se/]): + +---%<------------------------------------------------------------------------- +mkdir -p /usr/local/include/sys +cd /usr/local/include/sys +curl ftp://ftp.kernel.org/pub/linux/kernel/people/rml/inotify/headers/inotify.h +-O +curl +ftp://ftp.kernel.org/pub/linux/kernel/people/rml/inotify/headers/inotify-syscalls.h +>> inotify.h +---%<------------------------------------------------------------------------- + +/usr/local/include isn't in standard include lookup path, so you'll need to +specify that to configure: + +---%<------------------------------------------------------------------------- +CPPFLAGS=-I/usr/local/include ./configure --with-notify=inotify +---%<------------------------------------------------------------------------- + +Debian Etch ships 'sys/inotify.h' wrapped in the 'inotify-tools' package and +installs the header file into '/usr/include/inotifytools/'. To use the header +file use: + +---%<------------------------------------------------------------------------- +if ! test -e /usr/include/sys/inotify.h; then + aptitude install inotify-tools + ln -sf /usr/include/inotifytools/inotify.h /usr/include/sys/inotify.h +fi +---%<------------------------------------------------------------------------- + +Then pass 'CPPFLAGS' as in the example above: + +---%<------------------------------------------------------------------------- +CPPFLAGS=-I/usr/include/inotifytools ./configure --with-notify=inotify +---%<------------------------------------------------------------------------- + +Optional Configure Options +========================== + +--help: + gives a full list of available options + +--help=short: + just lists the options added by the particular package (= Dovecot) + +Options are usually listed as '--with-something' or '--enable-something'. If +you want to disable them, do it as '--without-something' or +'--disable-something'. There are many default options that come from autoconf, +automake or libtool. They are explained elsewhere. + +Here is a list of options that Dovecot adds. You should not usually have to +change these, but they are described here just for completeness: + +--enable-devel-checks: + Enables some extra sanity checks. This is mainly useful for developers. It + does quite a lot of unnecessary work but should catch some programming + mistakes more quickly. + +--enable-asserts: + Enable assertion checks, enabled by default. Disabling them may slightly save + some CPU, but if there are bugs they can cause more problems since they are + not detected as early. + +--without-shared-libs: + Link Dovecot binaries with static libraries instead of dynamic libraries. + +--disable-largefile: + Specifies if we use 32bit or 64bit file offsets in 32bit CPUs. 64bit is the + default if the system supports it (Linux and Solaris do). Dropping this to + 32bit may save some memory, but it prevents accessing any file larger than 2 + GB. + +--with-mem-align=BYTES: + Specifies memory alignment used for memory allocations. It is needed with + many non-x86 systems and it should speed up x86 systems too. Default is 8, to + make sure 64bit memory accessing works. + +--with-ioloop=IOLOOP: + Specifies what I/O loop method to use. Possibilities are 'select', 'poll', + 'epoll' and 'kqueue'. The default is to use the best method available on your + system. + +--with-notify=NOTIFY: + Specifies what file system notification method to use. Possibilities are + 'dnotify', 'inotify' (both on Linux), 'kqueue' (FreeBSD) and 'none'. The + default is to use the best method available on your system. See [CompilingSource.txt] above for more information. + +--with-storages=FORMATS: + Specifies what mailbox formats to support. Note: Independent of this option, + the formats /raw/ and /shared/ will be always built. + +--with-solr: + Build with Solr full text search support + +--with-zlib: + Build with zlib compression support (default if detected) + +--with-bzlib: + Build with bzip2 compression support (default if detected) + +SQL Driver Options +------------------ + +SQL drivers are typically used only for authentication, but they may be used as +a lib-dict backend too, which can be used by plugins for different purposes. + +--with-sql-drivers: + Build with specified SQL drivers. Defaults to all that were found with + autodetection. + +--with-pgsql: + Build with PostgreSQL support (requires pgsql-devel, libpq-dev or similar + package) + +--with-mysql: + Build with MySQL support (requires mysql-devel, libmysqlclient15-dev or + similar package) + +--with-sqlite: + Build with SQLite3 driver support (requires sqlite-devel, libsqlite3-dev or + similar package) + +Authentication Backend Options +------------------------------ + +The basic backends are built if the system is detected to support them: + +--with-shadow: + Build with [PasswordDatabase.Shadow.txt] password support + +--with-pam: + Build with [PasswordDatabase.PAM.txt] support + +--with-nss: + Build with [UserDatabase.NSS.txt] support + +--with-sia: + Build with Tru64 SIA support + +--with-bsdauth: + Build with [PasswordDatabase.BSDAuth.txt] support (if + supported by your OS) + +Some backends require extra libraries and are not necessarily wanted, so they +are built only if specifically enabled: + +--with-sql: + Build with generic SQL support (drivers are enabled separately) + +--with-ldap: + Build with LDAP support (requires openldap-devel, libldap2-dev or similar + package) + +--with-gssapi: + Build with GSSAPI authentication support (requires krb5-devel, libkrb5-dev or + similar package) + +--with-vpopmail: + Build with vpopmail support (requires vpopmail sources or a devel package) + +It's also possible to build these as plugins by giving e.g. --with-sql=plugin. + +Dynamic IMAP and POP3 Modules +============================= + +The 'mail_plugins' setting lists all plugins that Dovecot is supposed to load +from the 'mail_plugin_dir' directory at program start. These plugins can do +anything they want. They are only expected to contain the '_init' +and '_deinit' functions which are called at startup and at exit. + +The plugin filename is prefixed with a number which specifies the order in +which the plugins are loaded. This is important if one plugin depends on +another. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/ConfigFile.txt b/doc/wiki/ConfigFile.txt new file mode 100644 index 0000000000..0f7ab52b81 --- /dev/null +++ b/doc/wiki/ConfigFile.txt @@ -0,0 +1,312 @@ +Dovecot Configuration File +========================== + +Contents + + + 1. Dovecot Configuration File + + 1. Basic syntax + + 2. Sections + + 3. Filters + + 4. Including config files + + 5. External config files + + 6. Long lines + + 7. Reading value from file + + 8. Variable expansion + +Basic syntax +------------ + +The syntax generally looks like this: + +---%<------------------------------------------------------------------------- +# this is a comment + +settings_key = settings_value +---%<------------------------------------------------------------------------- + +If Dovecot doesn't seem to be reading your configuration correctly, use +'doveconf -n' to check how Dovecot actually parses it. You can also check more +complex configurations by providing filters, for example:'doveconf -n -f +service=imap -f local=10.0.0.1 -f remote=1.2.3.4' + +Sections +-------- + +Sections look like this: + +---%<------------------------------------------------------------------------- +section optional_name { + section_setting_key = section_setting_value + subsection optional_subname { + subkey = subvalue + } +} +---%<------------------------------------------------------------------------- + +Note that sections must be currently written with the linefeeds as shown above. +So for example this doesn't work: + +---%<------------------------------------------------------------------------- +section optional_name { key = value } # DOES NOT WORK +---%<------------------------------------------------------------------------- + +The sections can be optionally named. This is especially useful if you want to +update the same section later on in the config. For example: + +---%<------------------------------------------------------------------------- +namespace inbox { + inbox = yes +} +# ... +# possibly included from another file: +namespace inbox { + mailbox Trash { + special_use = \Trash + } +} +# The namespaces get merged into the same inbox namespace. +---%<------------------------------------------------------------------------- + +Without naming the namespace it would have created a new namespace. The section +name may also sometimes be used as part of the settings instead of simply a +name. For example: + +---%<------------------------------------------------------------------------- +service auth { + unix_listener auth-master { + # .. + } +} +---%<------------------------------------------------------------------------- + +Above the "auth-master" both uniquely identifies the section name, but also it +names the UNIX socket path. + +Filters +------- + +There are a few different filters that can be used to apply settings +conditionally. The filters look exactly like sections, which may be a bit +confusing. The currently supported filters are: + + * protocol : Name of the service/protocol that is reading the settings. + For example: imap, pop3, doveadm, lmtp, lda + * remote : Remote client's IP/network. For non-TCP connections + this will never match. For example 10.0.0.1 or 10.0.0.0/16. + * local_name : Matches TLS connection's SNI + [https://en.wikipedia.org/wiki/Server_Name_Indication] name, if it's sent by + the client. Commonly used to + [SSL.DovecotConfiguration.txt]. + * local : Locally connected IP/network. For non-TCP connections this + will never match. For example 127.0.0.1 or 10.0.0.0/16. + +These filters work for most of the settings, but most importantly auth settings +currently only support the protocol filter. Some of the other settings are also +global and can't be filtered, such as log_path. + +An example, which uses all of the filters: + +---%<------------------------------------------------------------------------- +local 127.0.0.1 { + local_name imap.example.com { + remote 10.0.0.0/24 { + protocol imap { + # ... + } + } + } +} +---%<------------------------------------------------------------------------- + +The nesting of the filters must be exactly in that order or the config parsing +will fail. + +When applying the settings, the settings within the most-specific filters +override the less-specific filter's settings, so the order of the filters in +config file doesn't matter. For example: + +---%<------------------------------------------------------------------------- +local 127.0.0.2 { + key = 127.0.0.2 +} +local 127.0.0.0/24 { + key = 127.0.0.0/24 +} +local 127.0.0.1 { + key = 127.0.0.1 +} +# The order of the above blocks doesn't matter: +# If local IP=127.0.0.1, key=127.0.0.1 +# If local IP=127.0.0.2, key=127.0.0.2 +# If local IP=127.0.0.3, key=127.0.0.0/24 +---%<------------------------------------------------------------------------- + +Similarly remote local filters override remote filters, which override +local_name filters, which override protocol filters. In some situations Dovecot +may also return an error if it detects that the same setting is being +ambiguously set by multiple matching filters. + +Including config files +---------------------- + +The main dovecot.conf file can also include other config files: + +---%<------------------------------------------------------------------------- +!include local.conf +!include /path/to/another.conf +!include conf.d/*.conf +---%<------------------------------------------------------------------------- + +The paths are relative to the currently parsed config file's directory. For +example: + +---%<------------------------------------------------------------------------- +# /etc/dovecot/dovecot.conf: +!include conf.d/imap.conf +# /etc/dovecot/conf.d/imap.conf: +!include imap2.conf +# /etc/dovecot/conf.d/imap2.conf is being included +---%<------------------------------------------------------------------------- + +If any of the includes fail (e.g. file doesn't exist or permission denied), it +results in an error. It's not an error if wildcards don't result in any +matching files. To avoid these errors, you can use !include_try instead: + +---%<------------------------------------------------------------------------- +!include_try passwords.conf +---%<------------------------------------------------------------------------- + +Including a file preserves the context where it's included from. For example: + +---%<------------------------------------------------------------------------- +protocol imap { + plugin { +!include imap-plugin-settings.conf + } +} +---%<------------------------------------------------------------------------- + +External config files +--------------------- + +Due to historical reasons there are still some config files that are external +to the main dovecot.conf, which are typically named '*.conf.ext'. For example: + + * passdb/userdb { args } for ldap/sql points to a dovecot-ldap.conf.ext and + dovecot-sql.conf.ext. + * dict { .. } points to dovecot-dict-*.conf.ext + +Although these external config files look similar to the main dovecot.conf +file, they have quite a lot of differences in details. Their parsing is done +with a completely different config parser, so things like filters, $variables, +!includes and and + for details), but with SMTP AUTH you'll need to use +PLAIN authentication mechanism, which requires you to build a base64-encoded +string in the correct format. The PLAIN authentication is also used internally +by both IMAP and POP3 to authenticate to dovecot-auth, so you see it in the +debug logs. + +The PLAIN mechanism's authentication format is: NUL + NUL . Authorization ID is the username who you +want to log in as, and authentication ID is the username whose password you're +giving. If you're not planning on doing a +[Authentication.MasterUsers.txt], you can either set both of these fields to +the same username, or leave the authorization ID empty. + +Encoding with mmencode +---------------------- + +printf(1) and mmencode(1) should be available on most Unix or GNU/Linux +systems. (If not, check with your distribution. GNU coreutils includes +printf(1), and metamail includes mmencode(1). In Debian, mmencode is called +mimencode(1).) + +---%<------------------------------------------------------------------------- +$ printf 'username\0username\0password' | mmencode +dXNlcm5hbWUAdXNlcm5hbWUAcGFzc3dvcmQ= +---%<------------------------------------------------------------------------- + +This string is what a client would use to attempt PLAIN authentication as user +"username" with password "password." With ''auth_debug_passwords=yes', it would +appear in your logs. + +Decoding with mmencode +---------------------- + +You can use mmencode -u to interpret the encoded string pasted into stdin as +follows: + +---%<------------------------------------------------------------------------- +# mmencode -u +bXl1c2VybmFtZUBkb21haW4udGxkAG15dXNlcm5hbWVAZG9tYWluLnRsZABteXBhc3N3b3Jk +myusername@domain.tldmyusername@domain.tldmypassword +# +---%<------------------------------------------------------------------------- + +You should see the correct user address (twice) and password. The null bytes +won't display. + +Encoding with Perl +------------------ + +Unfortunately, mmencode on FreeBSD chokes on "\0". As an alternate, if you +have MIME::Base64 on your system, you can use a perl statement to do the same +thing: + +---%<------------------------------------------------------------------------- +perl -MMIME::Base64 -e 'print +encode_base64("myusername\@domain.tld\0myusername\@domain.tld\0mypassword");' +---%<------------------------------------------------------------------------- + +As mmencode -u doesn't encounter any "\0" you can still do: + +---%<------------------------------------------------------------------------- +perl -MMIME::Base64 -e 'print +encode_base64("myusername\@domain.tld\0myusername\@domain.tld\0mypassword");' | +mmencode -u +---%<------------------------------------------------------------------------- + +to check that you have encoded correctly. + +Encoding with Python +-------------------- + +With python you can do: + +---%<------------------------------------------------------------------------- +python -c "import base64; +print(base64.encodestring('myusername@domain.tld\0myusername@domain.tld\0mypassword'));" +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Debugging.ProcessTracing.txt b/doc/wiki/Debugging.ProcessTracing.txt new file mode 100644 index 0000000000..89ea41ae6b --- /dev/null +++ b/doc/wiki/Debugging.ProcessTracing.txt @@ -0,0 +1,44 @@ +Process Tracing +=============== + +If a Dovecot's process hangs or is just really slow, the best way to debug it +is to see what it's really doing. Typically you'd be looking into imap or pop3 +processes. + +Linux +----- + +---%<------------------------------------------------------------------------- +strace -tt -o log -p +---%<------------------------------------------------------------------------- + +BSDs, OS X <= 10.4 +------------------ + +---%<------------------------------------------------------------------------- +# enable process tracing +ktrace -f log -p +# do whatever makes it break, then stop the process tracing: +ktrace -C +# and see what it's done: +kdump -T -f log +---%<------------------------------------------------------------------------- + +OS X >= 10.5 +------------ + +---%<------------------------------------------------------------------------- +dtruss -p +---%<------------------------------------------------------------------------- + +Solaris +------- + +---%<------------------------------------------------------------------------- +truss -d -r0 -w1 -o log -p +---%<------------------------------------------------------------------------- + +'-r0' and '-w1' cause all IMAP input/output to be logged. '-d' adds timestamps +to the log. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Debugging.Rawlog.txt b/doc/wiki/Debugging.Rawlog.txt new file mode 100644 index 0000000000..fc1ffdcfeb --- /dev/null +++ b/doc/wiki/Debugging.Rawlog.txt @@ -0,0 +1,148 @@ +Rawlog +====== + +Dovecot supports logging IMAP/POP3/LMTP/SMTP(submission) traffic (also TLS/SSL +encrypted). There are several possibilities for this: + + 1. rawlog_dir setting (v2.2.26+) + 2. Using 'rawlog' binary, which is executed as post-login script. + 3. Pre-login imap/pop3-login process via -R parameter. + 4. For [LMTP.txt], you need to use lmtp_rawlog_dir and + lmtp_proxy_rawlog_dir settings (since v2.3.2) + 5. For [Submission.txt], you can use rawlog_dir setting and + submission_relay_rawlog_dir (since v2.3.2) + +rawlog_dir setting (v2.2.26+) +----------------------------- + +Dovecot creates *.in and *.out rawlogs to the specified directory if it exists. +For example: + +---%<------------------------------------------------------------------------- +protocol imap { + rawlog_dir = /tmp/rawlog/%u + # if you want to put files into user's homedir, use this, do not use ~ + #rawlog_dir = %h/rawlog +} +---%<------------------------------------------------------------------------- + +lmtp_rawlog_dir (v2.3.2+) +------------------------- + +You can use lmtp_rawlog_dir to generate rawlogs on lmtp backend server. Unlike +the rawlog_dir setting, this does not accept variables. + +lmtp_proxy_rawlog_dir (v2.3.2+) +------------------------------- + +You can use lmtp_proxy_rawlog_dir to generate rawlogs on lmtp proxy server. +Unlike the rawlog_dir setting, this does not accept variables. + +submission_relay_rawlog_dir (v2.3.2+) +------------------------------------- + +You can use submission_relay_rawlog_dir to generate relay rawlogs on the +dovecot submission server. + +rawlog binary +------------- + +It works by checking if 'dovecot.rawlog/' directory exists in the logged in +user's home directory, and writing the traffic to 'yyyymmdd-HHMMSS-pid.in' and +'.out' files. Each connection gets their own in/out files. Rawlog will simply +skip users who don't have the 'dovecot.rawlog/' directory and the performance +impact for those users is minimal. + +Home directory +-------------- + +Note that for rawlog to work, your [UserDatabase.txt] must have +returned a home directory for the user.*IMPORTANT: The home directory must be +returned by userdb, mail_home setting won't work.* Verify that 'doveadm user -u +user@example.com' (with -u parameter) returns the home directory, for example: + +---%<------------------------------------------------------------------------- +% doveadm user -u user@example.com +userdb: user@example.com + user : user@example.com + uid : 1000 + gid : 1000 + home : /home/user@example.com +---%<------------------------------------------------------------------------- + +In above configuration rawlog would expect to find +'/home/user@example.com/dovecot.rawlog/' directory writable by uid 1000. + +If your userdb can't return a home directory directly, with v2.1+ you can add: + +---%<------------------------------------------------------------------------- +userdb { + # ... + default_fields = home=/home/%u + # or temporarily even e.g. default_fields = home=/tmp/temp-home +} +---%<------------------------------------------------------------------------- + +You can also set DEBUG environment to have rawlog log an info message why it's +not doing anything: + +---%<------------------------------------------------------------------------- +import_environment = $import_environment DEBUG=1 +---%<------------------------------------------------------------------------- + +Configuration +------------- + +To enable rawlog, you must use rawlog as a +[PostLoginScripting.txt]: + +---%<------------------------------------------------------------------------- +service imap { + executable = imap postlogin +} +service pop3 { + executable = pop3 postlogin +} + +service postlogin { + executable = script-login -d rawlog + unix_listener postlogin { + } +} +---%<------------------------------------------------------------------------- + +You can also give parameters to rawlog: + + * -b: Write IP packet boundaries (or whatever read() sees anyway) to the log + files. The packet is written between<<< and >>>. + * -t: Log a microsecond resolution timestamp at the beginning of each line. + * -I: Include IP address in the filename (v2.2.16+) + * v2.1 and newer: + * -f in: Log only to *.in files + * -f out: Log only to *.out files + * v2.0 and older: + * -i: Log only to *.in files + * -o: Log only to *.out files + +Pre-login rawlog (v2.1+) +------------------------ + +You can enable pre-login rawlog for all users by telling the login processes to +log to a rawlog directory, for example: + +---%<------------------------------------------------------------------------- +service imap-login { + executable = imap-login -R rawlogs +} +---%<------------------------------------------------------------------------- + +This tries to write the rawlogs under $base_dir/login/rawlogs directory. You +need to create it first with enough write permissions, e.g.: + +---%<------------------------------------------------------------------------- +mkdir /var/run/dovecot/login/rawlogs +chown dovenull /var/run/dovecot/login/rawlogs +chmod 0700 /var/run/dovecot/login/rawlogs +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Debugging.Thunderbird.txt b/doc/wiki/Debugging.Thunderbird.txt new file mode 100644 index 0000000000..db82307101 --- /dev/null +++ b/doc/wiki/Debugging.Thunderbird.txt @@ -0,0 +1,58 @@ +Debugging using Thunderbird's logging +------------------------------------- + +Thunderbird has the ability to log its client actions based on protocol; this +can be useful when you're experiencing a problem with Dovecot and want to trap +the commands that the client is sending. + +Windows batch file +------------------ + +Save the below as 'runtbird.bat' and place it on the desktop, then run this +instead of the Thunderbird icon. + +---%<------------------------------------------------------------------------- +set mydate=%date:~-4,4%%date:~-7,2%%date:~-10,2% +set mytime=%time:~0,2%%time:~+3,2% + +set NSPR_LOG_MODULES=IMAP:5 +set NSPR_LOG_FILE=%USERPROFILE%\thunderbird_%mydate%_%mytime%.log + +start /d "c:\program files\mozilla thunderbird" thunderbird.exe +---%<------------------------------------------------------------------------- + +Adjust the log location and Thunderbird install folder as appropriate. If you +want to log all modules instead of just IMAP (SMTP, e.g.) then replace "IMAP" +above with "all". The above will create a date/time stamped logfile for each +run, so you won't lose the previous logs. + +Linux/BSD/etc. shell script +--------------------------- + +Save the below as 'runtbird.sh', chmod it 0755, then run this instead of the +Thunderbird icon. + +---%<------------------------------------------------------------------------- +#!/bin/sh + +TB_PATH=`which thunderbird` +# or for MacOSX: +#TB_PATH="/Applications/Thunderbird.app/Contents/MacOS/thunderbird-bin" + +MYDATE=`date "+%Y%m%d_%H%M%S"` +NSPR_LOG_MODULES=IMAP:5 +NSPR_LOG_FILE=/tmp/thunderbird_${MYDATE}.log +export NSPR_LOG_MODULES NSPR_LOG_FILE + +$TB_PATH & +exit $? +---%<------------------------------------------------------------------------- + +Adjust the log location and Thunderbird launch binary as appropriate. If you +want to log all modules instead of just IMAP (SMTP, e.g.) then replace "IMAP" +above with "all". You can also get a timestamp at the begining of all the log +lines, if you add ",timestamp" after "NSPR_LOG_MODULES=IMAP:5" for example. The +above will create a date/time stamped logfile for each run, so you won't lose +the previous logs. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Arrays.txt b/doc/wiki/Design.Arrays.txt new file mode 100644 index 0000000000..2ecb7873ce --- /dev/null +++ b/doc/wiki/Design.Arrays.txt @@ -0,0 +1,109 @@ +Dynamic Arrays +============== + +'lib/array.h' and 'lib/array-decl.h' describes Dovecot's type-safe dynamic +arrays. Trying to add wrong typed elements gives a compiler warning. + +Declaring +--------- + +Arrays can be declared in two ways: + + 1. Directly: 'ARRAY_DEFINE(array_name, array_type);'. For example: + 'ARRAY_DEFINE(numbers, int);' or 'ARRAY_DEFINE(foos, struct foo);' + 2. Via predefined type: 'ARRAY_DEFINE_TYPE(foo, struct foo); ... + ARRAY_TYPE(foo) foos;' + +The main reason to define a type for an array is to be able to pass the array +as a function parameter, like: + +---%<------------------------------------------------------------------------- +void func(ARRAY_TYPE(foo) *foos) { .. } +---%<------------------------------------------------------------------------- + +Trying to do the same with 'ARRAY_DEFINE()' will generate a compiler warning. +'lib/array-decl.h' defines several commonly used types. + +Initializing +------------ + +Arrays are typically initialized by calling 'i_array_init()', 'p_array_init()' +or 't_array_init()' depending on where you want to allocate the memory from. +Arrays are internally handled as [Design.Buffers.txt], so the initial +size is just multiplied by element size and passed to +'buffer_create_dynamic()'. + +Example: + +---%<------------------------------------------------------------------------- +ARRAY_DEFINE(foo, struct foo *); + +i_array_init(&foo, 32); /* initialize array with 32 elements until it needs to +be grown */ +---%<------------------------------------------------------------------------- + +Arrays can be freed with 'array_free()', but this isn't necessary if the memory +gets freed by other means (i.e. it was allocated from alloconly-pool or data +stack). + +Writing +------- + + * 'array_append(array, data, count)' is the most common way to add data to + arrays + * 'array_append_array(dest, src)' + * 'array_insert(array, idx, data, count)' + * 'array_delete(array, idx, count)' + * 'array_idx_set(array, idx, data)' replaces (or adds) data to given index + * 'array_idx_clear(array, idx)' clears given index by writing NULs to it + * 'array_append_space(array, count)' + +Reading +------- + +'array_idx(array, idx)' returns pointer to given index in array. The index must +already exist, otherwise the call assert-crashes. This call adds extra overhead +for accessing arrays though, so usually it's better to just get list of all +elements and access them directly: + +---%<------------------------------------------------------------------------- +data = array_get(&array, &count); +---%<------------------------------------------------------------------------- + +You can also iterate through the whole array easily: + +---%<------------------------------------------------------------------------- +const char *str; + +array_foreach(&string_array, str) { + /* str changes in each iteration */ +} +---%<------------------------------------------------------------------------- + +There's also 'array_foreach_modifiable()' to get the data without const. + +Unsafe Read/Write +----------------- + +Functions below have similar problems to [[Design/Buffer|buffer]'s '*_unsafe()' +functions. Memory returned by them must not be accessed after calls to other +'array_*()' modifying functions, because they may reallocate the array +elsewhere in memory. + + * 'array_append_space(array)' + * 'array_insert_space(array, idx)' + * 'array_get_modifiable(array, &count)' + * 'array_idx_modifiable(array, idx)' + +Others +------ + + * 'array_cmp(array1, array2)' compares two arrays + * 'array_reverse(array)' reverses all elements in an array + * 'array_sort(array, cmp_func)' is a wrapper for 'qsort()' adding also type + safety. The parameters in cmp_func should be the same type as the array, + instead of 'const void *'. + * 'array_bsearch(array, key, cmp_func)' is a wrapper for 'bsearch()' also + adding type safety, just like 'array_sort()'. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.AuthProcess.txt b/doc/wiki/Design.AuthProcess.txt new file mode 100644 index 0000000000..dd4fc8959c --- /dev/null +++ b/doc/wiki/Design.AuthProcess.txt @@ -0,0 +1,365 @@ +Authentication process design +============================= + +See for an overview of how the authentication process +works. + +There are four major classes in the code: + + * 'struct mech_module': Authentication mechanism + * 'struct password_scheme': Password scheme + * 'struct passdb_module': Password database + * 'struct userdb_module': User database + +There are many implementations for each of these, and it's simple to add more +of them. They can also be added as plugins, although the current plugin loading +code doesn't allow loading authentication mechanisms cleanly, and it's not +possible to add new credentials (see below). + +The code flow usually goes like: + + * Dovecot-auth listens for new authentication client connections (the listener + socket is created by master process and passed in MASTER_SOCKET_FD -> + 'main.c:main_init()' -> + 'auth-master-connection.c:auth_master_listener_add()') + * A new authentication client connects via UNIX socket + ('auth-master-connection.c:auth_master_listener_accept()' -> + 'auth-client-connection.c:auth_client_connection_create()') + * Authentication client begins an authentication + ('auth-client-connection.c:auth_client_input()' -> + 'auth_client_handle_line()' -> + 'auth-request-handler.c:auth_request_handler_auth_begin()' [ -> + 'auth-request.c:auth_request_new()']) + * Authentication mechanism backend handles it ('mech->auth_initial()' and + 'mech->auth_continue()' in 'mech-*.c') + * The mechanism looks up the password from passdb + ('auth-request.c:auth_request_verify_plain()' and + 'auth_request_lookup_credentials()') and the password scheme code to + verifies it ('password-scheme.c:password_verify()' and + 'password_generate()') + * If user is logging in, the user information is looked up from the userdb + ('auth-master-connection.c:master_input()' -> 'master_input_request()' -> + 'auth-request-handler.c:auth_request_handler_master_request()' -> + 'auth-request.c:auth_request_lookup_user()') + * The authentication may begin new authentication requests even before the + existing ones are finished. + +It's also possible to request a userdb lookup directly, for example Dovecot's + [LDA.txt] needs that. The code path for that goes +'auth-master-connection.c:master_input()' -> 'master_input_user()' -> +'auth-request.c:auth_request_lookup_user()'. + +Authentication mechanisms +------------------------- + +These are [Sasl.txt] authentication mechanism implementations. See + for a list of mechanisms supported by Dovecot. + +A new mechanism is created by filling a 'struct mech_module' (in 'mech.h') and +passing it to 'mech_register_module()'. The struct fields are: + +mech_name: + The public name of the mechanism. This is shown to clients in the IMAP, POP3 + and SMTP capability lists. If you create a new non-standard mechanism, please + prefix it with "X-". + +flags: + Describes how secure the mechanism is. Also 'MECH_SEC_PRIVATE' flag specifies + that the mechanism shouldn't be advertised in the capability list. This is + currently used only for APOP mechanism, which is defined by the POP3 protocol + itself. + +passdb_need_plain: + This mechanism uses passdb's 'verify_plain()' function to verify the + password's validity. This means that the mechanism has access to the + plaintext password. This is true only for plaintext mechanisms such as PLAIN + and LOGIN. The main purpose of this flag is to make dovecot-auth complain at + startup if there are no passdbs defined in the configuration file. Note that + a configuration without any passdbs is valid with eg. GSSAPI mechanism which + doesn't need a passdb at all. + +passdb_need_credentials: + This mechanism uses passdb's 'lookup_credentials()' function. See below for + description of the credentials. + +auth_new(): + Allocates a new 'struct auth_request'. Typically with more complex mechanisms + it really allocates a 'struct _auth_request' which contains 'struct + auth_request' as the first field, followed by mechanism-specific fields. + +auth_initial(request, data, data_size): + This begins the authentication, data and data_size containing the initial + response sent by the client (decoded, not in base64). Call + 'request->callback()' once you're done (see below). + +auth_continue(request, data, data_size): + Continues the authentication. Works the same as 'auth_initial()'. + +auth_free(): + Free the request. Usually all the memory allocations for the request should + be allocated from 'request->pool', so you can use 'mech_generic_auth_free()' + which simply frees the pool. + +'auth_initial()' and 'auth_continue()' continue or finish the authentication by +calling 'request->callback()': + +---%<------------------------------------------------------------------------- +typedef void mech_callback_t(struct auth_request *request, + enum auth_client_result result, + const void *reply, size_t reply_size); +---%<------------------------------------------------------------------------- + +The 'reply' and 'reply_size' contain the server's mechanism-specific reply to +the client. If there is no need to return anything (which is usually the case +with the "success" reply), the 'reply_size' can be 0. The 'result' parameter is +one of: + + * AUTH_CLIENT_RESULT_CONTINUE: Client can continue the authentication. The + reply contains the mechanism-specific reply sent to the client. + * AUTH_CLIENT_RESULT_SUCCESS: Authentication successful. The reply is usually + empty. + * AUTH_CLIENT_RESULT_FAILURE: Authentication failed. The reply is always + ignored. + +The 'request->callback()' should actually be called directly only for +continuation requests (a new function should probably be added for this as +well). For success and failure replies, you should instead use one of these +functions: + + * 'auth_request_success()' + * 'auth_request_fail()' + * 'auth_request_internal_failure()': Use this if you couldn't figure out if + the authentication succeeded or failed, for example because passdb lookup + returned internal failure. + +SASL authentication in general works like: + + 1. Client begins the authentication, optionally sending an "initial response", + meaning some data that the mechanism sees in 'auth_initial()'. + * Note that not all protocols support the initial response. For example + IMAP supports it only if the server implements SASL-IR extension. + Because of this mechanisms, such as PLAIN, support doing the + authentication either in 'auth_initial()' or in 'auth_continue()'. + * If the client initiates the authentication (ie. server's initial reply + is empty, such as with PLAIN mechanism) you can use + 'mech_generic_auth_initial()' instead of implementing your own. + 2. Server processes the authentication request and replies back with + 'request->callback()'. + * If the authentication failed, it's placed into 'auth_failures_buf' + unless 'request->no_failure_delay=TRUE'. The failures are flushed from + the buffer once every 2 seconds to clients and 'mechanism->auth_free()' + is called. + * If the authentication succeeded and + * there is a master connection associated with the request (IMAP/POP3 + login), the authentication now waits for master connection to do a + verification request. If this for some reason doesn't happen in + 'AUTH_REQUEST_TIMEOUT' seconds (3,5 mins), it's freed. + * there isn't a master connection (SMTP AUTH), the authentication is + freed immediately. + 3. Client processes the reply: + * If the authentication continues, it sends back more data which is + processed in 'auth_continue()'. Goto 2. + * If the authentication failed, it's done. + 4. If the authentication succeeded, the client requests a login from the + master process, which in turn requests verification from the auth process. + * Besides verifying the authentication, dovecot-auth also does a userdb + lookup to return the userdb information to master. + * If the verification fails (normally because userdb lookup fails), the + client gets "internal authentication failure" + * If the verification succeeds, the user is now logged in + * In either case, 'mechanism->auth_free()' is called now. + +Credentials +----------- + +Most of the non-plaintext mechanisms need to verify the authentication by using +a special hash of the user's password. So either the passdb credentials lookup +returns a plaintext password from which the hash can be created, or the hash +directly. The plaintext to hash conversion is done by calling +'password_generate' function of the password scheme. + +Unfortunately the list of allowed credentials is currently hardcoded in 'enum +passdb_credentials'. The enum values are mapped to password scheme strings in +'passdb_credentials_to_str()'. Some day the enum will be removed so plugins can +add new mechanisms. Besides the mechanism-specific credentials, the enum +contains: + +_PASSDB_CREDENTIALS_INTERNAL: + I don't remember why this really exists. It should probably be called + _PASSDB_CREDENTIALS_INVALID or something and used only by some asserts.. + +PASSDB_CREDENTIALS_PLAINTEXT: + Request a plaintext password. + +PASSDB_CREDENTIALS_CRYPT: + Request the password in any scheme. This is especially useful if you only + want to verify a user's existence in a passdb. Used by + [UserDatabase.Static.txt] in userdb lookups. + +Password schemes +---------------- + +'struct password_scheme' has fields: + +name: + Name of the scheme. This only shows up in configuration files and maybe in + the passwords stored in passdb ("{scheme_name}password_hash"). + +password_verify(plaintext, password, user): + Returns TRUE if 'password' hash matches the plaintext password given in + 'plaintext' parameter. If the password hash depends on the username (eg. with + DIGEST-MD5), the 'user' parameter can also be used. + +password_generate(plaintext, user): + Returns the password hash for given plaintext password and username. + +You can create a new password scheme by simply creating a 'struct +password_scheme' named '_scheme', compiling a shared object and +placing it to '$moduledir/auth/' directory. + +Password databases +------------------ + +See for a description of passdbs and a list of already +implemented ones. + +'struct passdb_module' contains fields: + +cache_key: + A string containing [Variables.txt]. When expanded, it uniquely + identifies a passdb lookup. This is '%u' when the passdb lookup validity + depends only on the username. With more complex databases such as SQL and + LDAP this is created dynamically based on the password query in the + configuration file. If there are multiple variables, they should be separated + so that their contents don't get mixed, for example '%u%r%l'. + 'auth_cache_parse_key()' can be used to easily create a cache key from a + query string. + +default_pass_scheme: + The default scheme to use when it's not explicitly specified with a + "{scheme}" prefix. + +blocking: + If TRUE, the lookup is done in dovecot-auth worker process. This should be + used if the lookup may block. + +iface.preinit(auth_passdb, args): + Allocate 'struct passdb_module' and return it. This function is called before + chrooting and before privileges are dropped from dovecot-auth process, so if + should do things like read the configuration file.'auth_passdb' is typically + used for getting a memory pool and looking up some global settings such as + 'auth_passdb->auth->verbose_debug'. 'args' contains the args parameter in + configuration file. + +iface.init(module, args): + The privileges have been dropped before calling this. 'module' contains the + structure returned by 'preinit()'. 'args' is the same as in 'preinit()'. + Typically this function will do things like connect to the database. + +iface.deinit(module): + Close the connection to the password database and free all the memory you + used. + +iface.verify_plain(auth_request, password, callback): + Check if the given plaintext password matches. 'auth_request->credentials = + -1' always. When the verification is done, call the given callback with the + result in 'result' parameter. + +iface.lookup_credentials(auth_request, callback): + Look up the password credentials. 'auth_request->credentials' contains the + credentials that the mechanism wants. When the lookup is finished, call the + given callback with the result in 'result' parameter, and if the lookup was + successful the credentials in 'password' parameter. + +Plaintext authentication mechanisms typically call 'verify_plain()', which is +possible to implement with all the passdbs. Non-plaintext mechanisms typically +call 'lookup_credentials()', which isn't possible to implement always (eg. +PAM). If it's not possible to implement 'lookup_credentials()', you can leave +the pointer to it NULL. + +If the passdb uses connections to external services, it's preferred that they +use non-blocking connections. Dovecot does this whenever possible (PostgreSQL +and LDAP for example). If it's not possible, set 'blocking = TRUE'. + +With both functions 'auth_request->passdb->passdb' contains the passdb_module +returned by your 'preinit()' function. 'auth_request->user' contains the +username whose password we're verifying. You don't need to worry about [MasterUsers.txt] here. It's also possible to use any other fields in +'auth_request' to do the lookup, such as 'service', 'local_ip' or 'remote_ip' +if they exist. Often you want to let user to configure the lookup with + [Variables.txt] (eg. SQL query). In that case you can use +'auth_request_get_var_expand_table()' to retrieve the variable table for +'var_expand()'. + +The passdb lookup can return one of the following results: + +PASSDB_RESULT_INTERNAL_FAILURE: + The lookup failed. For example SQL server is down. + +PASSDB_RESULT_SCHEME_NOT_AVAILABLE: + 'lookup_credentials()' requested a scheme which isn't in the passdb + +PASSDB_RESULT_USER_UNKNOWN: + The user doesn't exist in the database. + +PASSDB_RESULT_USER_DISABLED: + The user is disabled either entirely, or for this specific login (eg. only + POP3 logins allowed). This isn't commonly implemented in passdbs. + +PASSDB_RESULT_PASS_EXPIRED: + The user's password had expired. This isn't commonly implemented in passdbs. + +PASSDB_RESULT_PASSWORD_MISMATCH: + The password given in 'verify_plain()' wasn't valid. + +PASSDB_RESULT_OK: + Success. + +User databases +-------------- + +See for a description of userdbs and a list of already +implemented ones. + +'struct userdb_module' is very similar to 'struct passdb_module'. The lookup +callback is a bit different though: + +---%<------------------------------------------------------------------------- +typedef void userdb_callback_t(enum userdb_result result, + struct auth_stream_reply *reply, + struct auth_request *request); +---%<------------------------------------------------------------------------- + +'result' contains one of: + +USERDB_RESULT_INTERNAL_FAILURE: + The lookup failed. For example SQL server is down. + +USERDB_RESULT_USER_UNKNOWN: + The user doesn't exist in the database. + +USERDB_RESULT_OK: + Success. + +There is no equivalent for PASSDB_RESULT_USER_DISABLED currently. Practically +the userdb result is used only by Dovecot's [LDA.txt] to figure out +if the user exists or not. When logging in with IMAP or POP3, the user's +existence was already checked in passdb lookup, so only in rare conditions when +a user is logging in at the same time as it's being deleted, the userdb result +is USER_UNKNOWN. + +The 'reply' parameter contains the username (it's allowed to be different from +the looked up username) and a list of key=value pairs that were found from the +userdb. The userdb should make sure that at least "uid" and "gid" keys were +returned. Here's an example code based on passwd userdb: + +---%<------------------------------------------------------------------------- +reply = auth_stream_reply_init(auth_request); +auth_stream_reply_add(reply, NULL, pw->pw_name); +auth_stream_reply_add(reply, "uid", dec2str(pw->pw_uid)); +auth_stream_reply_add(reply, "gid", dec2str(pw->pw_gid)); +auth_stream_reply_add(reply, "home", pw->pw_dir); +callback(USERDB_RESULT_OK, reply, auth_request); +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.AuthProtocol.txt b/doc/wiki/Design.AuthProtocol.txt new file mode 100644 index 0000000000..14ab2e6115 --- /dev/null +++ b/doc/wiki/Design.AuthProtocol.txt @@ -0,0 +1,251 @@ +Dovecot Authentication Protocol v1.1 +==================================== + +General +------- + +This is a line based protocol. Each line is a command which ends with an LF +character. The maximum line length isn't defined, but it's currently expected +to fit into 8192 bytes. Authentication mechanism specific data transfers are +the largest single parameters. + +Each command is in format: + +---%<------------------------------------------------------------------------- + TAB +---%<------------------------------------------------------------------------- + +Parameters are split into required and optional parameters. Required parameters +aren't in any specific format, but optional parameters are either booleans +without a value, or a name=value pair. If optional parameter name is unknown, +the parameter should just be ignored. + +Typical command looks like (without spaces): + +---%<------------------------------------------------------------------------- +command TAB param1 TAB param2 TAB optname=value TAB optboolean +---%<------------------------------------------------------------------------- + +There is no way to have TABs or LFs in parameters. + +Client <-> Server +----------------- + +Client is an untrusted authentication client process. It can serve one or more +users, so from user's point of view it's usually eg. IMAP or SMTP server +process. + +Server is an authentication server process. + +The connection starts by both client and server sending handshakes: + +---%<------------------------------------------------------------------------- +C: "VERSION" TAB TAB +C: "CPID" TAB +S: "VERSION" TAB TAB +S: "SPID" TAB +S: "CUID" TAB +S: "COOKIE" TAB +S: "MECH" TAB [TAB ] (multiple times) +S: "DONE" +---%<------------------------------------------------------------------------- + +Both client and server should check that they support the same major version +number. If they don't, the other side isn't expected to be talking the same +protocol and should be disconnected. Minor version can be ignored. This +document is version number 1.1. + + * CPID and SPID specify client and server Process Identifiers (PIDs). They + should be unique identifiers for the specific process. UNIX process IDs are + good choices. + * CUID is a server process-specific unique connection identifier. It's + different each time a connection is established for the server. + * CPID is used by master's REQUEST command. + * SPID can be used by authentication client to tell master which server + process handled the authentication. + * CUID is currently useful only for APOP authentication. + * COOKIE returns connection-specific 128 bit cookie in hex. It must be given + to REQUEST command. (Protocol v1.1+ / Dovecot v2.0+) + * DONE finishes the handshake from server. CPID finishes the handshake from + client. + +Authentication Mechanisms +------------------------- + +MECH command announces an available authentication SASL mechanism. Mechanisms +may have parameters giving some details about them: + +anonymous: + Anonymous authentication + +plaintext: + Transfers plaintext passwords + +dictionary: + Subject to passive (dictionary) attack + +active: + Subject to active (non-dictionary) attack + +forward-secrecy: + Provides forward secrecy between sessions + +mutual-auth: + Provides mutual authentication + +private: + Don't advertise this as available SASL mechanism (eg. APOP) + +Authentication Request +---------------------- + +---%<------------------------------------------------------------------------- +C: "AUTH" TAB TAB TAB service= [TAB ] +S1: "FAIL" TAB [TAB ] +S2: "CONT" TAB TAB +S3: "OK" TAB [TAB ] +---%<------------------------------------------------------------------------- + +ID is a connection-specific unique request identifier. It must be a 32bit +number, so typically you'd just increment it by one. + +Service is the service requesting authentication, eg. POP3, IMAP, SMTP. + +AUTH and USER (see below) common parameters are: + +lip=: + Local IP - in standard string format, + +rip=: + Remote IP - ie. for IPv4 127.0.0.1 and for IPv6 ::1 + +lport=: + Local port + +rport=: + Remote port + +AUTH-only parameters are: + +secured: + Remote user has secured transport to auth client] (e.g. localhost, SSL, TLS) + +valid-client-cert: + Remote user has presented a valid SSL certificate. + +no-penalty: + Ignore auth penalty tracking for this request + +cert_username: + Username taken from client's SSL certificate. + +resp=: + Initial response for authentication mechanism. NOTE: This must be the last + parameter. Everything after it is ignored. This is to avoid accidental + security holes if user-given data is directly put to base64 string without + filtering out tabs. + +FAIL parameters may contain: + +reason=: + should be sent to remote user instead of the standard "Authentication + failed" messages. For example "invalid base64 data". It must NOT be used to + give exact reason for authentication failure (i.e. "user not found" vs. + "password mismatch"). + +code=temp_fail (v2.3+), temp ( TAB +---%<------------------------------------------------------------------------- + +The must match the of the AUTH command. + +FAIL and OK may contain multiple unspecified parameters which authentication +client may handle specially. The only one specified here is "user=" +parameter, which should always be sent if the userid is known. + +Server <-> Master +----------------- + +Master is a trusted process which may query results of previous client +authentication or information about a specific user. Master is optional and in +SMTP AUTH case it's not needed. + +The connection starts by both server and master sending handshakes: + +---%<------------------------------------------------------------------------- +S: "VERSION" TAB TAB +S: "SPID" TAB +M: "VERSION" TAB TAB +---%<------------------------------------------------------------------------- + +Auth with client <-> server, both should check that the version numbers are +valid. + +SPID can be used to let master identify the server process. + +Master Requests +--------------- + +---%<------------------------------------------------------------------------- +M: "REQUEST" TAB TAB TAB TAB +M: "USER" TAB TAB TAB service= [TAB ] +S: "NOTFOUND" TAB +S: "FAIL" TAB TAB +S: "USER" TAB TAB [TAB ] +---%<------------------------------------------------------------------------- + +Master commands can request information about existing authentication request, +or about a specified user. + +USER command's service and parameters are the same as with AUTH client request. + +ID is a connection-specific unique request identifier. It must be a 32bit +number, so typically you'd just increment it by one. + +NOTFOUND reply means that the user wasn't found. (v1.x also reported unknown +request IDs with NOTFOUND.) + +FAIL reply means an internal error occurred. Usually either a configuration +mistake or temporary error caused by lost resource (eg. database down). Also +unknown request IDs are reported as FAILs (since v2.0). + +USER reply is sent if request succeeded. It can return parameters: + +uid=: + System user ID. + +gid=: + System group ID. + +home=: + Home directory. + +chroot=: + Chroot directory. + +mail=: + Mail location. + +system_user=: + System user name which can be used to get extra groups. This will probably + be replaced later by giving just multiple gid fields. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Buffers.txt b/doc/wiki/Design.Buffers.txt new file mode 100644 index 0000000000..03542dc073 --- /dev/null +++ b/doc/wiki/Design.Buffers.txt @@ -0,0 +1,65 @@ +Buffers +======= + +'lib/buffers.h' describes Dovecot's buffer API. Unless your code happens to be +VERY performance critical, you shouldn't handle writing to buffers/arrays +manually, but instead use the buffer API's safe functions to guarantee that +your code can't write past the buffer and cause a security hole. + +Dovecot's buffers are the basic building block for [Design.Arrays.txt] +and [Design.Strings.txt]. Use them instead if they make more sense +than buffers. + +There are a two different ways to create buffers: statically and dynamically +allocated. + +Static buffers +-------------- + +You can create statically allocated buffers with 'buffer_create_data()'. Trying +to write past the given buffer size will panic. The code to initialize this +looks like: + +---%<------------------------------------------------------------------------- +unsigned char buf_data[1024]; +buffer_t buf; + +buffer_create_data(&buf, buf_data, sizeof(buf_data)); +---%<------------------------------------------------------------------------- + +Trying to write more than 1024 bytes to the buffer will cause an assert-crash, +so these buffers shouldn't be used unless you know exactly what the maximum +buffer size is. + +To avoid accidental buffer overflows, don't use any more complex calculations +in the size parameter of 'buffer_create_data()'. It should always be +'sizeof(data_buffer)'. + +You can also create non-writable buffers with 'buffer_create_const_data()'. +Static buffers don't need to be freed. + +Dynamic buffers +--------------- + +Dynamically growing buffers can be created with 'buffer_create_dynamic(pool, +init_size)'. Memory for buffer is allocated from the given pool. When memory +needs to be grown, it's grown exponentially (2^n), with some exceptions to +avoid growing the given memory pool unless necessary. The initial buffer size +is always a guess - try to make it large enough that buffer wouldn't be grown +most of the time, but not so large that it wastes memory. + +You should be careful with memory returned by 'buffer_get_space_unsafe()' and +'buffer_append_space_unsafe()'. This returned memory should be accessed +immediately afterwards and it must not be accessed anymore after other +'buffer_*()' calls, because they may reallocate the buffer and move it +elsewhere in memory. + +Buffers always look like they're filled with NUL bytes. If you write past the +end of buffer, all the inserted bytes are filled with NULs. If you shrink the +buffer with 'buffer_set_used_size()' and again write past the end of used size, +all the old data is again gone and filled with NULs. If you for some reason +want to just temporarily shrink the buffer size and then change it back, you +can use 'buffer_set_used_size()' to grow it back to its original size (but no +larger). + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Code.txt b/doc/wiki/Design.Code.txt new file mode 100644 index 0000000000..5fbddbacf2 --- /dev/null +++ b/doc/wiki/Design.Code.txt @@ -0,0 +1,203 @@ +Code Design +=========== + +Generally Dovecot follows Linux kernel coding style +[https://www.kernel.org/doc/Documentation/CodingStyle]. + +Most of the coding style design is about getting as many compiler warnings and +errors as possible. Dovecot already has some patched-Clang-specific features to +get more warnings, and will likely have more in future. Issues found by static +analyzers should also be fixed in some way, e.g. adding extra asserts. + +Runtime errors should be caught by asserts or NULL pointer dereferences, which +cleanly crash the program instead of it continuing and possibly corrupting data +or causing other bad things. Dovecot's master process restarts the crashed +processes anyway. Obviously the master process should be very careful to avoid +crashing, but even then in some OSes a crashed Dovecot master gets restarted by +the init process. + +Dovecot's bottlenecks are primarily disk I/O and secondarily memory usage, so +using extra CPU for asserts and other extra checks costs practically nothing. + +Types +----- + + * Unsigned types are used whenever the value isn't allowed to be negative. + This makes it easier to do "value too large" checks when you don't also have + to check for negative values. Also in arithmetic it's better to have the + value wrap (and hopefully checked later!) than cause undefined behavior with + a signed integer overflow. + * 'char *' always points to a NUL-terminated string, 'unsigned char *' + doesn't. + * 'size_t' should generally be used when pointing to a large memory area, + especially for 'mmap()'. Since 'size_t' can be slower to access than + 'unsigned int' (or at least use memory), it's fine to use 'unsigned int' + when it's "very unlikely" that the size ever goes beyond 4 GB + (e.g.'string_t'). + * 'uoff_t' is used for file offsets/sizes. This is usually 64bit, even with + 32bit machines. + * 'uint32_t' vs. 'unsigned int': Use 'uint32_t' when the type really should be + 32bit, but don't spend too much energy trying to avoid mixing it with + 'unsigned int', since they are going to be the same types probably for the + rest of Dovecot's life.. + * 'uint8_t' vs. 'unsigned char': I doubt Dovecot will ever be compiled + anywhere where these differ from one another, but for readability use + 'uint8_t' for binary data and 'unsigned char' for text data. + +Function parameters +------------------- + + * Try to avoid using/allowing NULL pointers. For example for a public API + instead of having 'foo(struct bar *bar)' where bar can be NULL, you could + have both 'foo(void)' and 'foo_with_bar(struct bar *bar)'. Of course don't + try too hard if it makes the API otherwise ugly. + * Dovecot v2.2+ marks all such parameters with ATTR_NULL. These can be + verified with a patched clang. Unfortunately the API isn't very nice, it + would be better to mark each parameter separately with ATTR_NULL instead + of having a numbered list. This will likely change in future. + * Also unless you know that a parameter can be NULL, don't bother wasting + code on checking if it is. Ideally those are noticed by the patched clang + check, but even if not, it's not that bad to crash on NULL pointer + dereference. + * '_r' suffix in parameter is used for values that are returned (e.g. + 'foo(const char **error_r)'). + * Use const for pointers pretty much whenever possible. + * Some day in future I'd like compiler to give warnings if function parameters + are modified by the function itself. There's probably a lot of code that + does this currently, but try avoiding it in future. + +Function return values +---------------------- + + * 'int' type is commonly used for return values. Usually this is used to mean + one of: + * -1 = error, 0 = ok + * -1 = error, 0 = unfinished (e.g. non-blocking call), 1 = finished + * unfortunately there are also other uses. I've been wondering about using + different types for these some day, such as "err" and "err3", but haven't + figured out an easy enough to use design, especially one where compiler + verifies that types aren't accidentally mixed. + * 'bool' shouldn't be used as return value for ok/error, use int 0/-1 instead. + (Old code has some of these. Sometimes it's also not quite clear if + something is an "error" or not, so this rule isn't perfect.) + * Functions returning pointers shouldn't use NULL pointer to mean an error, + only for things like "not found". For example instead of 'if ((ctx = init()) + == NULL)' use 'if (init(&ctx) < 0)' + * Usually when calling a function, either save/check the return value or + explicitly say that you don't care about it by saying '(void)func();' + * Some functions where the return value can nearly always be ignored it's + annoying to add the '(void)' prefix. You can avoid these by adding + ATTR_NOWARN_UNUSED_RESULT to such function's prototype. A patched clang + can be used to find missing return value checks. + * If a system call fails unexpectedly, always log an error about it, possibly + even kill the process if it's vital for correct functionality + (e.g.'gettimeofday()'). '%m' in Dovecot's 'printf()'-style functions expands + to 'strerror(errno)'. + +Boolean expressions +------------------- + +Try to use boolean expressions the way they work in Java. C doesn't require +this, but I think it makes the code easier to understand and reduces bugs in +some cases (e.g.'if (!foo())' when thinking foo() returns bool/FALSE, but +actually returns int/-1 on error). We've a clang patch +[http://dovecot.org/patches/clang/] to give warnings in these cases. As +expected, it found quite a lot of bugs (some real bugs and a lot of "it just +accidentally worked" +[https://github.com/dovecot/core/commit/d9a7e950a9cd21f2b4a90ec7759fca9e8fcc7995]). + + * 'bool x' and 'bool x:1' are the boolean types + * TRUE and FALSE are the only valid explicit boolean values (not 0 or 1, and + currently also not true/false although that could be changed) + * !=, ==, <, >, etc. comparisons create a boolean + * if, for, while, etc. require a boolean + +So: + + * 'if (!ptr)' -> 'if (ptr == NULL)' + * 'if (!num)' -> 'if (num == 0)' + * 'if (flags & FLAG_FOO)' -> 'if ((flags & FLAG_FOO) != 0)' + +Memory +------ + +Memory is always allocated through one of Dovecot's wrappers, e.g. 'i_malloc()' +or 'i_new()'. All of Dovecot's memory allocations always succeed or kill the +process. There's no point in writing a lot of code to check for memory +allocation failures that happen just about never. The only reason some memory +allocations fail in Dovecot currently is because a process VSZ limit is +reached, which usually indicates either a memory leak or trying to access a +mailbox that is too large. In either of these cases it's better to just +completely restart the process than try to limp along without getting anything +useful done anymore. + +Memory allocations can be assumed to be zero-initialized. All of the memory +allocation functions do it, except 't_malloc()' and 't_buffer_get()', which you +should almost never use directly anyway. The code currently also assumes that +pointers in zero-initialized memory area are NULL, which isn't guaranteed by +ANSI-C, but practically Dovecot isn't going to be run in systems where it's not +true and you're not going to remember to NULL-initialize all of your pointers +anyway without compiler/runtime failure. + +When using a struct, always zero-initialize it with 'memset()' instead of +setting each field separately to 0. It's too easy to cause bugs by adding a new +field to the struct and forgetting to initialize it. + +Double-frees in C are bad. Better to crash with NULL pointer dereference than +possibly allow an attacker to exploit heap corruption and run executable code. +Most of the pointers are set to NULL when they are freed: + + * 'i_free_and_null()' is a macro to free the memory and set the pointer to + NULL. + * 'i_free()' also currently sets the pointer to NULL, but another thought was + to set this pointing to some other invalid pointer. But arguably this should + be defined to be exactly the same as 'i_free_and_null()'. + * Most deinit() functions take a pointer-to-pointer parameter and set the + original one to NULL. There's no need to explicitly set the same pointer to + NULL afterwards. + +Buffers +------- + +Use dynamically growing strings/buffers wherever necessary instead of a static +sized buffer, where on larger input the function fails or truncates the data. +It's of course not good to allow users to infinitely grow memory usage, so +there should be some limits added, but it shouldn't fail even if the limit is +set to infinite. + +Avoid explicitly calculating memory usage for allocations. If you do, mark it +with '/* @UNSAFE */' comment unless it's the calculation is "so obvious that +you see it's correct at the first glance". If in doubt, just mark it UNSAFE. +The idea is that anyone can easily grep for these and verify their correctness. + +Type safety +----------- + + * Try to avoid using void pointers. + * Try to avoid casting types to other types, especially if the cast isn't + necessary to avoid a compiler warning. + +It's better if compiler can give a warning when something is accidentally used +wrong. + +Callback functions +------------------ + +Callback functions make the code more difficult to follow, especially when a +callback calls another callback, or when using function pointers to jump to +different callbacks depending on state. Of course with asynchronous C code it's +pretty much impossible to avoid callbacks. Still, try to avoid them where +possible to keep the code readable. lib-fs/fs-api.h is an example API which +supports async operations but with a single common "do more now" callback +rather than every single operation having its own callback parameter. This +makes it similar to async network IO with read()/write() EAGAIN handling. + +Often callback functions can be avoided by creating iterator functions instead. +For example instead of 'parse(callback, context' use 'ctx = parse_init(); while +(parse_next(ctx)) { .. } parse_deinit(&ctx);' + +Dovecot has some helper macros to make callbacks' context parameters type-safe. +In v2.2+ see 'CALLBACK_TYPECHECK()' macro and for example 'io_add()' for +example usage. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Dcrypt.txt b/doc/wiki/Design.Dcrypt.txt new file mode 100644 index 0000000000..e17f017bbe --- /dev/null +++ b/doc/wiki/Design.Dcrypt.txt @@ -0,0 +1,113 @@ +lib-dcrypt +========== + +lib-dcrypt is component for abstracting asymmetric and symmetric cryptographic +operations. It can be used for public/private key handling too. Currently we +support OpenSSL backing, but it is possible to write alternative backends for +dcrypt. + +ECDH algorithm +-------------- + +ECDH (Elliptic curve Diffie-Hellman) is widely used in lib-dcrypt for both key +and data storage. This algorithm is also known as ECIES +[https://en.wikipedia.org/wiki/ECIES] (Elliptic curve Integrated Encryption +Scheme). + +When encrypting data, we perform following steps, this is the currently used +algorithm. There is also a legacy algorithm, but since that has not been used +publicly, we do not describe it here. You can deduce it from the code if you +want to. + +ENCRYPT(RECIPIENT-KEY, DATA): + + 1. Ensure recipient key is not point at infinity + 2. Generate new keypair from same group + 3. Choose ephemeral public key as R + 4. Calculate P = R * RECIPIENT-KEY + 5. From P = (x,y) choose x as S + 6. Generate random salt + 7. Use PBKDF2(SHA256, S, salt, 2000) to produce iv+key, and hmac seed or aad + 8. Encrypt data + 9. OUTPUT R, salt and encrypted data. + +In dcrypt-openssl.c, we use EVP_PKEY_derive_* for the actual derivations. +ephemeral public key is exported with EC_POINT_point2oct in compressed form. + +DECRYPT(PRIVATE-KEY, R, SALT, DATA): + + 1. Ensure R is not point at infinity + 2. Calculate P = R * PRIVATE-KEY + 3. From P = (x,y) choose x as S + 4. Use PBKDF2(SHA256, S, salt, 2000) to produce iv+key, and hmac seed or aad + 5. Decrypt data + 6. OUTPUT decrypted data + +Key formats +----------- + +lib-dcrypt can consume keys in PEM format [https://tools.ietf.org/html/rfc1421] +(with or without password), and in Dovecot's special format intended for dict +storage. + +Dovecot's format consists from unencrypted and encrypted keys. You can encrypt +keys using password or another key. There are also two version, version 1 +(deprecated) and version 2 (current). Both versions support either tab or : +separated fields. ECC keys are stored always in compressed form. Version 1 +format is not described as it's deprecated and should not be used. + +Version 2 format +---------------- + +---%<------------------------------------------------------------------------- +public key id: HEX(SHA256(public key in DER format)) +key data: + RSA: i2d_PrivateKey + ECC: BN_bn2mpi using compressed form + +public key: 2:HEX(public key in DER format):public key ID +private key (unencrypted) : 2:key algo oid:0:key data:public key ID +private key (encrypted, key) : 2:key algo oid:1:symmetric algo:salt:digest algo +(for pbkdf2):rounds:encrypted key data:ephemeral public key:digest of +encryption key:public key ID +private key (encrypted, pwd) : 2:key algo oid:2:symmetric algo:salt:digest algo +(for pbkd2f):rounds:encrypted key data:public key ID +---%<------------------------------------------------------------------------- + +File format +----------- + +This library can also generate encrypted files that are encrypted using +asymmetric key pair. File encryption can be done using whatever algorithm(s) +the underlying library supports. For integrity support, either HMAC based or +AEAD based system is used when requested. + +File format is described below + +---%<------------------------------------------------------------------------- +000 - 008 CRYPTED\x03\x07 (MAGIC) +009 - 009 \x02 (VERSION FIELD, 2) +010 - 013 MSB flags +014 - 017 MSB total header length (starting from 000) +018 - cod cipher oid in DER format +cod - mod MAC algorithm oid in DER format +mod - +4 MSB PBKDF2 rounds ++5 - +8 MSB length of key data ++9 - +9 number of key blocks +----- key block ----- ++10 - +10 key type (1 = RSA, 2 = ECC) ++11 - +43 public key id (SHA256 of public key in DER format, point compressed) ++44 - +48 MSB length of ephemeral key ++49 - epk ephemeral key +epk - +4 MSB length of encrypted key ++4 - ek encrypted key +----- end of key block (this can then repeat) ----- +eokb - +4 MSB length of encryption key hash ++4 - ekh encryption key hash +ekh - (eof-maclen) payload data +---%<------------------------------------------------------------------------- + +There is a small script for decrypting these files, see +[attachment:decrypt.rb]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.DoveadmProtocol.HTTP.txt b/doc/wiki/Design.DoveadmProtocol.HTTP.txt new file mode 100644 index 0000000000..a9718e421f --- /dev/null +++ b/doc/wiki/Design.DoveadmProtocol.HTTP.txt @@ -0,0 +1,113 @@ +Doveadm HTTP API +================ + +Doveadm HTTP API is available since v2.2.22. It is considered *experimental* in +v2.2.22. Can be considered as stable since 2.2.23. It lets you perform doveadm +commands over HTTP transport. + +Configuration +------------- + +To enable HTTP API, add following to your config file: + +---%<------------------------------------------------------------------------- +service doveadm { + inet_listener http { + port = 8080 + #ssl = yes # uncomment to enable https + } +} +---%<------------------------------------------------------------------------- + +To enable SSL make sure 'ssl=yes' or 'ssl=required' in global settings, and set +'ssl=yes' in the listener. + +You can use unix listener too, and define host to listen on. You also need to +either define 'doveadm_password', or 'doveadm_api_key'. With +'doveadm_password', the username is doveadm. This is going to change in future +release. With API Key you are expected to send + +---%<------------------------------------------------------------------------- +Authorization: X-Dovecot-API Base64(apikey) +---%<------------------------------------------------------------------------- + +header to access the API. (In v2.2.22-2.2.23, this key was incorrectly +"X-Doveadm-API".) + +Usage +----- + +You can see valid commands and their parameters by accessing +'http://host:port/doveadm/v1'. + +To send command(s), you can send following with 'Content-Type: +application/json' + +---%<------------------------------------------------------------------------- +[ + ["command", {"parameter":"value"}, "optional identifer"] +] +---%<------------------------------------------------------------------------- + +In following examples, you can either use Basic or X-Dovecot-API authorization. +X-Dovecot-API usage: + +---%<------------------------------------------------------------------------- +curl -H "Authorization: X-Dovecot-API " +---%<------------------------------------------------------------------------- + +Basic authorization uses "doveadm" as the username, and doveadm_password +setting as the password: + +---%<------------------------------------------------------------------------- +curl -H "Authorization: Basic " +---%<------------------------------------------------------------------------- + +To get acceptable routes + +---%<------------------------------------------------------------------------- +curl -H "Authorization: Basic " +http://server:8080/ +---%<------------------------------------------------------------------------- + +To get acceptable commands and their parameters + +---%<------------------------------------------------------------------------- +curl -H "Authorization: Basic " +http://server:8080/doveadm/v1 +---%<------------------------------------------------------------------------- + +an example command would be + +---%<------------------------------------------------------------------------- +curl -H "Content-Type: application/json" -H "Authorization: Basic " -d +'[["fetch",{"user":"username","field":["uid"],"query":["mailbox","INBOX"]},"c01"]]' +http://server:8080/doveadm/v1 +---%<------------------------------------------------------------------------- + +You can have multiple commands in the array, but for now it is safest *not* to +do so, as some commands may kill the server in certain error conditions and +leaving you without any response. + +Responses are in same format, command is replaced with either "error" or +"doveadmResponse" and parameters with array of response variables. In case of +*successful* command which has *no* output, response is going to be []. + +Errors are indicated with "error" and type, exit-code in response part. + +Logging +------- + +The logs will indicated execute command(s) and also Apache access.log format +strings on requests. In case of fatal errors, the access.log string might be +missing. + +Example clients +--------------- + + * A PoC API client/library written in Python is available at + https://github.com/hnsk/doveadm-http-cli + * See also internal doveadm protocol clients in + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.DoveadmProtocol.txt b/doc/wiki/Design.DoveadmProtocol.txt new file mode 100644 index 0000000000..10d8bd5a4d --- /dev/null +++ b/doc/wiki/Design.DoveadmProtocol.txt @@ -0,0 +1,95 @@ +Doveadm protocol +================ + +See also [Design.DoveadmProtocol.HTTP.txt]. + +doveadm-server can be accessed via UNIX sockets or TCP protocol (by adding +inet_listener to doveadm service). The protocol looks like: + +Initial handshake from client to server: + +---%<------------------------------------------------------------------------- +C: VERSION doveadm-server 1 0 +---%<------------------------------------------------------------------------- + +Note that the spaces you see are TABs. All the fields are TAB-separated. The +server will send you back either: + + * "+" means you are preauthenticated and can start sending commands. This + happens when connecting to the UNIX socket. + * "-" means you need to authenticate first. + +Authentication +-------------- + +The authentication is done with a regular SASL PLAIN authentication, i.e. +"PLAINbase64(\0username\0password)". Currently the username must be +"doveadm". For example for user=doveadm, password=secret use: + +---%<------------------------------------------------------------------------- +C: PLAIN AGRvdmVhZG0Ac2VjcmV0 +S: + +---%<------------------------------------------------------------------------- + +Running commands +---------------- + +The actual commands are in format: flagsusernamecommand +name[parameter[parameter2...]], where the flags can be either empty, +"v" (verbose) or "D" (debug). Note that if the command name has spaces, they +are sent as spaces instead of as tabs (e.g. "quota get", not "quotaget"). +So for example to get a quota for user tss: + +---%<------------------------------------------------------------------------- +C: tss quota get +S: user STORAGE 1814 - 0 user MESSAGE 6 - 0 + +S: + +---%<------------------------------------------------------------------------- + +The storage values are all given in kilobytes. + +The server replies using the same fields TAB-separated as what a regular +doveadm command sends. The reply itself ends with LF. So if the reply is large, +it may return a very long line as a reply. After the reply follows a status +line: + + * "+" = success. + * In future the "+" may be followed by more text, for now you should just + ignore those. + * "-" = failed (the error was probably logged to Dovecot's error log) + * "-NOUSER" = the user doesn't exist + * Other "-SOMETHING" errors may be added in future. + +Available commands +------------------ + +The command names and output are exactly the same as what regular doveadm +commands on command line do. Currently only "mail commands" are available via +doveadm protocol, but this will change in future. + +You can use the doveadm itself to find out what the output format will look +like. For example: + +---%<------------------------------------------------------------------------- +doveadm -f tab search mailbox inbox 1:2 +mailbox-guid uid +fa8cb722dfad9c52b62600007049b30b 125159 +fa8cb722dfad9c52b62600007049b30b 125160 +---%<------------------------------------------------------------------------- + +There are two fields, "mailbox-guid" and "uid" in the output. The title names +won't be sent via doveadm protocol, but everything else will be sent in one +line. So in the above case the protocol output will be: + +---%<------------------------------------------------------------------------- +fa8cb722dfad9c52b62600007049b30b125159fa8cb722dfad9c52b62600007049b30b125160 +---%<------------------------------------------------------------------------- + +Example Clients +--------------- + + * Perl: Net::Doveadm [https://metacpan.org/pod/Net::Doveadm] + * See also HTTP protocol-based clients in + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Dsync.txt b/doc/wiki/Design.Dsync.txt new file mode 100644 index 0000000000..7b51c939ec --- /dev/null +++ b/doc/wiki/Design.Dsync.txt @@ -0,0 +1,114 @@ +Dsync Design +============ + +FIXME: This describes the design for v2.0/v2.1. The v2.2 design is somewhat +different. + +Two-way synchronization +----------------------- + +dsync attempts to preserve all changes done by both sides of the synced +mailboxes. + +Mailbox list +------------ + +Mailboxes have 128 bit globally unique IDs, which are used for figuring out +when two mailboxes should actually be synchronized. This solves two major +problems: + + * If mailbox has been renamed in one side, dsync finds it because its GUID + hasn't changed. + * If mailbox has been deleted and recreated, dsync doesn't attempt to sync it + because it's a different mailbox. + +Then there's the problem of how to correctly sync mailbox renames and +deletions. How do you know which side of the sync has the most recent name for +the mailbox? How do you know if one side had deleted mailbox, or if the other +side had created it? To solve these problems, Dovecot v2.0 created a "mailbox +log", which adds a record with mailbox GUID and timestamp whenever mailbox is +renamed or deleted. So: + + * If mailbox has different names on two sides, its "last renamed" timestamp is + looked up from the mailbox list index. The side with the most recent + timestamp is assumed to contain the newer name and the other side's mailbox + is renamed to it. + * If neither side has a "last renamed" timestamp, one side is picked. This + shouldn't happen, except when mailbox log is deleted for some reason or + if the renaming is done outside Dovecot. + * If mailbox exists only on one side, the other side checks if mailbox log + contains a delete record for its GUID. If there is one, the mailbox is + deleted from the other side. If there's not, the mailbox is created and + synced. + * Subscriptions and unsubscriptions are synced in a similar way. But because + it's possible to be subscribed to nonexistent mailboxes, mailbox log can't + contain mailbox GUIDs for them. Instead the first 128 bits of SHA1 of + mailbox name are used. Collisions for mailbox names are highly unlikely, but + even if one happens, the worst that can happen is that user gets + unsubscribed from wrong mailbox. + +dsync writes timestamps to changelog using the original timestamps, so that +dsync's changes won't override changes done by user during sync. + +Mailbox +------- + +When saving new mails, dsync preserves all of their immutable state: + + * GUID + * Received date + * Save date + * Message contents + +It also attempts preserve IMAP UID. This works as long as the other side hasn't +already used the UID for another mail. If it has, dsync doesn't attempt to +preserve the UID, because an IMAP client might have already seen the UID and +cached another mail's contents for it. IMAP requires that message's contents +must never change, so UIDs can't be reused. So whenever an UID conflict +happens, dsync gives messages in both sides a new UID, because it can't know +which message the client had seen, or perhaps user used two clients and both +saw a different message. (This assumes a master/slave replication use case for +dsync.) + +The mutable metadata that dsync preserves is: + + * Message flags and keywords + * Modification sequences (modseqs) + +Flags and keywords are synced based on modseqs. Whichever side has a higher +modseq for the message, its flags and keywords are synced to the other side. +Currently there's no per-flag or per-keyword synchronization, so that if one +side had added \Seen flag and other side had added \Answered flag, one of them +would be dropped. + +Finding what to sync +-------------------- + +dsync can run in full mode or fast mode. Full mode means it goes through all +messages in all mailboxes, making sure everything is fully synchronized. In +fast mode it relies on uidvalidity, uid-next and highest-modseq values to find +out changes. If any of the values changed, the mailbox is included in sync. + +FIXME: A superfast mode should still be implemented, where once a mailbox is +selected for syncing, it should sync only mails whose modseq is higher than a +given one. This would improve performance and network traffic with large +mailboxes. + +Copy optimizations +------------------ + +Before dsync actually starts syncing anything, it first fetched a list of all +to-be-synced messages and adds them to a GUID -> message hash table. Whenever +dsync needs to sync a new message to the other side, it first checks if the +message's GUID already exists on the other side. If it does, it starts a +message copy operation instead of a full save. It's possible that this copy +operation fails if the message just gets expunged from the other side, so there +needs to be fallback handling for this. If the message exists in multiple +mailboxes, a copy from the next mailbox is attempted. If all of them fail, +dsync fallbacks to saving the message. + +FIXME: This optimization currently works only in full sync mode. If this were +to work in fast sync mode, the full mailbox list would have to be looked up +from local side. And this would slow it down.. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Events.txt b/doc/wiki/Design.Events.txt new file mode 100644 index 0000000000..c6df74034c --- /dev/null +++ b/doc/wiki/Design.Events.txt @@ -0,0 +1,264 @@ +Events (v2.3+) +============== + +Dovecot v2.3 introduces "events", which improves both logging and statistics. +See for list of all events. + +Each logging call can be attached to a specific event, which can provide more +metadata and context than just the log message string. This will eventually +allow implementing things like machine-parseable (e.g. JSON) log lines +containing key=value pairs, while still keeping the human readable text +available. Each logging event can also be captured and sent to stats, even if +it's not actually logged. Commonly statistics-related events are logged with +debug level. + +Events have: + + * Categories, such as "storage", "mailbox" or "auth". + * Fields, such as 'user=foo@example.com' or 'service=imap'. + * Creation timestamp with microsecond precision. + * Source code file and line number location when sending the event. + * It may have an easy human-readable name. This is important for events that + are expected to be used for statistics, so they can be easily referred to. + * "Forced debug"-flag. Debug logging is enabled for this event regardless of + the global debug log filters. A child event will inherit this flag. + +Events are hierarchical, so they can have parent events. The events always +inherit all of their parents' categories and fields. A child event can replace +a parent's field, and it can also remove a parent's field with +'event_field_clear()'. Ideally most events would have a parent hierarchy that +reaches the top event that was created for the current user/session. This +allows statistics to track which events happened due to which users. In some +cases this may not really be possible, such as an HTTP connection that is +shared across multiple users in the same process. Generic libraries should take +the parent event in function parameters or in a settings struct or similar. + +An event's lifetime is usually the same as the "object" it attaches to. For +example an IMAP client connection should have a single event created at the +beginning of the connection and destroyed at disconnection. The IMAP client +connection event could be used for logging things like "Client connected" and +"Client disconnected" and perhaps some other connection-specific events. +However, most of the logging should be done by new events that have the IMAP +client connection event as their parent. For example IMAP command event should +exist during the execution of the IMAP command, and its parent should be the +IMAP client connection event. Note that there's an automatic "duration" +statistics field that is calculated from the creation of the event to the +(last) sending of the event, so for it to make sense the event lifetime and its +logging also needs to make sense. So for example if the IMAP client connection +event was used for logging many things throughout the session, the "duration" +field would make little sense for most of those events. + +Events are "sent" by logging it. Any e_debug(), e_info(), e_warning() or +e_error() call will also send the event, which may be redirected to the stats +process. Often events that are intended for statistics are sent using the +e_debug() call. The event can be sent to statistics even if it's not actually +logged. Avoid sending events excessively. For example an e_debug() call every +time connection reads or writes something will likely result in a huge amount +of unnecessary debug logging. + +Event names +----------- + +Events that are expected to be used in statistics should have a name. Be +consistent when naming the events. The name's prefix should be the subsystem +that is logging the event. Usually this would be the primary category of the +event. For example imap related events should begin with "imap_" and mailbox +related events begin with "mailbox_". + +The name should consist of only [a-z], [0-9] and '_' characters. + +Current naming conventions for name suffixes: + + * _connected (for connections) + * _disconnected (for connections) + * _finished (when some operation finishes, e.g. IMAP command or HTTP request) + * e.g. http_request_finished, dns_request_finished, imap_command_finished + * This should be used regardless of whether the operation succeeded or + failed. The details would be in fields. + * _retried (if an operation is internally retried one or more times before + it's finished) + +Categories +---------- + +The event categories are hierarchical. For example "mail" category has parent +"mailbox", which has parent "storage". If an event filter contains +"category:storage", it will match the "mail" and "mailbox" child categories as +well. + +Note that a category isn't the same as a service/process name. So for example +imap process has an "imap" category for its IMAP-related events, such as IMAP +client connection and IMAP command related events. Because most events would be +child events under these IMAP events, they would all inherit the "imap" +category. So it would appear that using "category:imap" filter would match most +of the logging from imap process. However, there would likely be some events +that wouldn't have the IMAP client as their parent event, so these wouldn't +match the imap category. + +The same category name must not be duplicated within the process. This is +because event handling is optimized and performs category checking by comparing +the categories' pointers, not names' strings. (Then again, if the struct +event_category variable names were consistent, you'd get duplicate symbol +errors from linker as well.) + +Be careful naming events that go through client and server boundaries. For +example if both lib-dns and dns service use "dns" as their category and also +have identically named "dns_lookup" event, there's no easy way to differentiate +in event filters between these two. So a statistics filter could end up +counting each DNS lookup twice. Since it's more difficult to remember to check +for event naming conflicts, it would be safer to use different category names +entirely. + +The category name should consist of only [a-z], [0-9] and '_' characters. + +Fields +------ + +Each event can have any number of key=value fields. Parent event's fields are +inherited by the child event. + +There are 3 types of fields: + + * strings + * numbers (intmax_t = signed 64bit usually) + * timestamp (struct timeval) + +The fields can be used for various purposes: + + * Filtering events with field_name=value matching + * Counting fields in statistics (most commonly number fields) + * They can include metadata that are internally used by the code. For example + passing data from one plugin to others. + * Later on these fields can be used by the logging system. + +Field names should be consistent across the code. Besides making it easier for +admins to configure the events, this allows statistics code to sum up fields +from different unrelated events. For example if all the networking events +include "ip", "bytes_in" and "bytes_out" fields, statistics can globally track +how much network traffic Dovecot is doing from its own point of view, +regardless of whether it's HTTP traffic or IMAP traffic or something else. + +Current naming conventions: + + * The name should consist of only [a-z], [0-9] and '_' characters. + * Timestamps should have "_time" suffix + * Durations should have "_usecs" suffix and be in microseconds. + * Try to avoid adding extra duration fields for most events. There's the + automatic "duration" field already that contains how long the event has + existed. So usually the event lifetime should be the same as the wanted + duration field. + * Incoming TCP/IP connections should have "remote_ip", "remote_port", + "local_ip" and "local_port" fields + * Outgoing TCP/IP connections should have "ip" and "port" for the remote side. + + * For local side "client_ip" and "client_port" may optionally be used + * NOTE: These are all different from incoming connection's IP/port fields. + This is because often everything starts from an incoming connection, + which will be used as the root event. So we may want to filter e.g. + outgoing HTTP events going to port 80 which were initiated from IMAP + clients that connected to port 993 (port=80 local_port=993) + * Connection reads/writes should be counted in "bytes_in" and "bytes_out" + fields + * These fields were chosen over e.g. network_in/out because a lot of code + is rather generic and can work over TCP/IP or UNIX sockets, or maybe even + any other kind of iostreams. Using a generic bytes_in/out makes it + simpler to count these. If further differentiation is wanted on + statistics side, networking events can be filtered out with "ip". + * These fields are usually easiest updated with 'event_add_int(event, + "bytes_in", istream->v_offset)' and 'event_add_int(event, "bytes_out", + ostream->offset)'. If iostreams aren't used, 'event_inc_int()' maybe be + easier. + * (Local) disk reads should have "disk_read" and "disk_write" fields + * With remote filesystems like NFS it may be difficult to differentiate + between disk IO and network IO. Generally the disk_read/write should be + used for POSIX read() and write() calls from filesystem. + * Counting only read()s and write()s doesn't necessarily translate to + actual disk IO since it may only be accessing the kernel page cache. + Still, this may be useful. + * There is a lot of disk IO performed all over the code, so Dovecot will + likely never include events for all disk reads/writes. + * error= : The operation failed. The may be simply "y" or + contain more details. This field shouldn't exist at all for successful + operations. + * error_code= : Machine-readable error code for a failed operation. If + set, the "error" field must also be set. + +Note that events shouldn't be sent every time when receiving/sending network +traffic. Instead, the bytes_in/out fields should be updated internally so that +whenever the next event is sent it will have an updated traffic number. + +Generally it's not useful for events to be counting operations. Rather each +operation should be a separate event, and the statistics code should be the one +counting them. This way statistics can only be counting e.g. operations with +duration> 1 sec. If the statistics code was seeing only bulk operation counts +this wouldn't be possible. The bytes_in/out and such fields are more of an +exception, because it would be too inefficient to send individual events each +time those were updated. + +Note that even though internally updating a field for an event's parent will be +immediately visible to its children, the update won't be automatically sent to +the stats process. We may need to fix this if it becomes a problem. + +Field inheritance may become problematic also when multiple nested ioloops are +used. For example an outgoing imapc connection could receive a reply, which +synchronously triggers an outgoing quota SQL connection. The quota SQL +connection's parent event likely shouldn't be the imapc connection's event, +because otherwise they could be mixing the IP/port fields and perhaps others. +This isn't necessarily a problem though, but this is why when connection.c +performs outgoing UNIX socket connection it clears the IP/port fields to make +sure they don't exist for the connection event due to inheritance from a parent +event. + +Passthrough events +------------------ + +Passthrough events' main purpose is to make it easier to create temporary +events as part of the event parameter in e_error(), e_warning(), e_info() or +e_debug(). These passthrough events are automatically freed when the e_*() call +is finished. Because this makes the freeing less obvious, it should be avoided +outside e_*()'s event parameter. + +A passthrough event's creation timestamp is the same as the parent event's +timestamp, because its intention is to only complement it with additional +fields. This way the generated event "duration" field is preserved properly. + +The passthrough events also change the API to be more convenient towards being +used in a parameter. Instead of having to use e.g. + +---%<------------------------------------------------------------------------- +event_add_str(event_set_name(event_create(parent), "name"), "key", "value") +---%<------------------------------------------------------------------------- + +The event_passthrough API can be a bit more readable as: + +---%<------------------------------------------------------------------------- +event_create_passthrough(parent)->set_name("name")->add_str("key", +"value")->event(). +---%<------------------------------------------------------------------------- + +The passthrough event is converted to a normal event at the end with the +event() call. Note that this API works by modifying the last created +passthrough event, so it's not possible to have multiple passthrough events +created in parallel. + +Log prefixes +------------ + +Events allow replacing the current log prefix or appending to it. This way for +example opening a mailbox can add a "Mailbox: " prefix and then use +'e_debug(box->event, ...)' without having to specify the mailbox name in every +log message. + +Global events +------------- + +Sometimes there's not really any specific event that a log message would belong +to, or it would be difficult to transfer the event there. In these cases the +old i_debug(), i_info(), i_error(), etc. logging calls can still be used. These +will be using the global event and its logging prefix. + +The global events are pushed/popped in a stack. For example with IMAP the +initial global event is the user's event. During IMAP command execution the +global event is the IMAP command event. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Indexes.Cache.txt b/doc/wiki/Design.Indexes.Cache.txt new file mode 100644 index 0000000000..45e75a724d --- /dev/null +++ b/doc/wiki/Design.Indexes.Cache.txt @@ -0,0 +1,137 @@ +Cache file +========== + +Cache file is used for storing immutable data. It supports several different +kinds of fields: + +MAIL_CACHE_FIELD_FIXED_SIZE: + The field size doesn't need to be stored in the cache file. It's always the + same. + +MAIL_CACHE_FIELD_BITMASK: + A fixed size bitmask field. It's possible to add new bits by updating this + field. All the added fields are ORed together. + +MAIL_CACHE_FIELD_VARIABLE_SIZE: + Variable sized binary data. + +MAIL_CACHE_FIELD_STRING: + Variable sized string. + +MAIL_CACHE_FIELD_HEADER: + Variable sized message header. The data begins with a 0-terminated 'uint32_t + line_numbers[]'. The line number exists only for each header, header + continuation lines in multiline headers don't get listed. After the line + numbers comes the list of headers, including the "header-name: " prefix for + each line, LFs and the TABs or spaces for continued lines. + +The last 3 variable sized fields are treated identically by the cache file +code. Their main purpose is to make it easier for "dump cache file's contents" +programs ('src/util/idxview') to do their job. + +Locking +------- + +Because cache file is typically used in potentially long-running operations, +such as with IMAP command 'FETCH 1:* (BODY.PEEK[] ENVELOPE BODYSTRUCTURE)' it's +important that updating the cache file doesn't block out any other readers. +Also because the readers are often also writers (if something isn't cached, +it's added there), it's important that they don't block writers either. + +Reading cache files requires no locking. Writing is done by first locking the +file, reserving some space to write to, and immediately after that unlocking +the file. This way the transaction can keep writing to the cache file as long +as it wants to without blocking other writers. When the transaction is +committed, the updated cache offsets are written to the transaction log which +makes them visible to other processes. + +This also means that it's possible for two processes to write the same cached +fields twice to the cache file. Because the data written to the cache file are +really just cached data, the fields' contents are identical. Having the data +exist twice (or even more times) means wasting some disk space, but otherwise +it isn't a problem. The duplicates are dropped the next time the file is +compressed. + +Cache decisions +--------------- + +Dovecot tries to be smart about what it keeps in the cache file. If the client +never fetches the cached data, it's just waste of disk space and disk I/O. + +The caching decisions are: + +MAIL_CACHE_DECISION_NO: + This field isn't cached currently. + +MAIL_CACHE_DECISION_TEMP: + This field is cached for new mails. + +MAIL_CACHE_DECISION_YES: + This field is cached for all mails. + +Normally Dovecot changes the decisions based on what fields are fetched and for +what messages. A specific decision can be forced by ORing it with +'MAIL_CACHE_DECISION_FORCED'. + +'mail-cache-decisions.c' file contains the rules how Dovecot changes the +decisions. The following is copied from the file: + +Users can be divided to three groups: + + 1. Most users will use only a single IMAP client which caches everything + locally. For these users it's quite pointless to do any kind of caching as + it only wastes disk space. That might also mean more disk I/O. + 2. Some users use multiple IMAP clients which cache everything locally. These + could benefit from caching until all clients have fetched the data. After + that it's useless. + 3. Some clients don't do permanent local caching at all. For example Pine and + webmails. These clients would benefit from caching everything. Some locally + caching clients might also access some data from server again, such as when + searching messages. They could benefit from caching only these fields. + +After thinking about these a while, I figured out that people who care about +performance most will be using Dovecot optimized LDA anyway which updates the +indexes/cache immediately. In that case even the first user group would benefit +from caching the same way as second group. LDA reads the mail anyway, so it +might as well extract some information about it and store them into cache. + +So, group 1. and 2. could be optimally implemented by keeping things cached +only for a while. I thought a week would be good. When cache file is +compressed, everything older than week will be dropped. + +But how to figure out if user is in group 3? One quite easy rule would be to +see if client is accessing messages older than a week. But with only that rule +we might have already dropped useful cached data. It's not very nice if we have +to read and cache it twice. + +Most locally caching clients always fetch new messages (all but body) when they +see them. They fetch them in ascending order. Noncaching clients might fetch +messages in pretty much any order, as they usually don't fetch everything they +can, only what's visible in screen. Some will use server side sorting/threading +which also makes messages to be fetched in random order. Second rule would then +be that if a session doesn't fetch messages in ascending order, the fetched +field type will be permanently cached. + +So, we have three caching decisions: + + 1. Don't cache: Clients have never wanted the field + 2. Cache temporarily: Clients want this only once + 3. Cache permanently: Clients want this more than once + +Different mailboxes have different decisions. Different fields have different +decisions. + +There are some problems, such as if a client accesses message older than a +week, we can't know if user just started using a new client which is just +filling its local cache for the first time. Or it might be a client user hasn't +just used for over a week. In these cases we shouldn't have marked the field to +be permanently cached. User might also switch clients from non-caching to +caching. + +So we should re-evaluate our caching decisions from time to time. This is done +by checking the above rules constantly and marking when was the last time the +decision was right. If decision hasn't matched for two months, it's changed. I +picked two months because people go to at least one month vacations where they +might still be reading mails, but with different clients. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Indexes.MailIndexApi.txt b/doc/wiki/Design.Indexes.MailIndexApi.txt new file mode 100644 index 0000000000..e7f597bd92 --- /dev/null +++ b/doc/wiki/Design.Indexes.MailIndexApi.txt @@ -0,0 +1,47 @@ +Mail Index API +============== + +'lib-index/mail-index.h' contains the functions to access the index files. +'mail-cache.h' contains the functions to access the cache file. + +The purpose of the main structures are: + + * 'struct mail_index': Global state of the index. + * 'struct mail_index_view': You can have multiple views to the index. The + views see new messages come and expunged messages go only when it's being + explicitly synchronized. With mmaped indexes you can't really trust the + record data (flags, keywords, extensions) not to change. This doesn't matter + with IMAP. + * 'struct mail_index_map': Index file is accessed via maps. Views can point to + one or more maps. Maps can be shared by different views. Maps can contain + either mmap()ed memory areas pointing to the index file, or a in-memory copy + of it. + * 'struct mail_index_transaction': In-memory list of changes to be written to + the transaction log. The writing is done only when the transaction is + committed. + +Views and maps +-------------- + +In general you access all the data in the index files via views. The mails are +accessed using sequence numbers, which change only when the view is +synchronized. + +For accessing messages with their UIDs, you'll first need to convert them to +sequences with either 'mail_index_lookup_uid()' or +'mail_index_lookup_uid_range()'. + +'mail_index_lookup()' can be used to look up a single record's UID and flags. +The returned record points to the latest map, so that it contains the latest +flag changes. If the message was already expunged from the latest map, it +returns 0. + +'mail_index_lookup_full()' can be used to get also the map where the message +was found. This can be important with extensions. If extension record's state +depends on the extension header, they must be looked up from the same map. For +this reason there exists 'mail_index_map_get_header_ext()' and +'mail_index_lookup_ext_full()' functions which take the map as parameter. The +non-map versions return the data from the latest map if the message hasn't been +expunged. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Indexes.MainIndex.txt b/doc/wiki/Design.Indexes.MainIndex.txt new file mode 100644 index 0000000000..db0fb0f21b --- /dev/null +++ b/doc/wiki/Design.Indexes.MainIndex.txt @@ -0,0 +1,284 @@ +Main index +========== + +The main index can be used to quickly look up messages' UIDs, flags, keywords +and extension-specific data, such as cache file or mbox file offsets. + +Reading, writing and locking +---------------------------- + +Reading 'dovecot.index' file requires locking, unfortunately. Shared read +locking is done using the standard index locking method specified in +'lock_method' setting ('lock_method' parameter for 'mail_index_open()'). + +Writing to index files requires transaction log to be exclusively locked first. +This way the index locking only has to worry about existing read locks. The +locking works by first trying to lock the index with the standard locking +method, but if it couldn't acquire the lock in two seconds, it'll fallback to +copying the index file to a temporary file, and when unlocking it'll 'rename()' +the temporary file over the 'dovecot.index' file. Note that this is safe only +because of the exclusive transaction log lock. This way the writers are never +blocked by readers who are allowed to keep the shared lock as long as they +want. + +The copy-locking is used always when doing anything that could corrupt the +index file if it crashed in the middle of an operation. For example if the +header or record size changes, or if messages are expunged. New messages can be +appended however, because the message count in the header is updated last. +Expunging the last messages would probably be safe also (because only the +header needs updating), but it's not done currently. + +The index file should never be directly modified. Everything should go through +the transaction log, and the only time the index needs to be write-locked is +when transactions are written to it. + +Currently the index file is updated whenever the backend mailbox is +synchronized. This isn't necessary, because an old index file can be updated +using the transaction log. In future there could be some smarter decisions +about when writing to the index isn't worth the extra disk writes. + +Header +------ + +---%<------------------------------------------------------------------------- +struct mail_index_header { + uint8_t major_version; + uint8_t minor_version; + + uint16_t base_header_size; + uint32_t header_size; + uint32_t record_size; + + uint8_t compat_flags; + uint8_t unused[3]; + + uint32_t indexid; + uint32_t flags; + + uint32_t uid_validity; + uint32_t next_uid; + + uint32_t messages_count; + uint32_t unused_old_recent_messages_count; + uint32_t seen_messages_count; + uint32_t deleted_messages_count; + + uint32_t first_recent_uid; + uint32_t first_unseen_uid_lowwater; + uint32_t first_deleted_uid_lowwater; + + uint32_t log_file_seq; + uint32_t log_file_tail_offset; + uint32_t log_file_head_offset; + + uint64_t unused_old_sync_size; + uint32_t unused_old_sync_stamp; + + uint32_t day_stamp; + uint32_t day_first_uid[8]; +} +---%<------------------------------------------------------------------------- + +Fields that won't change without recreating the index: + +major_version: + If this doesn't match 'MAIL_INDEX_MAJOR_VERSION', don't try to read the + index. Dovecot recreates the index file then. + +minor_version: + If this doesn't match 'MAIL_INDEX_MINOR_VERSION' there are some backwards + compatible changes in the index file (typically header fields). Try to + preserve the headers and the minor version when updating the index file. + +base_header_size: + Extension headers begin after the base headers. This is normally the same as + 'sizeof(struct mail_index_header)'. + +header_size: + Records begin after base and extension headers. + +record_size: + Size of each record and its extensions. Initially the same as 'sizeof(struct + mail_index_record)'. + +compat_flags: + Currently there is just one compatibility flag: + 'MAIL_INDEX_COMPAT_LITTLE_ENDIAN'. Dovecot doesn't try to bother to read + different endianess files, they're simply recreated. + +indexid: + Unique index file ID. This is used to make sure that the main index, + transaction log and cache file are all part of the same index. + +Header flags: + +MAIL_INDEX_HDR_FLAG_CORRUPTED: + Set whenever the index file is found to be corrupted. If the reader notices + this flag, it shouldn't try to continue using the index. + +MAIL_INDEX_HDR_FLAG_HAVE_DIRTY: + This index has records with 'MAIL_INDEX_MAIL_FLAG_DIRTY' flag set. + +MAIL_INDEX_HDR_FLAG_FSCK: + Call 'mail_index_fsck()' as soon as possible. This flag isn't actually set + anywhere currently. + +Message UIDs and counters: + +uid_validity: + IMAP UIDVALIDITY field. Initially can be 0, but after it's set we don't + currently try to even handle the case of UIDVALIDITY changing. It's done by + marking the index file corrupted and recreating it. That's a bit ugly, but + typically the UIDVALIDITY never changes. + +next_uid: + UID given to the next appended message. Only increases. + +messages_count: + Number of records in the index file. + +recent_messages_count: + Number of records with 'MAIL_RECENT' flag set. + +seen_messages_count: + Number of records with 'MAIL_SEEN' flag set. + +deleted_messages_count: + Number of records with 'MAIL_DELETED' flag set. + +first_recent_uid_lowwater: + There are no UIDs lower than this with 'MAIL_RECENT' flag set. + +first_unseen_uid_lowwater: + There are no UIDs lower than this *without* 'MAIL_SEEN' flag set. + +first_deleted_uid_lowwater: + There are no UIDs lower than this with 'MAIL_DELETE' flag set. + +The lowwater fields are used to optimize searching messages with/without a +specific flag. + +Fields related to syncing: + +log_file_seq: + Log file the log_*_offset fields point to. + +log_file_int_offset, log_file_ext_offset: + All the internal/external transactions before this offset in the log file are + synced to the index. External transactions are synced more often than + internal, so 'log_file_int_offset' <= 'log_file_ext_offset'. + +sync_size, sync_stamp: + Used by the mailbox backends to store their synchronization information. Some + day these should be removed and replaced with extension headers. + +Then there are day fields: + +day_stamp: + UNIX timestamp to the beginning of the day when new records were last added + to the index file. + +day_first_uid[8]: + These fields are updated when 'day_stamp' < today. The [0..6] are first moved + to [1..7], then [0] is set to the first appended UID. So they contain the + first UID of the day for last 8 days when messages were appended. + +The 'day_first_uid[]' fields are used by cache file compression to decide when +to drop 'MAIL_CACHE_DECISION_TEMP' data. + +Extension headers +----------------- + +After the base header comes a list of extensions and their headers. The first +extension begins from 'mail_index_header.base_header_size' offset. The second +begins after the first one's 'data[]' and so on. The extensions always begin +64bit aligned however, so you may need to skip a few bytes always. Read the +extensions as long as the offset is smaller than +'mail_index_header.header_size'. + +---%<------------------------------------------------------------------------- +struct mail_index_ext_header { + uint32_t hdr_size; /* size of data[] */ + uint32_t reset_id; + uint16_t record_offset; + uint16_t record_size; + uint16_t record_align; + uint16_t name_size; + /* unsigned char name[name_size] */ + /* unsigned char data[hdr_size] (starting 64bit aligned) */ +}; +---%<------------------------------------------------------------------------- + +'reset_id', record offset, size and alignment is explained in +'s 'struct mail_transaction_ext_intro'. + +Records +------- + +There are 'hdr.messages_count' records in the file. Each record contains at +least two fields: Record UID and flags. The UID is always increasing for the +records, so it's possible to find a record by its UID with binary search. The +record size is specified by 'mail_index_header.record_size'. + +The flags are a combination of 'enum mail_flags' and 'enum +mail_index_mail_flags'. There exists only one index flag currently: +'MAIL_INDEX_MAIL_FLAG_DIRTY'. If a record has this flag set, it means that the +mailbox syncing code should ignore the flag in the mailbox and use the flag in +the index file instead. This is used for example with mbox and +'mbox_lazy_writes=yes'. It also allows having modifiable flags for read-only +mailboxes. + +The rest data is stored in record extensions. + +Keywords +-------- + +The keywords are stored in record extensions, but for better performance and +lower disk space usage in transaction logs, they are quite tightly integrated +to the index file code. + +The list of keywords is stored in "keywords" extension header: + +---%<------------------------------------------------------------------------- +struct mail_index_keyword_header { + uint32_t keywords_count; + /* struct mail_index_keyword_header_rec[] */ + /* char name[][] */ +}; +struct mail_index_keyword_header_rec { + uint32_t unused; /* for backwards compatibility */ + uint32_t name_offset; /* relative to beginning of name[] */ +}; +---%<------------------------------------------------------------------------- + +The unused field originally contained 'count' field, but while writing this +documentation I noticed it's not actually used anywhere. Apparently it was +added there accidentally. It'll be removed in later versions. + +So there exists 'keywords_count' keywords, each listed in a NUL-terminated +string beginning from 'name_offset'. + +Since crashing in the middle of updating the keywords list pretty much breaks +the keywords, adding new keywords causes the index file to be always copied to +a temporary file and be replaced. + +The keywords in the records are stored in a "keywords" extension bitfield. So +the nth bit in the bitfield points to the nth keyword listed in the header. + +It's not currently possible to safely remove existing keywords. + +Extensions +---------- + +The extensions only specify their wanted size and alignment, the index file +syncing code is free to assign any offset inside the record to them. The +extensions may be reordered at any time. + +Dovecot's current extension ordering code works pretty well, but it's not +perfect. If the extension size isn't the same as its alignment, it may create +larger records than necessary. This will be fixed later. + +The records size is always divisible by the maximum alignment requirement. This +isn't strictly necessary either, so it could be fixed later as well. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Indexes.TransactionLog.txt b/doc/wiki/Design.Indexes.TransactionLog.txt new file mode 100644 index 0000000000..e86751b9c5 --- /dev/null +++ b/doc/wiki/Design.Indexes.TransactionLog.txt @@ -0,0 +1,265 @@ +Transaction log +=============== + +The transaction log is a bit similar to transaction logs in databases. All the +updates to the main index files are first written to the transaction log, and +only after that the main index file is updated. There are several advantages to +this: + + * It provides atomic transactions: The transaction either succeeds, or it + doesn't. For example if a transaction sets a flag to one message and removes + it from another, it's guaranteed that both changes happen. + * When updating the changes to the main index file, the last thing that's + done is to update the "transaction log position" in the header. So if + Dovecot crashes after having updated only the first flag, the next time + the mailbox is opened both of the changes are done all over again. + * It allows another process to quickly see what changes have been made. For + example IMAP needs to get a list of external changes after each command. + * This is also important when storing the index files in NFS or in a + clustered filesystem. Instead of re-reading the whole index file after + each external change, Dovecot can simply read the new changes from the + transaction log and apply them to the in-memory copy of the main index. + In-memory caching of 'dovecot.index.cache' file also relies on the + transaction log telling what parts of the file has changed. + * In future the transaction logs can be somewhat easily used to implement + replication. + +Internal vs. external +--------------------- + +Transactions are either internal or external. The difference is that external +transactions describe changes that were already made to the mailbox, while +internal transactions are commands to do something to the mailbox. When +beginning to synchronize a mailbox with index files, the index file is first +updated with all the external changes, and the uncommitted internal +transactions are applied on top of them. + +When synchronizing the mailbox, using the synchronization transaction writes +only external transactions. Also if the index file is updated when saving new +mails to the mailbox, the append transactions must be external. This is because +the changes are already in the mailbox at the time the transaction is read. + +Reading and writing +------------------- + +Reading transaction logs doesn't require any locking at all. Writing is +exclusively locked using the index files' default lock method (as specified by +the 'lock_method' setting). + +A new log is created by first creating a 'dovecot.index.log.newlock' dotlock +file. Once you have the dotlock, check again that the 'dovecot.index.log' +wasn't created (or recreated) by another process. If not, go ahead and write +the log header to the dotlock file and finally 'rename()' it to +'dovecot.index.log'. + +Currently there doesn't exist actual transaction boundaries in the log file. +All the changes in a transaction are simply written as separate records to the +file. Each record begins with a 'struct mail_transaction_header', which +contains the record's size and type. The size is in +[Design.Indexes.txt]. + +The first transaction record is written with the size field being 0. Once the +whole transaction has been written, the 0 is updated with the actual size. This +way the transaction log readers won't see partial transactions because they +stop at the size=0 if the transaction isn't fully written yet. + +Note that because there are no transaction boundaries, there's a small race +condition here with mmap()ed log files: + + 1. Process A: write() half of the transaction + 2. Process B: mmap() the file. + 3. Process A: write() the rest of the transaction, updating the size=0 also + 4. Process B: parse the log file. it'll go past the original size=0 because + the size had changed in the mmap, but it stops in the middle of the + transaction because the mmap size doesn't contain the whole transaction + +This probably isn't a big problem, because I've never seen this happen even +with stress tests. Should be fixed at some point anyway. + +Header +------ + +The transaction log's header never changes, except the indexid field may be +overwritten with 0 if the log is found to be corrupted. The fields are: + +major_version: + If this doesn't match 'MAIL_TRANSACTION_LOG_MAJOR_VERSION', don't try to + parse it. If Dovecot sees this, it'll recreate the log file. + +minor_version: + If this doesn't match 'MAIL_TRANSACTION_LOG_MINOR_VERSION', the log file + contains some backwards compatible changes. Currently you can just ignore + this field. + +hdr_size: + Size of the log file's header. Use this instead of 'sizeof(struct + mail_transaction_log_header)', so that it's possible to add new fields and + still be backwards compatible. + +indexid: + This field must match to main index file's indexid field. + +file_seq: + The file's creation sequence. Must be increasing. + +prev_file_seq, prev_file_offset: + Contains the sequence and offset of where the last transaction log ended. + When transaction log is rotated and the reader's "sync position" still points + to the previous log file, these fields allow it to easily check if there had + been any more changes in the previous file. + +create_stamp: + UNIX timestamp when the file was created. Used in determining when to rotate + the log file. + +Record header +------------- + +The transaction record header ('struct mail_transaction_header') contains +'size' and 'type' fields. The 'size' field is in +[Design.Indexes.txt]. A single transaction record may contain multiple changes +of the same type, although some types don't allow this. Because the size of the +transaction record for each type is known (or can be determined from the +type-specific record contents), the 'size' field can be used to figure out how +many changes need to be done. So for example a record can contain: + + * 'struct mail_transaction_header { type = MAIL_TRANSACTION_APPEND, size = + sizeof(struct mail_index_record) * 2 }' + * 'struct mail_index_record { uid = 1, flags = 0 } ' + * 'struct mail_index_record { uid = 2, flags = 0 } ' + +UIDs +---- + +Many record types contain 'uint32_t uid1, uid2' fields. This means that the +changes apply to all the messages in uid1..uid2 range. The messages don't +really have to exist in the range, so for example if the first messages in the +mailbox had UIDs 1, 100 and 1000, it would be possible to use uid1=1, uid2=1000 +to describe changes made to these 3 messages. This also means that it's safe to +write transactions describing changes to messages that were just expunged by +another process (and already written to the log file before our changes). + +Appends +------- + +As described above, the appends must be in external transactions. The append +transaction's contents is simply the 'struct mail_index_record', so it contains +only the message's UID and flags. The message contents aren't written to +transaction log. Also if the message had any keywords when it was appended, +they're in a separate transaction record. + +Expunges +-------- + +Because expunges actually destroy messages, they deserve some extra protection +to make it less likely to accidentally expunge wrong messages in case of for +example file corruption. The expunge transactions must have +'MAIL_TRANSACTION_EXPUNGE_PROT' ORed to the transaction type field. If an +expunge type is found without it, assume a corrupted transaction log. + +Flag changes +------------ + +The flag changes are described in: + +---%<------------------------------------------------------------------------- +struct mail_transaction_flag_update { + uint32_t uid1, uid2; + uint8_t add_flags; + uint8_t remove_flags; + uint16_t padding; +}; +---%<------------------------------------------------------------------------- + +The 'padding' is ignored completely. A single flag update structure can add new +flags or remove existing flags. Replacing all the files works by setting +'remove_flags = 0xFF' and the 'add_flags' containing the new flags. + +Keyword changes +--------------- + +Specific keywords can be added or removed one keyword at a time: + +---%<------------------------------------------------------------------------- +struct mail_transaction_keyword_update { + uint8_t modify_type; /* enum modify_type : MODIFY_ADD / MODIFY_REMOVE +*/ + uint8_t padding; + uint16_t name_size; + /* unsigned char name[]; + array of { uint32_t uid1, uid2; } + */ +}; +---%<------------------------------------------------------------------------- + +There is padding after 'name[]' so that uid1 begins from a 32bit aligned +offset. + +If you want to replace all the keywords (eg. IMAP's 'STORE 1:* FLAGS (keyword)' +command), you'll first have to remove all of them with +'MAIL_TRANSACTION_KEYWORD_RESET' and then add the new keywords. + +Extensions +---------- + +Extension records allow creating and updating extension-specific header and +message record data. For example messages' offsets to cache file or mbox file +are stored in extensions. + +Whenever using an extension, you'll need to first write +'MAIL_TRANSACTION_EXT_INTRO' record. This is a bit kludgy and hopefully will be +replaced by something better in future. The intro contains: + +---%<------------------------------------------------------------------------- +struct mail_transaction_ext_intro { + /* old extension: set ext_id. don't set name. + new extension: ext_id = (uint32_t)-1. give name. */ + uint32_t ext_id; + uint32_t reset_id; + uint32_t hdr_size; + uint16_t record_size; + uint16_t record_align; + uint16_t unused_padding; + uint16_t name_size; + /* unsigned char name[]; */ +}; +---%<------------------------------------------------------------------------- + +If the extension already exists in the index file (it can't be removed), you +can use the 'ext_id' field directly. Otherwise you'll need to give a name to +the extension. It's always possible to just give the name if you don't know the +existing extension ID, but this uses more space of course. + +'reset_id' contains kind of a "transaction validity" field. It's updated with +'MAIL_TRANSACTION_EXT_RESET' record, which also causes the extension records' +contents to be zeroed. If an introduction's 'reset_id' doesn't match the last +EXT_RESET, it means that the extension changes are stale and they must be +ignored. For example: + + * 'dovecot.index.cache' file's 'file_seq' header is used as a 'reset_id'. + Initially it's 1. + * Process A: Begins a cache transaction, updating some fields in it. + * Process B: Decides to compress the cache file, and issues a 'reset_id = 2' + change. + * Process A: Commits the transaction with 'reset_id = 1', but the cache file + offsets point to the old file, so the changes must be ignored. + +'hdr_size' specifies the number of bytes the extension wants to have in the +index file's header.'record_size' specifies the number of bytes it wants to use +for each record. The sizes may grow or shrink any time.'record_align' contains +the required alignmentation for the field. For example if the extension +contains a 32bit integer, you want it to be 32bit aligned so that the process +won't crash in CPUs which require proper alignmentation. Then again if you want +to access the field as 4 bytes, the alignmentation can be 1. + +Extension record updates typically are message-specific, so the changes must be +done for each message separately: + +---%<------------------------------------------------------------------------- +struct mail_transaction_ext_rec_update { + uint32_t uid; + /* unsigned char data[]; */ +}; +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Indexes.txt b/doc/wiki/Design.Indexes.txt new file mode 100644 index 0000000000..00b55a2a86 --- /dev/null +++ b/doc/wiki/Design.Indexes.txt @@ -0,0 +1,70 @@ +Dovecot's index files +===================== + +Dovecot's index files consist of three different files: + + *
[Design.Indexes.MainIndex.txt] ('dovecot.index') + * [Design.Indexes.TransactionLog.txt] ('dovecot.index.log' + and 'dovecot.index.log.2') + * [Design.Indexes.Cache.txt] ('dovecot.index.cache') + +See for more generic information about what they contain and +why. + +The index files can be accessed using +[Design.Indexes.MailIndexApi.txt]. + +Locking +------- + +The index files are designed so that readers cannot block a writer, and write +locks are always short enough not to cause other processes to wait too long. +Dovecot v0.99's index files didn't do this, and it was common to get lock +timeouts when using multiple connections to the same large mailbox. + +The main index file is the only file which has read locks. They can however +block the writer only for two seconds (and even this could be changed to not +block at all). The writes are locked only for the duration of the mailbox +synchronization. + +Transaction logs don't require read locks. The writing is locked for the +duration of the mailbox synchronization, and also for single transaction +appends. + +Cache files doesn't require read locks. They're locked for writing only for the +duration of allocating space inside the file. The actual writing inside the +allocated space is done without any locks being held. + +In future these could be improved even further. For example there's no need to +keep any index files locked while synchronizing, as long the mailbox backend +takes care of the locking issues. Also writing to transaction log could work in +a similar way to cache files: Lock, allocate space, unlock, write. + +Lockless integers +----------------- + +Dovecot uses several different techniques to allow reading files without +locking them. One of them uses fields in a "lockless integer" format. Initially +these fields have "unset" value. They can be set to a wanted value in range +0..2^28 (with 32bit fields) once, but they cannot be changed. It would be +possible to set them back to "unset", but setting them the second time isn't +safe anymore, so Dovecot never does this. + +The lockless integers work by allocating one bit from each byte of the value to +"this value is set" flag. The reader then verifies that the flag is set for the +value's all bytes. If all of them aren't set, the value is still "unset". +Dovecot uses the highest bit for this flag. So for example: + + * 0x00000000: The value is unset + * 0xFFFF7FFF: The value is unset, because one of the bytes didn't have the + highest bit set + * 0xFFFFFFFF: The value is 2^28-1 + * 0x80808080: The value is 0 + * 0x80808180: The value is 0x80 + +Dovecot contains 'mail_index_uint32_to_offset()' and +'mail_index_offset_to_uint32()' functions to translate values between integers +and lockless integers. The "unset" value is returned as 0, so it's not possible +to differentiate between "unset" and "set" 0 values. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.InputStreams.txt b/doc/wiki/Design.InputStreams.txt new file mode 100644 index 0000000000..c64d4cb453 --- /dev/null +++ b/doc/wiki/Design.InputStreams.txt @@ -0,0 +1,207 @@ +Input Streams +============= + +'lib/istream.h' describes Dovecot's input streams. Input streams can be stacked +on top of each others as many times as wanted. + +Input streams actually reading data: + + * file: Read data from fd using 'pread()' for files and 'read()' for + non-files. + * unix: Read data from UNIX socket. Similar to file, but supports receiving + file descriptors. + * mmap: Read data from file using 'mmap()'. This usually seems to be slower + than just using it with 'read()', so this input stream is probably quite + unnecessary. + * data: Read data from memory. + +Input stream filters: + + * concat: Concatenate multiple input streams together + * chain: Chain multiple input streams together. Similar to istream-concat, but + more istreams can be added after initialization and EOF needs to be + explicitly added. + * seekable: Make a number of (possibly non-seekable) input streams into a + single seekable input stream. If all of the input streams are already + seekable, a concat stream is created instead. + * Usually the only non-seekable input streams are non-file fds, such as + pipes or sockets. + * crlf: Change all newlines to either LFs or CRLFs, by adding or removing CRs + as necessary. + * limit: Limit input stream's length, so after reading a given number of bytes + it returns EOF. + * sized: Require istream's length to be exactly the given size, or the last + read returns error. + * timeout: Fail the read when given timeout is reached. + * try: Read from the first input stream that doesn't fail with EINVAL. + * tee: Fork an input stream to multiple streams that can be read + independently. + * multiplex: Multiplex-iostreams support multiple iostream channels inside a + single parent istream. + * callback: Build an input stream by calling callback functions that return + the data. + * base64-encoder, base64-decoder: Encode/decode base64. + * failure-at: Insert a failure at the specified offset. This can be useful for + testing. + * hash: Calculate hash of the istream while it's being read. + * lib-mail/dot: Read SMTP-style DATA input where the input ends with an empty + "." line. + * lib-mail/header-filter: Add/remove/modify email headers. + * lib-compression/*: Read zlib/bzlib/lz4/lzma compressed data. + +Reading +------- + +'i_stream_read()' tries to read more data into the stream's buffer. It returns: + + * -2: Nothing was read, because buffer is full. + * -1: Either input reached EOF, or read failed and stream_errno was set. + * 0: Input stream is non-blocking, and no more input is available now. + * >0: Number of bytes read. + +Reading from a stream doesn't actually go forward in the stream, that needs to +be done manually with 'i_stream_skip()'. This makes it easy to read full data +records into the stream directly instead of creating separate buffers. For +example when reading line-based input you can keep reading input into the +stream until you find LF and then just access the string directly from the +input buffer. There are actually helper functions for +this:'i_stream_next_line()' attempts to return the next line if available, +'i_stream_read_next_line()' does the same but does a read to try to get the +data. + +Because more and more data can be read into the buffer, the buffer size is +typically limited, and once this limit is reached read returns -2. The buffer +size is usually given as parameter to the 'i_stream_create_*()', filters use +their parent stream's buffer size. The buffer size can be also changed with +'i_stream_set_max_buffer_size()'. Figuring out what the buffer size should be +depends on the situation. It should be large enough to contain all valid input, +but small enough that users can't cause a DoS by sending a too large record and +having Dovecot eat up all the memory. + +Once read returns -1, the stream has reached EOF. 'stream->eof=TRUE' is also +set. In this situation it's important to remember that there may still be data +available in the buffer. If 'i_stream_have_bytes_left()' returns FALSE, there +really isn't anything left to read. + +Whenever i_stream_read() returns >0, all the existing pointers are potentially +invalidated. v2.3+: When i_stream_read() returns<= 0, the data previously +returned by i_stream_get_data() are still valid, preserved in "snapshots". +(w_buffer + stream->pos'. + * 'i_stream_alloc(size) is like 'i_stream_try_alloc()', except it always + succeeds allocating'size` bytes, even if it has to grow the buffer larger + then the stream's max buffer size. + * Lower-level memory allocation functions: + * 'i_stream_w_buffer_realloc(old_size)' reallocates 'w_buffer' to the + current 'buffer_size'. If memarea's refcount is 1, this can be done with + 'i_realloc()', otherwise new memory is allocated. + * 'i_stream_grow_buffer(bytes)' grows the 'w_buffer' by the given number of + bytes, if possible. It won't reach the stream's current max buffer size. + The caller must verify from 'buffer_size' how large the buffer became as + a result of this call. + * 'i_stream_compress()' attempts to compress the current 'w_buffer' by + removing already-skipped data with 'memmove()'. If 'skip' is 0, it does + nothing. Note that this function must not be called if 'memarea' has + refcount>1. Otherwise that could be modifying a snapshotted memarea. + +The snapshots have made implementing slightly more complicated than earlier. +There are a few different ways to implement istreams: + + * Always point 'buffer=w_buffer' and use 'i_stream_try_alloc()' and/or + 'i_stream_alloc()' to allocate the 'w_buffer'. The generic code will handle + all the snapshotting. Use 'i_stream_read_memarea()' to read data from parent + stream so multiple snapshots aren't unnecessarily created. + * Guarantee that if 'read()' returns <=0, the existing 'buffer' will stay + valid. Use 'ISTREAM_CREATE_FLAG_NOOP_SNAPSHOT' flag in 'i_stream_create()' + so your filter stream isn't unnecessarily snapshotted (or causing a panic + due to missing 'snapshot()' implementation). + * One way of doing this with filter streams is to read from the parent + stream via 'i_stream_read(parent)' and always use + 'buffer=i_stream_get_data(parent)'. The parent's snapshotting guarantees + that the buffer will stay valid. + * Implement the 'snapshot()' yourself in the stream. You'll need to create a + new memarea of the current data available via 'i_stream_get_data()' and it + must not change, i.e. most likely you'll need to duplicate the allocated + memory. Create a new 'struct istream_snapshot' and assign the allocated + memarea to its 'old_memarea'. Fill 'prev_snapshot' field and return your new + snapshot. The snapshot will be freed by the generic istream code either when + the next 'read()' returns >0 or when the istream is destroyed. + * Filter streams that only pass through parent stream's contents without + changes can just point to the parent stream. The default snapshotting causes + the parent to be snapshotted, so the filter stream can simply use + 'i_stream_read_memarea()' and point to the parent's buffer. + +When Dovecot is configured with '--enable-devel-checks', 'i_stream_read()' will +verify that the first and the last two bytes of the buffer didn't unexpectedly +change due to a 'read()'. While developing istream changes you should use this +to make sure the istream is working properly. Running the istream unit test +also via valgrind can also be used to verify that the buffer wasn't freed. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Lua.txt b/doc/wiki/Design.Lua.txt new file mode 100644 index 0000000000..805e00b032 --- /dev/null +++ b/doc/wiki/Design.Lua.txt @@ -0,0 +1,249 @@ +Dovecot Lua support +=================== + +Since v2.3.0 dovecot supports Lua scripting. Dovecot supports lua 5.0 or newer. +See also: + + * [AuthDatabase.Lua.txt]. Since v2.3.0 + * [Plugins.PushNotification.txt]. Since v2.3.4 + +lib-lua +------- + +Dovecot provides a lib-lua internal helper as part of libdovecot.so. It has +facilities for loading scripts from various sources, and also helps with +reusing scripts by keeping track of which scripts are loaded. Each script has +it's own memory pool, which is guaranteed to be released when script is +unloaded. + +When script is loaded, *script_load* function is called if found. This can +return non-zero to indicate that the script has a problem. + +C API +----- + + * 'dlua_register_dovecot(script)' + +Register dovecot variable. This item can also be extended by context specific +tables, like authentication database adds *dovecot.auth*. + + * 'dlua_push_event(event)' + +Pushes an Dovecot Event to stack. + +Lua API +------- + + * 'i_debug(text)' + * 'i_error(text)' + * 'i_info(text)' + * 'i_warning(text)' + +Functions within dovecot variable that do generic logging + +Event functions are available from v2.3.4 on + + * 'dovecot.event()' + * 'dovecot.event(parent)' + +Creates a new event for dovecot + +object event +------------ + +Note: + + * object event_passthrough has same API, except the 'passthrough_event' method + is not present. + +Functions: + + * 'append_log_prefix("prefix")' - set prefix to append + * 'replace_log_prefix("prefix")' - replace append prefix + * 'set_name("name")' - set name for event + * 'add_str("key","value")' - add a key-value pair to event + * 'add_int("key",1)' - add a key-value pair to event + * 'add_timeval("key",seconds)' - add a key-value pair to event + * 'inc_int("key",1)' - increment key-value pair + * 'log_debug("message")' - emit debug message + * 'log_info("message")' - emit info message + * 'log_warning("message")' - emit warning message + * 'log_error("message")' - emit error message + * 'passthrough_event()' - returns an passthrough event. A log message *must + be* logged or else a panic will occur. + +mail-lua +-------- + +Available from v2.3.4 on + +mail-lua is a plugin that can be loaded to provide API for mail storage Lua +plugins. Mail-lua provides a common script to be used in mail storage instead +of per-plugin scripts. + +C API +----- + + * 'dlua_register_mail_storage(script)' + +Register storage Lua interface to script context + + * 'bool mail_lua_plugin_get_script(user, script_r)' + +Returns script context if available. If FALSE is returned, no Lua script has +been loaded, and you should optionally deal this yourself. + + * 'dlua_push_mail_user(script, user)' + +Pushes a mail user on top of stack. + + * 'dlua_push_mailbox(script, box)' + +Pushes a mailbox on top of stack. + + * 'dlua_push_mail(script, mail)' + +Pushes a mail on top of stack. + +Lua API +------- + +When mail user is created, a script is loaded if present as *mail_lua_script* +and *mail_user_created* is called if present in script. + +On deinitialization, *mail_user_deinit_pre* is called first, if present, +followed by *mail_user_deinit*. + +dovecot.storage +--------------- + +Following constants are specified + + * 'STATUS_MESSAGES' + * 'STATUS_RECENT' + * 'STATUS_UIDNEXT' + * 'STATUS_UIDVALIDITY' + * 'STATUS_UNSEEN' + * 'STATUS_FIRST_UNSEEN_SEQ' + * 'STATUS_KEYWORDS' + * 'STATUS_HIGHESTMODSEQ' + * 'STATUS_PERMANENT_FLAGS' + * 'STATUS_FIRST_RECENT_UID' + * 'STATUS_HIGHESTPVTMODSEQ' + * 'MAILBOX_FLAG_READONLY' + * 'MAILBOX_FLAG_SAVEONLY' + * 'MAILBOX_FLAG_DROP_RECENT' + * 'MAILBOX_FLAG_NO_INDEX_FILES' + * 'MAILBOX_FLAG_KEEP_LOCKED' + * 'MAILBOX_FLAG_IGNORE_ACLS' + * 'MAILBOX_FLAG_AUTO_CREATE' + * 'MAILBOX_FLAG_AUTO_SUBSCRIBE' + * 'MAILBOX_SYNC_FLAG_FULL_READ' + * 'MAILBOX_SYNC_FLAG_FULL_WRITE' + * 'MAILBOX_SYNC_FLAG_FAST' + * 'MAILBOX_SYNC_FLAG_NO_EXPUNGES' + * 'MAILBOX_SYNC_FLAG_FIX_INCONSISTENT' + * 'MAILBOX_SYNC_FLAG_EXPUNGE' + * 'MAILBOX_SYNC_FLAG_FORCE_RESYNC' + +object mail_user +---------------- + +Meta: + + * has tostring + * is comparable (by username) + +Functions: + + * 'plugin_getenv(key)' - returns key from user plugin settings or userdb + environment + * 'var_expand(template)' - expands mail user variables (see ) + * 'mailbox(name, flags)' - allocates a mailbox, flags optional + +Variables: + + * 'home' - home directory (if available) + * 'username' - user's name + * 'uid' - system uid + * 'gid' - system gid + * 'service' - IMAP/POP3/LMTP/LDA/... + * 'session_id' - Current session ID + * 'session_create_time' - When session was created + * 'nonexistent' - If user does not really exist + * 'anonymous' - If user is anonymous + * 'autocreated' - If user was automatically created internally for some + operation + * 'mail_debug' - If debugging is turned on + * 'fuzzy_search' - FIXME: Undocumented + * 'dsyncing' - If user is being dsync'd + * 'session_restored' - If this is a restored session + +object mailbox +-------------- + +Meta: + + * has tostring + * is comparable (by full mailbox name) + +Functions: + + * 'open()' - opens the mailbox + * 'close()' - closes the mailbox + * 'free()' - releases mailbox (must be done) + * 'sync(flags)' - synchronizes the mailbox (should usually be done, flags + optional) + * 'status(item,item,item...)' - returns requested mailbox status items as + table + +Variables: + + * 'vname' - Full mailbox name + * 'name' - Mailbox name + +table mailbox status +-------------------- + +Variables: + + * 'mailbox' - full name of mailbox + * 'messages' - number of messages + * 'recent' - number of \Recent messages + * 'unseen' - number of \Unseen messages + * 'uidvalidity' - current UID validity + * 'uidnext' - next UID + * 'first_unseen_seq' - first seqno of unseen mail + * 'first_recent_uid' - first UID of unseen mail + * 'highest_modseq' - highest modification sequence + * 'highest_pvt_modseq' - highest private modification sequence + * 'permanent_flags' - supported permanent flags as a bitmask + * 'flags' - supported flags as a bitmask + * 'permanent_keywords' - if permanent keywords are supported + * 'allow_new_keywords' - if new keywords can be added + * 'nonpermanent_modseqs' - whether non-permanent keywords are allowed + * 'no_modseq_tracking' - no modification sequence tracking + * 'have_guids' - whether GUIDs exist + * 'have_save_guids' - whether GUIDs can be saved + * 'have_only_guid128' - whether GUIDs are 128 bit always + * 'keywords' - table of current keywords + +object mail +----------- + +Meta: + + * has tostring + * is comparable (within same mailbox, by UID) + +Functions: + + * None yet + +Variables: + + * 'mailbox' - mailbox object + * 'seq' - Sequence number (can change) + * 'uid' - UID number (immutable) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.MailProcess.txt b/doc/wiki/Design.MailProcess.txt new file mode 100644 index 0000000000..b583d46412 --- /dev/null +++ b/doc/wiki/Design.MailProcess.txt @@ -0,0 +1,12 @@ +Mail Process Design +=================== + +FIXME: + + * IMAP/POP3 protocol specific code + * Mail storage API + * Index files under everything + * Plugins + * etc. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Memory.txt b/doc/wiki/Design.Memory.txt new file mode 100644 index 0000000000..64ad0139e0 --- /dev/null +++ b/doc/wiki/Design.Memory.txt @@ -0,0 +1,193 @@ +Memory Allocations +================== + +C language requires explicitly allocating and freeing memory. The main two +problems with this are: + + 1. A lot of allocations and frees cause memory fragmentation. The longer a + process runs, the more it could have leaked memory because there are tiny + unused free spots all around in heap. + 2. Freeing memory is easy to forget, causing memory leaks. Sometimes it can be + accidentally done multiple times, causing a potential security hole. A lot + of free() calls all around in the code also makes the code more difficult + to read and write. + +The second problem could be solved with Boehm garbage collector, which Dovecot +can use optionally (prior to 2.3), but it's not very efficient. It also doesn't +help with the first problem. + +To reduce the problems caused by these issues, Dovecot has several ways to do +memory management. + +Common Design Decisions +----------------------- + +All memory allocations (with some exceptions in data stack) return memory +filled with NULs. This is also true for new memory when growing an allocated +memory with realloc. The zeroing reduces accidental use of uninitialized memory +and makes the code simpler since there is no need to explicitly set all fields +in allocated structs to zero/NULL. (I guess assuming that this works correctly +for NULLs isn't strictly ANSI-C compliant, but I don't see this assumption +breaking in any system anyone would really use Dovecot.) The zeroing is cheap +anyway. + +In out-of-memory situations memory allocation functions die internally by +calling 'i_fatal_status(FATAL_OUTOFMEM, ..)'. There are several reasons for +this: + + * Trying to handle malloc() failures explicitly would add a lot of error + handling code paths and make the code much more complex than necessary. + * In most systems malloc() rarely actually fails because the system has run + out of memory. Instead the kernel will just start killing processes. + * Typically when malloc() does fail, it's because the process's address space + limit is reached. Dovecot enforces these limits by default. Reaching it + could mean that the process was leaking memory and it should be killed. It + could also mean that the process is doing more work than anticipated and + that the limit should probably be increased. + * Even with perfect out-of-memory handling, the result isn't much better + anyway than the process dying. User isn't any happier by seeing "out of + memory" error than "server disconnected". + +When freeing memory, most functions usually also change the pointer to NULL. +This is also the reason why most APIs' deinit functions take pointer-to-pointer +parameter, so that when they're done they can change the original pointer to +NULL. + +malloc() Replacements +--------------------- + +'lib/imem.h' has replacements for all the common memory allocation functions: + + * 'malloc', 'calloc' -> 'i_malloc()' + * 'realloc()' -> 'i_realloc()' + * 'strdup()' -> 'i_strdup()' + * 'free()' -> 'i_free' + * etc. + +All memory allocation functions that begin with 'i_' prefix require that the +memory is later freed with 'i_free()'. If you actually want the freed pointer +to be set to NULL, use 'i_free_and_null()'. Currently 'i_free()' also changes +the pointer to NULL, but in future it might change to something else. + +Memory Pools +------------ + +'lib/mempool.h' defines API for allocating memory through memory pools. All +memory allocations actually go through memory pools. Even the 'i_*()' functions +get called through 'default_pool', which by default is 'system_pool' but can be +changed to another pool if wanted. All memory allocation functions that begin +with 'p_' prefix have a memory pool parameter, which it uses to allocate the +memory. + +Dovecot has many APIs that require you to specify a memory pool. Usually (but +not always) they don't bother freeing any memory from the pool, instead they +assume that more memory can be just allocated from the pool and the whole pool +is freed later. These pools are usually alloconly-pools, but can also be data +stack pools. See below. + +Alloc-only Pools +---------------- + +'pool_alloconly_create()' creates an allocate-only memory pool with a given +initial size. + +As the name says, alloconly-pools only support allocating more memory. As a +special case its last allocation can be freed.'p_realloc()' also tries to grow +the existing allocation only if it's the last allocation, otherwise it'll just +allocates new memory area and copies the data there. + +Initial memory pool sizes are often optimized in Dovecot to be set large enough +that in most situations the pool doesn't need to be grown. To make this easier, +when Dovecot is configured with --enable-devel-checks, it logs a warning each +time a memory pool is grown. The initial pool size shouldn't of course be made +too large, so usually I just pick some small initial guessed value and later if +I get too many "growing memory pool" warnings I start growing the pool sizes. +Sometimes there's just no good way to set the initial pool size and avoid the +warnings, in that situation you can prefix the pool's name with MEMPOOL_GROWING +and it doesn't log warnings. + +Alloconly-pools are commonly used for an object that builds its state from many +memory allocations, but doesn't change (much of) its state. It's a lot easier +when you can do a lot of small memory allocations and when destroying the +object you'll just free the memory pool. + +Data Stack +---------- + +'lib/data-stack.h' describes the low-level data stack functions. Data stack +works a bit like C's control stack.'alloca()' is quite near to what it does, +but there's one major difference: In data stack the stack frames are explicitly +defined, so functions can return values allocated from data +stack.'t_strdup_printf()' call is an excellent example of why this is useful. +Rather than creating some arbitrary sized buffer and using snprintf(), which +might truncate the value, you can just use t_strdup_printf() without worrying +about buffer sizes being large enough. + +Try to keep the allocations from data stack small, since the data stack's +highest memory usage size is kept for the rest of the process's lifetime. The +initial data stack size is 32kB, which should be enough in normal use. If +Dovecot is configured with --enable-devel-checks, it logs a warning each time +the data stack needs to be grown. + +Stack frames are preferably created using T_BEGIN/T_END block, for example: + +---%<------------------------------------------------------------------------- +T_BEGIN { + string_t *str1 = t_str_new(256); + string_t *str2 = t_str_new(256); + /* .. */ +} T_END; +---%<------------------------------------------------------------------------- + +In the above example two strings are allocated from data stack. They get freed +once the code goes past T_END, that's why the variables are preferably declared +inside the T_BEGIN/T_END block so they won't accidentally be used after they're +freed. + +T_BEGIN and T_END expand to 't_push()' and 't_pop()' calls and they must be +synchronized. Returning from the block without going past T_END is going to +cause Dovecot to panic in next T_END call with "Leaked t_pop() call" error. + +Memory allocations have similar disadvantages to alloc-only memory pools. +Allocations can't be grown, so with the above example if str1 grows past 256 +characters, it needs to be reallocated, which will cause it to forget about the +original 256 bytes and allocate 512 bytes more. + +Memory allocations from data stack often begin with 't_' prefix, meaning +"temporary". There are however many other functions that allocate memory from +data stack without mentioning it. Memory allocated from data stack is usually +returned as a const pointer, so that the caller doesn't try to free it (which +would cause a compiler warning). + +When should T_BEGIN/T_END used and when not? This is kind of black magic. In +general they shouldn't be used unless it's really necessary, because they make +the code more complex. But if the code is going through loops with many +iterations, where each iteration is allocating memory from data stack, running +each iteration inside its own stack frame would be a good idea to avoid +excessive memory usage. It's also difficult to guess how public APIs are being +used, so I've tried to make such API functions use their own private stack +frames. Dovecot's ioloop code also wraps all I/O callbacks and timeout +callbacks into their own stack frames, so you don't need to worry about them. + +You can create temporary memory pools from data stack too. Usually you should +be calling 'pool_datastack_create()' to generate a new pool, which also tries +to track that it's not being used unsafely across different stack frames. Some +low-level functions can also use 'unsafe_data_stack_pool' as the pool, which +doesn't do such tracking. + +Data stack's advantages over malloc(): + + * FAST, most of the time allocating memory means only updating a couple of + pointers and integers. Freeing memory all at once also is a fast operation. + * No need to 'free()' each allocation resulting in prettier code + * No memory leaks + * No memory fragmentation + +It also has some disadvantages: + + * Allocating memory inside loops can accidentally allocate a lot of memory + * Memory allocated from data stack can be accidentally stored into a permanent + location and accessed after it's already been freed. + * Debugging invalid memory usage may be difficult using existing tools + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.OutputStreams.txt b/doc/wiki/Design.OutputStreams.txt new file mode 100644 index 0000000000..97bc74158c --- /dev/null +++ b/doc/wiki/Design.OutputStreams.txt @@ -0,0 +1,96 @@ +Output Streams +============== + +'lib/ostream.h' describes Dovecot's output streams. Output streams can be +stacked on top of each others as many times as wanted. + +Output streams actually writing data: + + * file: Write to given fd using 'pwrite()' for files and 'write()' for + non-files. + * unix: Write to given UNIX socket. Similar to file, but supports sending file + descriptors. + * buffer: Write to [Design.Buffers.txt]. + +Output stream filters: + + * hash: Calculate hash of the ostream while it's being written. + * escaped: Write output escaped via callback. Built-in support for HEX and + JSON escaping. + * multiplex: Multiplex-iostreams support multiple iostream channels inside a + single parent istream. + * null: All the output is discarded. + * failure-at: Insert a failure at the specified offset. This can be useful for + testing. + * lib-mail/ostream-dot: Write SMTP-style DATA input where the output ends with + an empty "." line. + * lib-dcrypt/encrypt: Write encrypted data. + * lib-compression/*: Write zlib/bzlib/lz4/lzma compressed data. + +A typical life cycle for an ostream can look like: + + * create + * 'o_stream_cork()' + * 'o_stream_nsend*()' one or more times + * 'o_stream_uncork()' + * If necessary, check errors with 'o_stream_flush()' + * 'o_stream_cork()' + * 'o_stream_nsend*()' one or more times + * 'o_stream_uncork()' + * finalize the ostream with 'o_stream_finish()' + * optionally close the ostream with 'o_stream_close()' + * unref or destroy + +Once the ostream is finished, it can't be written to anymore. The +'o_stream_finish()' call writes any potential trailer that the ostream may have +(e.g. ostream-gz, ostream-encrypt, ostream-dot) while still allowing the caller +to check if the trailer write failed. After 'o_stream_finish()' is called, any +further write will panic. The ostreams that require a trailer will panic if +'o_stream_finish()' hasn't been called before the stream is destroyed, but +other ostreams don't currently require this. Still, it's not always easy to +know whether there might be ostreams that require the trailer, so if there's +any doubt, it's preferred to call 'o_stream_finish()' just before destroying +the ostream. + +Usually calling 'o_stream_finish()' will also finish its parent ostream. This +may or may not be wanted depending on the situation. For example ostream-dot +must be finished to write the last "." line, but ostream-dot is always a +sub-stream of something else that must not be finished yet. This is why +ostream-dot by default has called 'o_stream_set_finish_also_parent(FALSE)', so +finishing the ostream-dot won't finish the parent stream. Similarly +'connection.c' API sets 'o_stream_set_finish_via_child(FALSE)' so none of the +socket connections created via it will be finished even though one of their +sub-streams is finished. These functions may need to be called explicitly in +other situations. + +When doing a lot of writes, you can simplify the error handling by delaying the +error checking. Use the 'o_stream_nsend*()' functions and afterwards check the +error with 'o_stream_flush()' or 'o_stream_finish()'. If you forgot to do this +check before the ostream is destroyed, it will panic with:'output stream %s is +missing error handling' regardless of whether there is an error or not. If you +don't care about errors for the ostream (e.g. because it's a client socket and +there's nothing you can do about the write errors), you can use +'o_stream_set_no_error_handling()' to fully disable error checks. You can also +use 'o_stream_ignore_last_errors()' to ignore the errors so far, but not for +future writes. + +Writes are non-buffered by default. To add buffering, use 'o_stream_cork()' to +start buffering and 'o_stream_uncork()' to stop/flush. When output buffer gets +full, it's automatically flushed even while the stream is corked. The term +"cork" is used because with TCP connections the call actually sets/removes TCP +cork option. It's quite easy to forget to enable the corking with files, making +the performance worse. The corking/uncorking is done automatically when flush +callbacks are called. Using 'o_stream_uncork()' will trigger an automatic +'o_stream_flush()' but the error is ignored. This is why it acts similarly to +'o_stream_nsend*()', i.e. it requires another explicit 'o_stream_flush()', +'o_stream_finish()' or error ignoring before the ostream is destroyed. + +If output buffer's size isn't unlimited, the writes can also fail or be partial +because there's no more space in the buffer and 'write()' syscall is returning +'EAGAIN'. This of course doesn't happen with blocking fds (e.g. files), but you +need to handle this in some way with non-blocking network sockets. A common way +in Dovecot to handle this is to just use unlimited buffer sizes and after each +write check if the buffer size becomes too large, and when it does it stops +writing until more space is available. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.ParameterForwarding.txt b/doc/wiki/Design.ParameterForwarding.txt new file mode 100644 index 0000000000..4413b58649 --- /dev/null +++ b/doc/wiki/Design.ParameterForwarding.txt @@ -0,0 +1,65 @@ +Forwarding parameters in IMAP/POP3/LMTP/SMTP proxying +===================================================== + +Dovecot supports proxying various pieces of information and even variables for +various protocols when forwarding connection.It requires that the sender is +listed under 'trusted_networks'. For IMAP, it uses the 'ID' command, for other +protocols,'XCLIENT' is used. + +This feature is supported since v1.2, except for x-forward which was added in +v2.2.29. + +IMAP protocol +------------- + +For IMAP protocol, this is done by extending the ID command. The maximum length +of parameters to parse is 255 bytes.The parameters are forwarded as part of the +ID command key-value list. + +'5 ID (x-originating-ip "127.0.0.1" x-originating-port "143"... )' + +Supported parameters +-------------------- + + * 'x-originating-ip' - Client IP + * 'x-originating-port' - Client port + * 'x-connected-ip' - Server IP + * 'x-connected-port' - Server port + * 'x-proxy-ttl' - TTL which is reduced by each hop, loop prevention. When TTL + drops to 0, the connection is dropped. + * 'x-session-id' / 'x-session-ext-id' - Session ID to be used. + * 'x-forward-' - Forwarded variable, see + +POP3 protocol +------------- + +For POP3 protocol, this is done with custom 'XCLIENT' command which accepts a +space separated list of key=value parameters. + +Supported parameters +-------------------- + + * 'ADDR' - Client IP + * 'PORT' - Client port + * 'SESSION' - Session ID + * 'TTL' - TTL which is reduced by each hop, loop prevention. When TTL drops to + 0, the connection is dropped. + * 'FORWARD' - Base64 encoded key=value parameter to be stored. + +SMTP/LMTP protocol +------------------ + +See http://www.postfix.org/XCLIENT_README.html + + * 'ADDR' - Client IP, IPV6: prefix is needed for IPv6. + * 'PORT' - Client port + * 'SESSION' - Session ID + * 'HELO' - Original HELO/EHLO + * 'LOGIN' - Original LOGIN value + * 'TIMEOUT' - Original TIMEOUT + * 'TTL' - TTL which is reduced by each hop, loop prevention. When TTL drops to + 0, the connection is dropped. + * 'PROTO' - Forwarded protocol, can be one of SMTP, ESTMP, LMTP. + * 'FORWARD' - Base64 encoded key=value parameter to be stored. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Plugins.txt b/doc/wiki/Design.Plugins.txt new file mode 100644 index 0000000000..7d28af4204 --- /dev/null +++ b/doc/wiki/Design.Plugins.txt @@ -0,0 +1,62 @@ +Plugins +======= + +Plugins in Dovecot are really simple. They basically have two functions: + + * _init(module) is called when 'module_dir_init()' is called. + * _deinit() is called when 'module_dir_deinit()' or + 'module_dir_unload()' is called. + +The is the short version of the plugin name, based on the +filename. For example if the filename is 'lib11_imap_quota_plugin.so', the + is "imap_quota" and the init function to be called is +imap_quota_plugin_init(). + +Versioning +---------- + +Since different Dovecot versions can have different APIs, your plugin should +usually also define_version, like: + +---%<------------------------------------------------------------------------- +const char *imap_quota_plugin_version = DOVECOT_ABI_VERSION; +---%<------------------------------------------------------------------------- + +If the version string in plugin doesn't match the version of the running +binary, the plugin loading fails. The DOVECOT_ABI_VERSION is defined in +Dovecot's 'config.h', which you're typically including. + +Dependencies +------------ + +Some plugins depend on another one. In some systems (but not all) it's possible +to handle this by giving a nicer error message than "symbol xyz not found". +There are two steps for this: + +First create _dependencies array listing plugin names that the +plugin depends on, like: + +---%<------------------------------------------------------------------------- +const char *imap_quota_plugin_dependencies[] = { "quota", NULL }; +---%<------------------------------------------------------------------------- + +Then you'll also have to make the plugin .so binary link to the other plugins: + +---%<------------------------------------------------------------------------- +if PLUGIN_DEPS +lib11_imap_quota_plugin_la_LIBADD = \ + ../quota/lib10_quota_plugin.la +endif +---%<------------------------------------------------------------------------- + +PLUGIN_DEPS is set only if plugin dependencies are actually supported. +Otherwise the build might fail or plugin loading might fail. + +Once all this is done, trying to load imap_quota plugin without quota plugin +gives a nice error message: + +---%<------------------------------------------------------------------------- +Error: Can't load plugin imap_quota_plugin: Plugin quota must be loaded also +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Processes.txt b/doc/wiki/Design.Processes.txt new file mode 100644 index 0000000000..189960c7d1 --- /dev/null +++ b/doc/wiki/Design.Processes.txt @@ -0,0 +1,137 @@ +Dovecot processes +================= + +Dovecot is split into multiple processes where each process does only one +thing. This is partially because it makes the code cleaner, but alsobecause it +allows setting up different privileges for each process. Themost important +processes are: + + * Master process (dovecot) + * Login processes (imap-login, pop3-login) + * Authentication process (dovecot-auth) + * Mail processes (imap, pop3) + +Master process +-------------- + +This process keeps all the other processes running. If a child process dies, +another one is restarted automatically. It always runs as root,unless you're +specifically running everything under a single normal UID. + +The master process reads the configuration file and exports the settings to +other processes via environment variables. + +All logging also goes through master process. This avoids problems with +rotating log files, as there's only a single process to send a signal toreopen +the log file. Also writing to the same log file (if not using syslog)isn't +necessarily safe to do in multiple processes concurrently. + +Making the logging go through master process also gives a couple of advantages +from security and reliability point of view: All log lines canbe prefixed with +the process's name and the username of the user who was logged in, withoutthe +possibility for the process itself to forge them. Flooding logs canalso be +prevented. By default Dovecot allows non-privileged processes towrite 10 lines +per second before it begins to delay reading their input,which finally causes +the badly behaving process to start blocking onwriting to stderr instead of +eating all the CPU and disk space. + +In the Dovecot 2.0 design, the master process is split to three parts: the +Masterprocess which does nothing more than keep the processes running, the +configprocess which handles reading the configuration file (supporting also eg. +SQL storages!) and the log process which handles the logging. + +Login processes +--------------- + +The login processes implement the required minimum of the IMAP and POP3 +protocolsbefore a user logs in successfully. There are separate processes (and +binaries) to handle IMAP and POP3 protocols. + +These processes are run with least possible privileges. Unfortunately the +default UNIX security model still allows them to do much more than theywould +have to: Accept new connections on a socket, connect to new UNIXsockets and +read and write to existing file descriptors. Still, the loginprocess is by +default run under a user account that has no special accessto anything, and +inside a non-writable chroot where only a couple of filesexist. Doing any +damage inside there should be difficult. + +When a new connection comes, one of the login processes accept()s it. After +that the client typically does nothing more than ask the server'scapability +list and then log in. The client may also start TLS sessionbefore logging in. + +Authentication is done by talking to the authentication process. The login +process is completely untrusted by the authentication process, so even if +anattacker is able to execute arbitrary code inside a login process, they won't +be able tolog in without a valid username and password. + +After receiving a successful authentication reply from the authentication +process, the login process sends the file descriptor to the master processwhich +creates a new mail process and transfers the fd into it. Before doingthat, the +master process verifies from the authentication process that theauthentication +really was successful. + +By default each login process will handle only a single connection and +afterwards kill itself (but see SSL proxying below). This way attacker can't +see other people'sconnections. This can however be disabled +('login_process_per_connection=no'), in which case the security of the design +suffers greatly. + +The login processes handle SSL/TLS connections themselves completely. They keep +proxying the connection to mail processes for the entire lifetime ofthe +connection. This way if a security hole is found from the SSL library,an +authenticated user still can't execute code outside the login process. + +See for more information about different settings related to +login processes. + +Authentication process +---------------------- + +The authentication process handles everything related to the actual +authentication: SASL authentication mechanisms, looking up and verifyingthe +passwords and looking up user information. + +It listens for two different kinds of connections: untrusted authentication +client connections (from login processes) and master connections (frommaster +process, but also from Dovecot LDA). The client connections are onlyallowed to +try to authenticate. The master connections are allowed to askif an +authentication request with a given ID was successful, and also to lookup user +information based on a username. This user lookup feature is usedby Dovecot +LDA. + +Each client connection tells their process ID to the authentication process in +a handshake. If a connection with the same PID already exists, an erroris +logged and the new connection is refused. Although this makes DoSattacks +possible, it won't go unnoticed for long and I don't see this as areal issue +for now. + +Having the authentication process know the PID of the client connection allows +all authentication requests to be mapped to one specific client +connection.Since the master process knows the login process's real PID, it's +used whenasking from authentication process if the request was successful. This +makes it impossible for a login process to try to fake another login +process'slogin requests. Faking PIDs will also be quite pointless. + +Once the master process has done the verification request for a successful +authentication request, the request is freed from memory. The requests arealso +freed about 2 minutes after their creation, regardless of the statethey +currently are in. + +For blocking password and user database backends (eg. MySQL) separate "worker +processes" are used. Initially only one of them exists, butmore are created as +needed. [PasswordDatabase.PAM.txt] can be configured to use worker +processes instead of doing the forking itself, but this isn'tcurrently done by +default and there may be problems related to it. Also +[PasswordDatabase.CheckPassword.txt] currently does the forking itself. + +Mail processes +-------------- + +These processes handle the actual post-login mail handling using the privileges +of the logged in user. It's possible to chroot these processes,but practically +it's usually more trouble than worth. + +See [Design.MailProcess.txt] for their internal design +documentation. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.ErrorHandling.txt b/doc/wiki/Design.Storage.ErrorHandling.txt new file mode 100644 index 0000000000..749759862a --- /dev/null +++ b/doc/wiki/Design.Storage.ErrorHandling.txt @@ -0,0 +1,63 @@ +Lib-storage Error Handling +========================== + +'src/lib-storage/mail-error.h' describes different types of errors and has some +other error-related functions and macros. + +Only errors returning "int" can actually return a failure. + + * Functions that return a pointer will never return failure with NULL. Only + "find" type of functions can return NULL, which means "not found". + * Iterators usually work by the init() returning iterator pointer and next() + returning a boolean. If there were any errors in either init() or next(), + deinit() finally returns a failure. + +Getting lib-storage errors +-------------------------- + + * 'mailbox_list_*()' functions set their errors to the given mailbox_list + structure. You can get these errors with 'mailbox_list_get_last_error()'. + * All other functions that have some way of accessing mail_storage (mailbox, + mail, transactions, etc.) set their errors to the storage. You can get these + errors with 'mail_storage_get_last_error()'. + * Mail user and namespace functions have their own error handling, + typically by returning error strings as parameters. + * Both '*_get_last_error()' functions should be called soon after the error is + noticed, before other failing lib-storage calls overwrite the error. + * In deinit failures it usually doesn't matter if you get the first or the + last error, so it's easier to just call all the different deinit + functions and finally look up what the last failure was. + +Setting lib-storage errors +-------------------------- + +Errors can be set with two calls: + + * 'mail_storage_set_error()' and 'mailbox_list_set_error()' should be used + when the error is user's fault in some way. For example invalid mailbox + name, out of quota, etc. The error string will be shown to user. It won't be + written to a log file. + * 'mail_storage_set_critical()' and 'mailbox_list_set_critical()' should be + used when the error is a problem in the system and sysadmin should be + notified. For example out of disk space or just in general an unexpected + syscall failure. The error string that will be shown to user is the + "Internal error occurred", but it will be logged as an error. + * The reason for the separation of these two is: + 1. Only log errors that sysadmin can do something about. + 2. Never show user anything even potentially sensitive about the system, + such as path names. + +There are also a few other calls that aren't used as often, but can be helpful: + + * 'mail_storage_set_internal_error()' and 'mailbox_list_set_internal_error()' + simply set the user-visible error message to "Internal error occurred". + These can be used if the actual error was already logged. + * 'mail_storage_set_error_from_errno()' and + 'mailbox_list_set_error_from_errno()' set the user-visible error message + based on some common 'errno' values. Currently: + * EACCESS, EPERM, EROFS: Permission denied + * ENOSPC, EDQUOT: Not enough disk space + * ENOENT, ENOTDIR: Not found + * ELOOP: Directory structure is broken + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mail.txt b/doc/wiki/Design.Storage.Mail.txt new file mode 100644 index 0000000000..a86d722a94 --- /dev/null +++ b/doc/wiki/Design.Storage.Mail.txt @@ -0,0 +1,100 @@ +Mail +==== + +Mail is first allocated with 'mail_alloc()'. Mails always belong to a + [Design.Storage.Mailbox.Transaction.txt]. Even if mail is treated +read-only, Dovecot might write data to cache file, so whenever possible, mail +transactions should be committed. When mail is allocated, you can specify a +list of fields and headers that you're (most likely) going to need. This allows +Dovecot to optimize the later 'mail_get_*()' lookups so that it doesn't need to +parse the message multiple times. These fields are also added to cache file, so +you shouldn't list fields unless you're fairly certain you need them. + +If you need to jump around in the mailbox based on already known sequences or +UIDs, you can do this with 'mail_set_seq()' and 'mail_set_uid()'. This way you +don't have to go through all the trouble of setting up a mailbox search. + +Getting data +------------ + +Some of the mail fields can be accessed directly: + + * 'box': Mail's mailbox, same as the transaction's. + * 'transaction': Mail's transaction, the same that was given to + 'mail_alloc()'. + * 'seq': Currently selected message's sequence number. + * 'uid': Currently selected message's UID. + * 'expunged': We already detected that the message is expunged and can't be + accessed. This may also be set (and looked up) later when some + 'mail_get_*()' function fails. + * 'has_nuls' and 'has_no_nuls': Message body is known to (not) have NUL + characters. + +The final field is 'lookup_abort', which is write-only. Normally when doing a +'mail_get_*()' operation for a field that isn't in a cache, the field is +generated and added to cache. If you don't want this, but instead have figured +out some better optimized way to do non-cached lookups, you can change this +field so that 'mail_get_*()' lookups fail instead. This is primarily used by +searching code internally. + + * 'MAIL_LOOKUP_ABORT_NEVER': The default - do whatever it takes to return the + value. + * 'MAIL_LOOKUP_ABORT_READ_MAIL': If returning the value would require reading + message headers/body, abort. + * 'MAIL_LOOKUP_ABORT_NOT_IN_CACHE': If the value isn't already in cache, + abort. For example if looking up message's physical size would require + 'stat()'ing the file this wouldn't be done. + +Most of the 'mail_get_*()' should be fairly obvious in how they work. Only +functions returning int can fail, others don't. + + * Keywords can be looked up either by getting an array of keyword strings or + keyword indexes. The index lookups are slightly faster. Keyword indexes can + be converted to strings by using the 'status.keywords' array returned by + 'mailbox_get_status()'. + * 'mail_get_first_header()' return 0 if header wasn't found, 1 if it was. + * 'mail_get_special()' can return various special fields. If a special isn't + implemented by some backend, the call returns success and sets the value to + "". + * 'mail_get_stream()' returns an input stream that can be used to access the + mail. If this function is called multiple times, each call seeks the stream + to beginning. Don't unreference the stream, it's destroyed automatically. + * Typically the input contains the raw data in disk, lines may end with LF + or CRLF depending on how they're on disk. + * mbox drops Dovecot's internal headers from the stream (X-UID:, Status:, + etc.). + * Plugins (e.g. zlib) can also hook into this call and modify the input + stream. + +Sometimes you might notice that some looked up field is actually corrupted. For +example you might notice that input stream returns EOF earlier than previous +'mail_get_physical_size()' said. This might have been caused by various +different things, but in any case all you can really do then is to just call +'mail_set_cache_corrupted()' and try again. + +Setting data +------------ + +Some of the messages' metadata can be updated: + + * 'mail_update_flags()' and 'mail_update_keywords()' changes flags/keywords. + * Usually you should set 'modify_type' parameter to 'MODIFY_ADD' or + 'MODIFY_REMOVE', instead of replacing all the flags. That allows + concurrent flag updates not to overwrite each others changes. + * 'mail_expunge()' expunges a message. + +Other functions are mainly intended for mailbox replication or restoring an +existing mailbox (e.g. dsync): + + * 'mail_update_modseq()' can be used to increase message's modseq + * 'mail_update_uid()' can be used to give a new (higher) UID for the message. + If such UID can't be given, this call is just ignored. + * 'mail_update_pop3_uidl()' can be used to give a specific POP3 UIDL for the + message. This is used internally when 'pop3_save_uidl=yes'. + +Other metadata can't be changed. IMAP protocol requires that messages are +immutable, so it's not possible to change message's received date, headers or +body. If you wish to modify any of them, you need to create a new message and +expunge the old one. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.MailNamespace.txt b/doc/wiki/Design.Storage.MailNamespace.txt new file mode 100644 index 0000000000..1613a87f34 --- /dev/null +++ b/doc/wiki/Design.Storage.MailNamespace.txt @@ -0,0 +1,91 @@ +Mail Namespace +============== + +'src/lib-storage/mail-namespace.h' describes mail namespaces. See + for more information about what they are actually about. + +Hierarchy separators and namespace prefixes +------------------------------------------- + +A namespace and [Design.Storage.MailboxList.txt] has 1:1 +relationship. A namespace is mainly about dealing with hierarchy separators and +namespace prefixes, which mailbox list doesn't know or care much about. + +Mailbox lists have their native hierarchy separators. For example with FS +layout the separator is '/', because child mailboxes are physically in +subdirectories and '/' is the separator for directories. With Maildir++ layout +the separator is (currently) hardcoded to '.' in the maildir directory name, so +that's its native hierarchy separator. + +Dovecot allows separators to be configurable, so namespaces have two +separators: + + * 'ns->sep' is the configured virtual separator, which defaults to same as + native separator. + * 'ns->real_sep' is the native separator. + +Namespaces also have prefixes. The prefixes are visible for users/clients and +they appear to be part of the actual mailbox name. One commonly used namespace +prefix is "INBOX.", so all mailboxes (other than INBOX itself) appear to be +children of the INBOX. + +So the same mailbox can be visible in three different forms: + + * Virtual name uses the virtual separator and namespace prefix. For example + "INBOX/foo/bar". + * Storage name uses the native separator and doesn't have a namespace prefix. + For example "foo.bar". + * Physical directory name on disk can be different again. For example with + Maildir++ it could be ".../Maildir/.foo.bar" (not the leading dot before + "foo"). + +Users and owners +---------------- + +When accessing other users' shared mailboxes, there's a difference between a +namespace's user and owner: + + * 'ns->user' points to the mail user actually accessing the mailbox (e.g. the + IMAP connection's mail user). + * 'ns->owner' points to the mail user who shared the mailbox. + +The distinction can be important sometimes. For example if user adds or removes +messages from the shared mailbox, the owner's quota must be updated instead of +the user's. + +Functions +--------- + +Functions about finding namespaces: + + * 'mail_namespace_find()' returns namespace for given virtual name and changes + the virtual name -> storage name. It also has a few variations: + * 'mail_namespace_find_visible()' skips hidden=yes namespaces. + * 'mail_namespace_find_subscribable()' skips subscriptions=no namespaces. + * 'mail_namespace_find_unsubscribable()' skips subscriptions=yes + namespaces. + * 'mail_namespace_find_inbox()' returns the namespace with inbox=yes. (There + can be only one.) + * 'mail_namespace_find_prefix()' returns namespace that has the given prefix. + * 'mail_namespace_find_prefix_nosep()' does the same, but ignores the + trailing separator in prefix (e.g. "foo" would find namespace with + prefix=foo/). + +Functions about translating between virtual and storage names when the +namespace is known: + + * 'mail_namespace_fix_sep()' changes virtual separators -> native separators. + * 'mail_namespace_get_storage_name()' changes virtual name -> storage name. + * 'mail_namespace_get_vname()' changes storage name -> virtual name. + * 'mail_namespace_update_name()' returns FALSE if virtual name doesn't + actually match the given namespace. Otherwise returns TRUE and changes + virtual name -> storage name. + +A single namespace can currently point to only a single storage, but there is +already some code that attempts to make the transition to multiple storages per +namespace easier. In general you shouldn't try to access 'ns->storage' +directly. When creating new mailboxes,'mail_namespace_get_default_storage()' +returns the storage that should be used. For other purposes you should find the +storage via [Design.Storage.MailboxList.txt] functions. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.MailStorage.txt b/doc/wiki/Design.Storage.MailStorage.txt new file mode 100644 index 0000000000..3052d8f947 --- /dev/null +++ b/doc/wiki/Design.Storage.MailStorage.txt @@ -0,0 +1,41 @@ +Mail Storage +============ + +'src/lib-storage/mail-storage.h' and 'mail-storage-private.h' describes mail +storage. Mail storage is mainly about being a common container for its +mailboxes. For example with [MailboxFormat.dbox.txt] each storage +has one directory where all the message bodies are written to, while the +per-mailbox directories only contain index files. With other mailbox formats +mail storage doesn't do much else than allow allocating +[Design.Storage.Mailbox.txt]. + +The only public functions for mail storage are: + + * 'mail_storage_purge()' frees disk space used by expunged messages. Currently + the only mailbox format that uses this is multi-dbox. + * 'mail_storage_get_settings()' returns mail storage settings. + * 'mail_storage_set_callbacks()' can be used to specify "OK" and "NO" + callbacks, which are called when a long running operation wants to send a + status update. For example "OK Stale mailbox lock file detected, will + override in n seconds" or "NO Mailbox is locked, will abort in n seconds". + +Methods that mail storage backends need to implement are: + + * 'get_setting_parser_info()': Returns storage-specific settings parser + information. + * 'alloc()': Allocate memory for a storage and set its virtual functions. + * 'create(ns)': Initialize the storage based on given namespace settings. The + same storage can be used by other namespaces, but they don't call 'create()' + again. This function typically shouldn't fail, except when storage can't + handle the wanted namespace settings. + * 'destroy()': Destroys the storage. + * 'add_list(list)': Called every time the storage is attached to a new + namespace / mailbox list. + * 'get_list_settings(ns, set)': Used to get storage's default settings. + * 'autodetect(ns, set)': Returns TRUE if based on the given settings it looks + like this storage should be handling the namespace. This is done when + mail_location doesn't explicitly specify the mailbox format. + * 'mailbox_alloc()': Allocate memory for + [Design.Storage.Mailbox.txt]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.MailUser.txt b/doc/wiki/Design.Storage.MailUser.txt new file mode 100644 index 0000000000..1656aef08f --- /dev/null +++ b/doc/wiki/Design.Storage.MailUser.txt @@ -0,0 +1,25 @@ +Mail User +========= + +'src/lib-storage/mail-user.h' describes mail user. The struct contains all +kinds of useful information about the user that can be accessed directly. Some +of the most useful things you can do with a user are: + + * 'user->username' gives you the actual username string (e.g. + 'user@domain.org'). + * 'user->set' gives you access to user's settings. + * 'user->namespaces' points to a linked list of user's namespaces. + * 'mail_user_get_home()' returns user's home directory, if there's one. + * 'mail_user_home_expand()' expands '~/' at the beginning of given path to + user's actual home directory. + * 'mail_user_plugin_getenv()' returns value for a setting defined in 'plugin + {}' section. + +Typically each new IMAP/POP3/etc. connection creates a single mail user. +Currently multiple connections for same user don't even try to share the mail +user, but this may change in future. If a user has shared mailboxes from other +users (not public namespaces), a mail user is also created whenever necessary +to list/access the user's mailboxes. Again there is no attempt to share the +created mail user with other connections. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mailbox.Save.txt b/doc/wiki/Design.Storage.Mailbox.Save.txt new file mode 100644 index 0000000000..69ee7549da --- /dev/null +++ b/doc/wiki/Design.Storage.Mailbox.Save.txt @@ -0,0 +1,51 @@ +Mailbox Saving +============== + +Both saving and copying messages begins by calling 'mailbox_save_alloc()'. +After that you can set message's metadata fields: + + * 'mailbox_save_set_flags()' sets flags and keywords. + * 'mailbox_save_set_received_date()' sets message's received date (IMAP + INTERNALDATE). It also supports specifying timezone, but most backends don't + support saving it. + * 'mailbox_save_set_dest_mail()' specifies a mail that can be used to access + the partially saved mail after save/copy is finished (but not committed). + You should be able to do pretty much anything with the mail, but its UID is + usually 0 at this point. + * 'mailbox_save_set_from_envelope()' sets the envelope sender. Currently this + is only used by mbox format to specify the address in From_-line. + +When copying, most of the metadata fields are automatically copied from the +source message. The only exception is message's flags and keywords. If you want +to preserve them, the easiest way is to call 'mailbox_save_copy_flags()'. + +Some metadata fields are mainly useful when you're replicating or restoring an +existing mailbox and want to preserve metadata: + + * 'mailbox_save_set_min_modseq()' sets message's modseq to at least the + specified modseq. If the modseqs are already higher in the mailbox, the + resulting modseq is what it would have been without this call. + * 'mailbox_save_set_uid()' sets message's UID. If mailbox's next_uid is + already higher than the specified UID, the UID request is ignored. + * 'mailbox_save_set_guid()' sets message's globally unique ID. A new GUID is + generated by default, and if there already exists a message with the same + GUID a different one may or may not be given. For example with maildir the + GUID is same as the base filename, while dbox has an explicit GUID metadata + field. + * 'mailbox_save_set_pop3_uidl()' sets POP3 UIDL value. Not all backends + support this. + * 'mailbox_save_set_save_date()' sets "message saved" date, i.e. the date/time + when the message was saved to this mailbox. The default is the current time. + +Once you're done with setting the metadata fields, you can either copy an +existing mail with 'mailbox_copy()' or provide message body with: + + * 'mailbox_save_begin()' starts saving from given input stream. + * 'mailbox_save_continue()' saves all data from input stream. If input stream + is blocking, typically a single call to this function should be enough. If + input stream is non-blocking, you need to call this function until you're + done. In any case call this until 'i_stream_is_eof()' returns TRUE. + * 'mailbox_save_finish()' finishes saving the mail, or 'mailbox_save_cancel()' + aborts it. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mailbox.Search.txt b/doc/wiki/Design.Storage.Mailbox.Search.txt new file mode 100644 index 0000000000..d1438baec4 --- /dev/null +++ b/doc/wiki/Design.Storage.Mailbox.Search.txt @@ -0,0 +1,101 @@ +Mailbox Searching +================= + +'mailbox_search_*()' functions should be used always when you're iterating +through multiple messages. The search queries can be complex or as simple as +"all messages". Without searching there's also a way to directly switch to a +specific message by its sequence number or UID, but this should be avoided +usually. + +Initializing +------------ + +Search is initialized with 'mailbox_search_init()' on top of a +[Design.Storage.Mailbox.Transaction.txt]. Although it would appear that a +search is a read-only operation, it can actually write data to Dovecot's cache +file. For example if you search for a specific header not yet in cache file, +the results are saved to cache file so the next search will be fast. This is +why you should always commit search transactions, even if the rest of your +operation fails (you should use separate transactions for search and for +updates if necessary). + +You'll need a search query. 'src/lib-storage/mail-search.h' and +'mail-search-build.h' contain all the functions and structures related to it. +Usually you should start with 'mail_search_build_init()' and then start adding +the search parameters, either manually or with one of the existing +'mail_search_build_add_*()' helper functions. The same search query structures +can be saved and used for other searches later, but because search state is +stored inside the structs you need to be careful that multiple searches aren't +using them at the same time. This is usually more trouble than worth, so avoid +doing it. + +Search results can be sorted by giving a sort program. Dovecot optimizes this +by keeping sort indexes in the index files. + +Reading search results +---------------------- + +While 'mailbox_search_next()' returns TRUE, a new search result is found and it +changes the given [Design.Storage.Mail.txt] to point to the search +result. The mail's "wanted fields/headers" parameters don't need to include +anything needed by the search query, Dovecot optimizes them internally. + +If the search needs to parse message bodies and the mailbox is large, this call +can take a long time. If you want to do other things while searching, you can +use 'mailbox_search_next_nonblock()' that does only a bit of work and then +returns either with a result or "try again later" status. Dovecot attempts to +keep each non-matching call to this function between 200 and 250 milliseconds, +although the upper bound can't be guaranteed. + +It's possible that messages are being expunged while Dovecot is searching them, +so it can't determine if they would have matched the search rule or not. In +this case it skips over them, but if you want to know if this has happened, you +can see if 'mailbox_search_seen_lost_data()' returns TRUE. + +Deinitializing +-------------- + +'mailbox_search_deinit()' finishes the search. If it returns -1, some error +occurred and some search results might not have been returned. + +Example +------- + +Iterating through all messages in a mailbox goes like: + +---%<------------------------------------------------------------------------- +/* build search query */ +search_args = mail_search_build_init(); +mail_search_build_add_all(search_args); + +search_ctx = mailbox_search_init(trans, search_args, NULL); +/* search context keeps args referenced */ +mail_search_args_unref(&search_args); + +mail = mail_alloc(trans, 0, NULL); +while (mailbox_search_next(ctx, mail)) { + printf("matched uid %u\n", mail->uid); +} +mail_free(&mail); +if (mailbox_search_deinit(&search_ctx) < 0) + i_error("search failed"); +---%<------------------------------------------------------------------------- + +Saving search results +--------------------- + +Search results can be saved for future use by calling +'mailbox_search_result_save()' just after initializing the search. The results +as returned as UIDs with 'mailbox_search_result_get()' and may contain UIDs +that are already expunged. Once you're done with the saved result, free it with +'mailbox_search_result_free()'. + +The search result can also be automatically updated whenever mailbox is synced +if 'MAILBOX_SEARCH_RESULT_FLAG_UPDATE' is set. The update is optimized, so +Dovecot doesn't do a full re-search, but matches only new and changed messages. +If 'MAILBOX_SEARCH_RESULT_FLAG_QUEUE_SYNC' is also set, search result additions +and removals are also tracked and can be retrieved with +'mailbox_search_result_sync()', i.e. with this you can implement "what changed +in search results since last time I checked". + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mailbox.Sync.txt b/doc/wiki/Design.Storage.Mailbox.Sync.txt new file mode 100644 index 0000000000..a7d295a77a --- /dev/null +++ b/doc/wiki/Design.Storage.Mailbox.Sync.txt @@ -0,0 +1,128 @@ +Mailbox Synchronization +======================= + +The idea behind synchronization is to find out what changes other sessions have +done to the mailbox and to finalize our own changes to the mailbox. + +For example if you expunge a message in a transaction and commit it, the commit +will only write a "please expunge UID n" record to Dovecot's transaction log +file. The message still exists on the disk. The next time Dovecot syncs the +mailbox (either the session that wrote the record or another one), it goes +through all the non-synchronized records in transaction log and applies the +requested changes to the backend mailbox. Syncing can be a bit heavyweight +operation, so it's possible to commit multiple transactions and perform a +single sync for all of them. Dovecot attempts to do this with IMAP protocol +when pipelining commands. + +The other important job of syncing is to refresh mailbox's state: + + * Finding out about external modifications to mailbox (e.g. a new mail + delivered to Maildir/new/). + * Updating in-memory view of what messages exist, what their flags are, etc. + +When a mailbox is opened, its state starts with what index files contain at the +time. Since the backend mailbox may have already changed, and syncing an +up-to-date mailbox is usually really cheap, there isn't much point in not +syncing mailbox immediately after opening. The mailbox state stays the same +until you synchronize the mailbox again, before that no new messages show up +and no messages get expunged. + +Typically you would sync the mailbox + + * after committing a transaction that modifies backend mailbox in any way + (instead of just internal index data), such as after changing message flags + or expunging a message. + * whenever you want to find out if there are any changes. With IMAP protocol + this is done every time after running a command. + +Initializing +------------ + +'mailbox_sync_init()' initializes syncing. + +There are some flags that control how much effort is spent on syncing: + + * 'MAILBOX_SYNC_FLAG_FAST' can be given when you're ready for mailbox to be + refreshed, but don't care much if it actually is or not. When this flag is + set, Dovecot still notices all internal changes, but external changes are + checked only once every few seconds or so. + * 'MAILBOX_SYNC_FLAG_FULL_READ' is mainly useful with mboxes. If + 'mbox_dirty_syncs=yes' and a new mail gets appended to mbox by an external + program, Dovecot assumes that the only change was the added mail, even + though the program may have also modified existing messages' flags by + rewriting Status: headers. If 'mbox_very_dirty_syncs=no', these changes are + noticed after the next time mailbox is opened. So when this flag is enabled, + it means Dovecot should try harder to find out if there were any external + unexpected changes. It's currently used only with IMAP SELECT and CHECK + commands and POP3 startup. Probably unnecessary elsewhere. + * 'MAILBOX_SYNC_FLAG_FULL_WRITE' is again mainly useful with mboxes. If + 'mbox_lazy_writes=no', Dovecot delays writing flag changes to mbox file + until mailbox is closed or IMAP CHECK command is issued. Using this + elsewhere is probably unnecessary, except as an optimization if mailbox is + in any case synced just before closing it, you might as well give this flag + to it to avoid double-syncing with mbox. + * 'MAILBOX_SYNC_FLAG_FORCE_RESYNC' is used to force resyncing indexes. The + only time this should be done is when manually triggered by administrator. + +Then there are also other syncing flags: + + * 'MAILBOX_SYNC_FLAG_NO_EXPUNGES': No expunged messages are removed from the + in-memory mailbox view. Their removal is delayed until syncing is done + without this flag. Attempting to access the expunged messages may or may not + work, depending on what information is accessed and what storage backend is + used. + * 'MAILBOX_SYNC_FLAG_FIX_INCONSISTENT': Normally when the internal mailbox + state can't be consistently updated (typically due to index file + corruption), the syncing fails. When this flag is set, it means that the + caller doesn't care about mailbox's previous state and just wants to get it + accessible again. Typically this is used when the mailbox is being opened, + but not afterwards. + * 'MAILBOX_SYNC_FLAG_EXPUNGE' is mainly intended for virtual plugin with IMAP + protocol. You probably shouldn't use it. + +Reading changes +--------------- + +While 'mailbox_sync_next()' returns TRUE, it fills out sync record: + + * seq1, seq2: Message sequence numbers that were affected + * type: expunge, flag change or modseq change. + +Expunge records don't immediately change the view's sequence numbers. After +seeing an expunge record you can still fetch the expunged messages' flags and +possibly other information. Only after syncing is deinitialized, the sequences +change. + +Message flag change records don't actually show what the changes were. You can +find the new flags just by fetching them ('mail_get_flags()', etc.), they're +available immediately. You'll need to create a +[Design.Storage.Mailbox.Transaction.txt] and a [Design.Storage.Mail.txt] +for that. For example: + +---%<------------------------------------------------------------------------- +sync_ctx = mailbox_sync_init(box, flags); +trans = mailbox_transaction_begin(box, 0); +mail = mail_alloc(trans, MAIL_FETCH_FLAGS, 0); +---%<------------------------------------------------------------------------- + +If you don't actually care about sync records, you don't necessarily have to +even call 'mailbox_sync_next()'. In that case it's actually easiest to perform +the whole sync using a one-step 'mailbox_sync()' function. This function also +sets 'MAILBOX_SYNC_FLAG_FIX_INCONSISTENT' flag automatically. + +Deinitializing +-------------- + +'mailbox_sync_deinit()' finalizes the syncing. If any errors occurred during +sync, it'll return -1. + +If 'MAILBOX_SYNC_FLAG_NO_EXPUNGES' was used and some expunges were actually +delayed,'status_r->sync_delayed_expunges' is set to TRUE. + +Implementing sync for a storage backend +--------------------------------------- + +FIXME: talk about mail_index_sync_*() and how to change stuff and how to update +internal state. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mailbox.Transaction.txt b/doc/wiki/Design.Storage.Mailbox.Transaction.txt new file mode 100644 index 0000000000..63a37e1a36 --- /dev/null +++ b/doc/wiki/Design.Storage.Mailbox.Transaction.txt @@ -0,0 +1,52 @@ +Mailbox Transactions +==================== + +Before you can read any mails or do any changes to mails, you need to create a +transaction with 'mailbox_transaction_begin()'. It has a few flags: + + * 'MAILBOX_TRANSACTION_FLAG_HIDE': Mark the changes in a way that when later + [Design.Storage.Mailbox.Sync.txt] the mailbox in this session, + 'mailbox_sync_next()' won't return sync records for the changes done by this + transaction. This is primarily meant for flag and keyword changes, you can't + hide expunges. For example IMAP's 'STORE FLAGS.SILENT' command is + implemented by setting this flag for the transaction. + * 'MAILBOX_TRANSACTION_FLAG_EXTERNAL': Changes done by this transaction should + be marked as external changes. Internal changes can be thought of as "change + requests" that syncing later finishes, while external changes are done + immediately and syncing ignores them. Normally you would use this flag when + you want to save or copy messages, nothing else. + * 'MAILBOX_TRANSACTION_FLAG_ASSIGN_UIDS': Require assigning UIDs to + saved/copied messages immediately. Normally this is done only when it's easy + (maildir: if dovecot-uidlist can be locked without waiting, mbox: if mbox is + already fully synced). + * 'MAILBOX_TRANSACTION_FLAG_REFRESH': Do a quick index refresh, so any recent + flag/modseq changes done by other Dovecot sessions will be visible. You + shouldn't usually need this, because usually you should have recently done a + mailbox sync. + +Changes for a transaction are kept in memory until the transaction is +committed. If you want to cancel the changes, you can call +'mailbox_transaction_rollback()'. Transaction can be commited with +'mailbox_transaction_commit()'. If you want to know a bit more about the +results of the transaction, use 'mailbox_transaction_commit_get_changes()' +instead. It returns a change structure: + + * uid_validity: UIDVALIDITY used by returned UIDs + * saved_uids: UIDs assigned to saved/copied mails. Typically they're in an + ascending order, unless you explicitly requested some specific UIDs for + mails while saving them (e.g. dsync does this). + * ignored_uid_changes: Number of UIDs that couldn't be changed by + 'mail_update_uid()' calls, because the UIDs were less than next_uid's value. + + * ignored_modseq_changes: Number of modseqs that couldn't be changed by + 'mail_update_modseq()' calls, because they would have lowered the modseq. + +Once you're done with reading the change structure, be sure to free the memory +used by it with 'pool_unref(&changes->pool)'. + +'mailbox_transaction_set_max_modseq()' can be used to implement atomic +conditional flag changes. If message's modseq is higher than the given +max_modseq while transaction is being committed, the change isn't done and the +message's sequence number is added to the given array. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Mailbox.txt b/doc/wiki/Design.Storage.Mailbox.txt new file mode 100644 index 0000000000..5af8263136 --- /dev/null +++ b/doc/wiki/Design.Storage.Mailbox.txt @@ -0,0 +1,34 @@ +Mailbox +======= + +'src/lib-storage/mail-storage.h' and 'mail-storage-private.h' describes mailbox +API, among others. Mailbox life cycle often goes like: + + * 'mailbox_alloc()' allocates memory for the mailbox and initializes some + internal settings, but doesn't actually try to open it. + * 'mailbox_open()' opens the mailbox. Instead of opening a mailbox, you can + also create it with 'mailbox_create()'. + * If you're immediately syncing the mailbox, you don't need to open it, + because it's done implicitly. This reduces your code and error handling a + bit. + * 'mailbox_close()' closes the mailbox, so that it needs to be opened again if + it's wanted to be accessed. This is rarely needed. + * 'mailbox_free()' closes and frees the mailbox. + +There are a lot of functions to deal with mailboxes. The most important ones +are: + + * 'mailbox_get_status()' to get a summary of mailbox, such as number of + messages in it. + * [Design.Storage.Mailbox.Sync.txt]: 'mailbox_sync_*()' to + synchronize changes from the backend to memory. + * [Design.Storage.Mailbox.Transaction.txt]: + 'mailbox_transaction_*()' for transaction handling. All message reads and + writes are done in a transaction. + * [Design.Storage.Mailbox.Search.txt]: 'mailbox_search_*()' is + used for searching messages. Even simple operations like "get all messages" + go through this API, it'll then simply do "search all". + * [Design.Storage.Mailbox.Save.txt]: 'mailbox_save_*()' and + 'mailbox_copy()' is used for saving/copying new messages to mailbox. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.MailboxList.txt b/doc/wiki/Design.Storage.MailboxList.txt new file mode 100644 index 0000000000..6c27883848 --- /dev/null +++ b/doc/wiki/Design.Storage.MailboxList.txt @@ -0,0 +1,159 @@ +Mailbox List +============ + +'src/lib-storage/mailbox-list.h' and 'mailbox-list-private.h' describes mailbox +list. The purpose of mailbox list is to manage mailbox storage name<-> physical +directory path mapping. Its most important functions are: + + * listing existing mailboxes, + * creating directories for new mailboxes (but not the mailboxes themselves, + that's storage's job), + * deleting mailboxes, + * renaming mailboxes and + * managing mailbox subscriptions. + +Mailbox list code also internally creates and updates mailbox changelog (in +'dovecot.mailbox.log' file), which keeps track of mailbox deletions, renames +and subscription changes. This is primarily useful for dsync utility. + +Mailbox list is configured by [MailLocation.txt] setting, which +fills 'struct mailbox_list_settings': + + * root_dir: The root mail directory (e.g. with + 'mail_location=maildir:~/Maildir' it would be the '~/Maildir'). + * index_dir: Directory under which index files are written to. Empty string + means in-memory indexes. Defaults to root_dir. + * control_dir: Directory under which control files are written to. Control + files are files that contain some important metadata information about + mailbox so (unlike index files) they should never be deleted. For example + subscriptions file is a control file. Defaults to root_dir. + * alt_dir: This is currently [MailboxFormat.dbox.txt]-specific setting. + + * inbox_path: Path to INBOX mailbox. This exists mainly because with mbox + format INBOX is often in a different location than other mailboxes. + * subscription_fname: Filename used by subscriptions file. + * dir_guid_fname: Filename used to store directories' (not mailboxes') global + UIDs. Directory GUIDs are mainly useful for dsync. + * maildir_name: Directory name under which the actual mailboxes are stored in, + such as dbox-Mails/ with dbox. See the .h file for more detailed + description. + * mailbox_dir_name: If non-empty, store all mailboxes under + root_dir/mailbox_dir_name/. + +Listing mailboxes +----------------- + +First the list operation is initialized with one of the init functions: + + * 'mailbox_list_iter_init()' lists mailboxes that match the given pattern. + * 'mailbox_list_iter_init_multiple()' lists mailboxes that match any of the + given patterns list. + * 'mailbox_list_iter_init_namespaces()' lists matching mailboxes from all + namespaces. + * 'MAILBOX_LIST_ITER_SKIP_ALIASES' flag skips namespaces that have + 'alias_for' set. You usually want to set this flag to avoid processing + the same mailbox multiple times. + +The patterns are IMAP-style patterns with '%' and '*' wildcards as described by +RFC 3501: '%' matches only up to next hierarchy separator, while '*' matches +the rest of the string. + +These flags control what mailboxes are returned: + + * 'MAILBOX_LIST_ITER_NO_AUTO_INBOX' doesn't list INBOX unless it physically + exists. Normally INBOX is listed, because INBOX doesn't need to be (and + cannot be) explicitly created. It can always be opened and messages can be + saved to it, it's just automatically created when it doesn't exist. + * 'MAILBOX_LIST_ITER_SELECT_SUBSCRIBED' lists only subscribed mailboxes. + * 'MAILBOX_LIST_ITER_SELECT_RECURSIVEMATCH' is currently only useful when + combined with '_SELECT_SUBSCRIBED' flag. Then it adds + 'MAILBOX_CHILD_SUBSCRIBED' flags for mailboxes whose children are + subscribed. It also lists mailboxes that aren't themselves subscribed, but + have children that do. + +These flags control what is returned for matching mailboxes: + + * 'MAILBOX_LIST_ITER_RETURN_NO_FLAGS' can be set when you don't care about + mailbox flags. They're then set only if it can be done without any + additional disk I/O. + * 'MAILBOX_LIST_ITER_RETURN_SUBSCRIBED' returns mailbox's subscription state. + * 'MAILBOX_LIST_ITER_RETURN_CHILDREN' sets "has child mailboxes" or "doesn't + have child mailboxes" flag. + +Other flags: + + * 'MAILBOX_LIST_ITER_RAW_LIST' should usually be avoided. It ignores ACLs and + just returns everything. + * 'MAILBOX_LIST_ITER_VIRTUAL_NAMES' enables listing to use virtual names + instead of storage names in patterns and returned mailbox names. + +Once listing is initialized, 'mailbox_list_iter_next()' can be called until it +returns NULL. The returned mailbox_info struct contains: + + * 'name': Mailbox's name, either virtual or storage name depending on + '_VIRTUAL_NAMES' flag. + * 'ns': Mailbox's namespace. This is useful only when mailboxes are listed + using 'mailbox_list_iter_init_namespaces()'. + * 'flags': Mailbox flags: + * 'MAILBOX_NOSELECT': Mailbox exists, but can't be selected. It's possible + that it can be created and then it becomes selectable. For example with + mbox and FS layout the directories aren't selectable mailboxes. + * 'MAILBOX_NONEXISTENT': Mailbox doesn't exist. It's listed only because it + has child mailboxes that do exist but don't match the pattern. + * Example: "foo/bar" exists, but "foo" doesn't. "%", "foo" or "*o" + pattern would list "foo", because it matches the pattern but its child + doesn't. Then again "*", "*bar" or "%/%" wouldn't list "foo", because + "foo/bar" matches the pattern (and is also listed). Something like + "*asd*" wouldn't match either "foo" or "foo/bar" so neither is + returned. + * 'MAILBOX_CHILDREN' and 'MAILBOX_NOCHILDREN': Mailbox has or doesn't have + children. If neither of these flags are set, it's not known if mailbox + has children. + * 'MAILBOX_NOINFERIORS': Mailbox doesn't have children and none can ever be + created. For example with mbox and FS layout the mailboxes have this flag + set, because files can't be created under files. + * 'MAILBOX_MARKED' and 'MAILBOX_UNMARKED': Mailbox has or doesn't have + messages with \Recent flags. If neither is set, the state is unknown. + Because this check is done in a very cheap way, having 'MAILBOX_MARKED' + doesn't always mean that there are \Recent flags. However, if + 'MAILBOX_UNMARKED' is returned it is guaranteed to be correct. (False + positives are ok, false negatives are not ok.) + * 'MAILBOX_SUBSCRIBED': Mailbox is subscribed. + * 'MAILBOX_CHILD_SUBSCRIBED': Mailbox has a child that is subscribed (and + '_SELECT_RECURSIVEMATCH' flag was set). + +Finally the listing is deinitalized with 'mailbox_list_iter_deinit()'. If it +returns -1, it means that some mailboxes perhaps weren't listed due to some +internal error. + +If you wish to get mailbox_info flags only for a single mailbox, you can use +'mailbox_list_mailbox()'. + +Directory permissions +--------------------- + +'mailbox_list_get_permissions()' and 'mailbox_list_get_dir_permissions()' can +be used to get wanted permissions for newly created files and directories. + + * For global files, give NULL as the mailbox name. The permissions are then + based on the root_dir. If root_dir doesn't exist, it returns 0700/0600 mode. + * For per-mailbox files, give the mailbox name. The permissions are then based + on the mailbox's directory. + +The returned permissions are: + + * mode: Creation mode, like 0600. + * gid: Group that should be set, unless it's '(gid_t)-1'. There are 3 reasons + why it could be that: + * directory has g+s bit set, so the wanted group is set automatically + * group is the same as process's effective GID, so it gets set + automatically + * mode's group permissions are the same as world permissions, so group + doesn't matter. + * gid_origin: This string points to the directory where the group (and + permissions in general) was based on, or "defaults" for internal defaults. + +If changing the group fails with EPERM, 'eperm_error_get_chgrp()' can be used +to log a nice and understandable error message. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Storage.Plugins.txt b/doc/wiki/Design.Storage.Plugins.txt new file mode 100644 index 0000000000..349ab2eb7e --- /dev/null +++ b/doc/wiki/Design.Storage.Plugins.txt @@ -0,0 +1,124 @@ +Mail Plugins +============ + +Typically plugins add hooks in their init() function by calling +'mail_storage_hooks_add()', and remove the hooks at deinit() with +'mail_storage_hooks_remove()'. Hooks that are currently supported: + + * mail_user_created: A new mail user was created. It doesn't yet have any + namespaces. + * mail_storage_created: A new mail storage was created. It's not connected to + any namespaces/mailbox lists yet. + * mailbox_list_created: A new mailbox list was created. It's not connected to + any storages yet. Because of this, some internal virtual methods haven't + been overridden by the storage yet, so plugins rarely want to use this hook. + Instead they should use: + * mail_namespace_storage_added: Storage was connected to its first + namespace/mailbox list. This hook should usually be used if plugin wants to + override mailbox_list's methods. + * mail_namespaces_created: User's all namespaces have been created. This hook + is called only per user at startup. More internal namespaces may be created + later when using shared mailboxes. + * mailbox_allocated: 'mailbox_alloc()' was called. + * mailbox_opened: Mailbox (and its index) was actually opened, either + explicitly with 'mailbox_open()' or implicitly by some other function. + +Overriding methods +------------------ + +When the hook gets called, you usually want to override some method of the +created object. This is the easy part, for example: + +---%<------------------------------------------------------------------------- +static void plugin_mailbox_allocated(struct mailbox *box) +.. + box->v.transaction_begin = plugin_transaction_begin; +---%<------------------------------------------------------------------------- + +The problem is that once 'plugin_transaction_begin()' is called, it should call +the original 'transaction_begin()'. There may also be multiple plugins that +want to override the same method, so the idea is to just have each plugin call +the previous 'transaction_begin()'. The next problem is where do you save the +previous value? Most objects have a 'module_contexts' array for storing +per-plugin pointers for this purpose. There are several helper functions to +make setting and accessing them in a quite safe way. + +Easiest way to set up the module context is to just copy&paste code from an +existing plugin that sets the same context. Here's some documentation about it +anyway: + +First you start by creating register for the plugin. There are different +registers for different types of objects: + + * mail_user_module_register: For mail_user. + * mailbox_list_module_register: For mailbox_list. + * mail_storage_module_register: For mail_storage, mailbox, mailbox_transaction + and mail_search. + * mail_module_register: For mail. + +We'll assume you want to use mail_storage_module_register: + +---%<------------------------------------------------------------------------- +static MODULE_CONTEXT_DEFINE_INIT(plugin_storage_module, +&mail_storage_module_register); +---%<------------------------------------------------------------------------- + +If you need to make it external, use: + +---%<------------------------------------------------------------------------- +extern MODULE_CONTEXT_DEFINE(plugin_storage_module, +&mail_storage_module_register); +struct plugin_storage_module plugin_storage_module = + MODULE_CONTEXT_INIT(&mail_storage_module_register); +---%<------------------------------------------------------------------------- + +Next you'll need to allocate memory for the structure you want to place in the +context. If you only want to override some methods, you can use: + +---%<------------------------------------------------------------------------- +union mailbox_module_context *mbox; +struct mailbox_vfuncs *v = box->vlast; + +mbox = p_new(box->pool, union mailbox_module_context, 1); +mbox->super = *v; +box->vlast = &mbox->super; + +v->transaction_begin = plugin_transaction_begin; +MODULE_CONTEXT_SET_SELF(box, plugin_storage_module, mbox); +---%<------------------------------------------------------------------------- + +If you want to store some more plugin-specific data to the object instead of +just the super methods, you can do: + +---%<------------------------------------------------------------------------- +struct plugin_mailbox { + /* must be called module_ctx */ + union mailbox_module_context module_ctx; +}; +/* .. */ + +struct plugin_mailbox *mbox; +struct mailbox_vfuncs *v = box->vlast; + +mbox = p_new(box->pool, struct plugin_mailbox, 1); +mbox->module_ctx.super = *v; +box->vlast = &mbox->super; + +v->transaction_begin = plugin_transaction_begin; +MODULE_CONTEXT_SET(box, plugin_storage_module, mbox); +---%<------------------------------------------------------------------------- + +Note that when using union directly you use 'MODULE_CONTEXT_SET_SELF()', while +when it's inside a struct you use 'MODULE_CONTEXT_SET()'. + +Once all this initialization is done, you can look up the module context with: + +---%<------------------------------------------------------------------------- +#define PLUGIN_CONTEXT(obj) MODULE_CONTEXT(obj, plugin_storage_module) +/* .. */ +struct plugin_mailbox *mbox = PLUGIN_CONTEXT(box); +---%<------------------------------------------------------------------------- + +(Yes, this API seems a bit too difficult to use and could use a redesign.) + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.Strings.txt b/doc/wiki/Design.Strings.txt new file mode 100644 index 0000000000..72ca055bca --- /dev/null +++ b/doc/wiki/Design.Strings.txt @@ -0,0 +1,70 @@ +Dynamic Strings +=============== + +'lib/str.h' describes Dovecot's dynamically growing strings. Strings are +actually only a simple wrapper on top of [Design.Buffers.txt]. Even +the 'string_t' type is only a typedef of 'buffer_t', so it's possible to use +'buffer_*()' functions with strings (although it's ugly so it should be +avoided). The decision of whether to use a string_t or a buffer_t is mainly for +human readability: if the buffer's contents are (ASCII/UTF8) text use string_t, +otherwise for binary data use buffer_t. + +Once you're done modifying a string with 'str_*()' functions, you can get it +out as a NUL-terminated string with 'str_c()' or 'str_c_modifiable()'. These +pointers shouldn't be accessed after modifying the string again, they could +have moved elsewhere in memory and they're no longer guaranteed to be +NUL-terminated. + +Example: + +---%<------------------------------------------------------------------------- +T_BEGIN { + string_t *str = t_str_new(64); + + str_append(str, "hello world"); + str_printfa(str, "\nand %u", str_len(str)); + + printf("%s\n", str_c(str)); +} T_END; +---%<------------------------------------------------------------------------- + +String Handling Functions +========================= + +'lib/strfuncs.h' contains a lot of functions intended to make string handling +easier. They use C's NUL-terminated strings instead of Dovecot's dynamic +strings. + + * '[pt]_strdup_printf()' and '[pt]_strconcat()' are the most commonly used + functions.'*_strconcat' is slightly faster than '*_strdup_printf()', so use + it if you simply need to concatenate strings. + * Various functions for doing a 'strdup()' from wanted input. + * 'i_snprintf()' is a wrapper to 'snprintf()' that makes it easier to check if + result was truncated. It also adds few other safety checks. This should be + avoided in general, except in situations where you just don't want to use + data stack and there's no way for the result to get truncated. + * 'i_strocpy()' is similar to 'strlcpy()', but makes it easier to check if + result was truncated. This has the same problems as 'i_snprintf()'. + * Functions for uppercasing and lowercasing strings. + * Functions you can pass to 'bsearch()' and 'qsort()' for handling string + arrays. + * '[pt]_strsplit()' is an easy way to split a string into an array of strings + from given separator. + * 't_strarray_join()' reverses this. + * There are also some other functions to handle array of strings, like + getting its length or finding a given string. + * 'dec2str()' can be used to convert a number to a string. This can be useful + if you don't know the correct type and don't want to add casting (that could + potentially truncate the string). For example:'print("pid = %s\n", + dec2str(getpid()));' + +String Escaping +=============== + +'lib/strescape.h' contains functions to escape and unescape <">. <'> and <\> +characters in strings using<\> character.\ + +Dovecot's internal protocols are often line-based with TAB as the field +separator. This file also contains functions to escape and unescape such data. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Design.txt b/doc/wiki/Design.txt new file mode 100644 index 0000000000..c27f4ec1b6 --- /dev/null +++ b/doc/wiki/Design.txt @@ -0,0 +1,66 @@ +Dovecot Design +============== + + * [Design.Processes.txt] + * [Design.Indexes.txt] + * [Design.Indexes.MailIndexApi.txt] + * [Design.AuthProcess.txt] + * [Design.AuthProtocol.txt] + * [Design.MailProcess.txt] + * [Design.DoveadmProtocol.txt] and [Design.DoveadmProtocol.HTTP.txt] + * [Design.Dsync.txt] + * [Design.Lua.txt] + +Protocol extensions +------------------- + + * + [Design.ParameterForwarding.txt] + +Code APIs +--------- + + * [Design.Code.txt] - explanations how and why the coding style + is the way it is + +Look at the *.h files for the actual API documentation. The documentation below +doesn't attempt to list full API documentation. + +liblib: + + * [Design.Memory.txt] + * [Design.Buffers.txt] + * [Design.Arrays.txt] + * [Design.Strings.txt] + * [Design.InputStreams.txt] + * [Design.OutputStreams.txt] + * [Design.Events.txt] + * [Design.Plugins.txt] + +lib-dcrypt: + + * [Design.Dcrypt.txt] + +lib-storage: + + * [Design.Storage.MailUser.txt] contains everything related to a + single user. + * [Design.Storage.MailNamespace.txt]: A single user can + contain multiple [Namespaces.txt]. + * [Design.Storage.MailboxList.txt] is used to list/manage a + list of mailboxes for a single namespace (1:1 relationship). + * [Design.Storage.MailStorage.txt] is used to access mails in a + specific location with a specific mailbox format. Multiple namespaces can + point to the same storage. A single namespace may in future (but not + currently) point to multiple storages (e.g. a mixed mbox and Maildir + directory). + * [Design.Storage.Mailbox.txt] is used to access a specific mailbox + in a storage. + * [Design.Storage.Mail.txt] is used to access a specific mail in a + mailbox. + * [Design.Storage.ErrorHandling.txt]. + * [Design.Storage.Plugins.txt] - how to hook into lib-storage + functions. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Dict.txt b/doc/wiki/Dict.txt new file mode 100644 index 0000000000..082fb4ae0a --- /dev/null +++ b/doc/wiki/Dict.txt @@ -0,0 +1,45 @@ +Dict Proxy Process +================== + +Dict server is used for providing [Dictionary.txt] access via +server processes instead of doing it directly from whichever process wants to +access the dictionary. This is useful for some backends with relatively high +connection cost (e.g. SQL), but not necessarily for others (e.g. Redis). + +When a mail process uses the dict proxy, it needs to have access the dict UNIX +socket. By default only the "dovecot" user has access to the dict socket, which +doesn't typically work in any installation. However, giving too wide +permissions by default might allow untrusted users to access the dict and cause +problems. + +If all users share a single UNIX UID (e.g. "vmail"), you could make the dict +socket accessible only to it: + +---%<------------------------------------------------------------------------- +service dict { + unix_listener dict { + mode = 0600 + user = vmail + } +} +---%<------------------------------------------------------------------------- + +If you use multiple UNIX UIDs, you can add an extra group for all Dovecot mail +processes. This works even if you have untrusted system users who have shell +access to the server: + +---%<------------------------------------------------------------------------- +mail_access_groups = dovecot + +service dict { + unix_listener dict { + mode = 0660 + group = dovecot + } +} +---%<------------------------------------------------------------------------- + +However, it works with only if it's started as root. If this isn't +possible, look into using instead. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Dictionary.txt b/doc/wiki/Dictionary.txt new file mode 100644 index 0000000000..1eb078bfe5 --- /dev/null +++ b/doc/wiki/Dictionary.txt @@ -0,0 +1,268 @@ +Dovecot Dictionaries +==================== + +Dovecot's lib-dict can be used to access simple key-value databases. This is +used by for example [Quota.Dict.txt], +[AuthDatabase.Dict.txt], [Plugins.LastLogin.txt], + [ImapMetadata.txt], etc. The dictionaries can be accessed either +directly by the mail processes or they can be accessed via +[Dict.txt] processes. + +Currently supported dict backends are: + + * Flat files + * FS (lib-fs wrapper) + * Memcached (ASCII protocol) + * Memcached (Binary protocol) + * Redis + * Proxy + * SQL + * LDAP (read only) + +Flat Files +---------- + +---%<------------------------------------------------------------------------- +file: +---%<------------------------------------------------------------------------- + +The file will simply contain all the keys that are used. Not very efficient for +large databases, but good for small ones such as a single user's quota. + +FS (v2.2.11+) +------------- + +---%<------------------------------------------------------------------------- +fs:: +---%<------------------------------------------------------------------------- + +This is a wrapper for lib-fs, which most importantly has the "posix" backend. +So using: + +---%<------------------------------------------------------------------------- +fs:posix:prefix=/var/lib/dovecot/dict +---%<------------------------------------------------------------------------- + +Would create a separate file under /var/lib/dovecot/dict for each key. + +Memcached (Binary Protocol) (v2.2.9+) +------------------------------------- + +This driver uses the "new" Memcache binary protocol. + +https://code.google.com/p/memcached/wiki/MemcacheBinaryProtocol + +---%<------------------------------------------------------------------------- +memcached:param=value:param2=value2:... +---%<------------------------------------------------------------------------- + +Supported parameters are: + + * host: Memcached server host (default: 127.0.0.1) + * port: Memcached server port (default: 11211) + * prefix: Prefix to add to all keys (default: empty) + * timeout_msecs: Abort lookups after specified number of milliseconds + (default: 30000) + +Memcached (ASCII Protocol) (v2.2.9+) +------------------------------------ + +This driver uses the "legacy" Memcache ASCII protocol. + +https://github.com/memcached/memcached/blob/master/doc/protocol.txt + +---%<------------------------------------------------------------------------- +memcached_ascii:param=value:param2=value2:... +---%<------------------------------------------------------------------------- + +Supported parameters are: + + * host: Memcached server host (default: 127.0.0.1) + * port: Memcached server port (default: 11211) + * prefix: Prefix to add to all keys (default: empty) + * timeout_msecs: Abort lookups after specified number of milliseconds + (default: 30000) + +Redis (v2.2.9+) +--------------- + +---%<------------------------------------------------------------------------- +redis:param=value:param2=value2:... +---%<------------------------------------------------------------------------- + +Supported parameters are: + + * host: Redis server host (default: 127.0.0.1) + * port: Redis server port (default: 6379) + * prefix: Prefix to add to all keys (default: empty) + * db: Database number (default: 0) + * expire_secs=: Set expiration value to all the keys + * timeout_msecs: Abort lookups after specified number of milliseconds + (default: 30000) + +Proxy +----- + +---%<------------------------------------------------------------------------- +proxy:[]: +---%<------------------------------------------------------------------------- + +Proxying is used to perform all dictionary accessing via the dict processes. +(The dict processes exist only if dict proxying is used.) This is especially +useful with backends where their initialization is relatively expensive, such +as SQL. The dict processes will then perform also connection pooling. + +If is specified, it points to the socket where the dict server is +answering. The default is to use $base_dir/dict. Usually this is changed to +"dict-async" if the dict backend support asynchronous lookups (e.g. ldap, +pgsql, cassandra). The dict-async service allows more than one client, so this +configuration prevents creating unnecessarily many dict processes. + +The contains the dict name in the dict { .. } settings. For +example:'proxy:dict-async:quota' + +See for more information about the dict server. + +SQL +--- + +---%<------------------------------------------------------------------------- +: +---%<------------------------------------------------------------------------- + +The contains the SQL driver name, such as "mysql", "pgsql", +"sqlite" or "cassandra". + +The dict-sql config file consists of SQL server configuration and mapping of +keys to SQL tables/fields. + +SQL Connect String +------------------ + +---%<------------------------------------------------------------------------- +connect = host=localhost dbname=mails user=sqluser password=sqlpass +---%<------------------------------------------------------------------------- + +The connect setting is exactly the same as used for +[AuthDatabase.SQL.txt]. See 'example-config/dovecot-sql.conf.ext' for detailed +information. + +SQL Mapping +----------- + +SQL mapping is done with a dict key pattern and fields. When a dict lookup or +update is done, Dovecot goes through all the maps and uses the first one whose +pattern matches the dict key. + +For example when using dict for a per-user quota value the map looks like: + +---%<------------------------------------------------------------------------- +map { + pattern = priv/quota/storage + table = quota + username_field = username + value_field = quota_bytes +} +---%<------------------------------------------------------------------------- + +This means that: + + * The dict key must match exactly "priv/quota/storage". The dict keys are + hardcoded in the Dovecot code, so depending on what functionality you're + configuring you need to know the available dict keys used it. + * This is a private dict key ("priv/" prefix), which means that there must be + a username_field. The username_field is assumed to be (at least part of) the + primary key. In this example we don't have any other primary keys. + * With MySQL the above map translates to SQL queries: + * 'SELECT quota_bytes FROM quota WHERE username = '$username_field'' + * 'INSERT INTO quota (username, quota_bytes) VALUES ('$username_field', + '$value') ON DUPLICATE KEY UPDATE quota_bytes='$value'' + +You can also access multiple SQL fields. For example acl_shared_dict can +contain: + +---%<------------------------------------------------------------------------- +map { + pattern = shared/shared-boxes/user/$to/$from + table = user_shares + value_field = dummy + + fields { + from_user = $from + to_user = $to + } +} +---%<------------------------------------------------------------------------- + + * The acl_shared_dict always uses "1" as the value, so here the value_field is + called "dummy". + * The SQL from_user and to_user fields are the interesting ones. Typically the + extra fields would be part of the primary key. + * With MySQL the above map translates to SQL queries: + * 'SELECT dummy FROM user_shares WHERE from_user = '$from' AND to_user = + '$to'' + * 'INSERT INTO user_shares (from_user, to_user, dummy) VALUES ('$from', + '$to', '$value') ON DUPLICATE KEY UPDATE dummy='$value'' + +LDAP (v2.2.24+) +--------------- + +LDAP support is very similar to SQL support, but there is no write support. + +Configuration +------------- + +---%<------------------------------------------------------------------------- +dict { + somedict = ldap:/path/to/dovecot-ldap-dict.conf.ext +} +---%<------------------------------------------------------------------------- + +Then in ext file put + +---%<------------------------------------------------------------------------- +uri = ldap://hostname +bind_dn = optional bind dn +password = optional password +timeout = optional timeout +debug = 0 or 1 (optional, as well) +tls = yes|try|no (default is try) +---%<------------------------------------------------------------------------- + + * uri - LDAP connection URI as expected by OpenLDAP. + * bind_dn - DN or upn to use for binding + * password - password to use, only SIMPLE auth is supported at the moment + * timeout - How long to wait for reply, default is 30 seconds + * debug - 0 off, 1 on, will produce metric ton of output + * tls - yes = require either ldaps or successful start TLS, try = send start + TLS if necessary, no = do not send start TLS + +To map some key to a search do + +---%<------------------------------------------------------------------------- +map { + pattern = priv/test/mail + filter = (mail=*) # the () is required + base_dn = ou=container,dc=domain + username_attribute = uid # default is cn + value_attribute = mail +} +---%<------------------------------------------------------------------------- + +To do some more complex search + +---%<------------------------------------------------------------------------- +map { + pattern = priv/test/mail/$location + filter = (&(mail=*)(location=%{location}) # the () is required + base_dn = ou=container,dc=domain + username_attribute = uid # default is cn + value_attribute = mail + + fields { + location=$location + } +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Director.txt b/doc/wiki/Director.txt new file mode 100644 index 0000000000..1ffa3641a1 --- /dev/null +++ b/doc/wiki/Director.txt @@ -0,0 +1,366 @@ +Director +======== + +Director can be used by +[PasswordDatabase.ExtraFields.Proxy.txt] to keep a temporary user -> mail +server mapping. As long as user has simultaneous connections, the user is +always redirected to the same server. Each proxy server is running its own +director process, and the directors are communicating the state to each others. +Directors are mainly useful for setups where all of the mail storage is seen by +all servers, such as with NFS or a cluster filesystem. + +First test non-director proxying +-------------------------------- + +The director is simply a small add-on for Dovecot proxy. Before configuring +director, you should test that a simple proxying setup with static destination +server works. See the [PasswordDatabase.ExtraFields.Proxy.txt] page for +more information about how to configure it. If you have a simple setup, you can +test this easily using a static passdb: + +---%<------------------------------------------------------------------------- +passdb { + driver = static + args = proxy=y host=10.2.0.20 nopassword=y +} +---%<------------------------------------------------------------------------- + +Once finished testing, remember to remove the "host" field. + +Servers +------- + +You need one or more servers assigned for Dovecot proxies. The same servers +could also act as backends handling the mails, but you need to run [RunningDovecot.txt] in different ports. This +may get a bit confusing, so it's not recommended (although v2.1 makes it easier +with 'instance_name' setting). + +The directors are going to connect to each others in a ring. For example if you +have servers called A, B and C, director will create connections A->B, B->C and +C->A. + +Director configuration +---------------------- + +In example configuration you can configure director from +'conf.d/10-director.conf'. + +Listeners +--------- + +Configure the listeners that director requires: + +---%<------------------------------------------------------------------------- +service director { + unix_listener login/director { + mode = 0666 + } + fifo_listener login/proxy-notify { + mode = 0600 + user = $default_login_user + } + # NOTE: director-userdb socket is actually used only for passdb lookups, not +userdb lookups + unix_listener director-userdb { + mode = 0600 + } + inet_listener { + port = 9090 + } + ## uncomment this if you want to use + ## doveadm director status -a ip:9091 + #inet_listener director-doveadm { + # port = 9091 + #} +} +---%<------------------------------------------------------------------------- + +The port 9090 will be used for listening and connecting to other directors. +You're free to use any port number you want. + +Configuring list of director servers +------------------------------------ + +List all of your directors in 'director_servers' setting separated by spaces. +You can use: + + * IP addresses + * hostnames + * hostnames that expand to multiple IPs (e.g. you could have a "directors-all" + DNS entry that expands to all directors' IPs) + +You can also add :port after the IP/host. The default port is the same as what +director service's inet_listener is using (the port 9090 above). + +Note that the same director must not be listed multiple times with different +IPs. This especially means that a hostname can't expand to both IPv4 and IPv6 +address. Otherwise Dovecot becomes confused about what directors actually +exist. This also means that a single director ring must use either IPv4 or IPv6 +addresses, but not both at the same time. + +For example if you have 3 directors, you could set: + +---%<------------------------------------------------------------------------- +director_servers = 10.1.0.2 10.1.0.3 10.1.0.4 +---%<------------------------------------------------------------------------- + +Configuring list of mail servers +-------------------------------- + +List all of your backend mail servers in 'director_mail_servers' setting +separated by spaces. You can use: + + * IP addresses + * IP ranges (e.g. 10.2.0.10-10.2.0.30) + * hostnames + * hostnames that expand to multiple IPs + +For example if you had 20 mail servers with consecutive IPs: + +---%<------------------------------------------------------------------------- +director_mail_servers = 10.2.0.11-10.2.0.30 +---%<------------------------------------------------------------------------- + +Enabling director +----------------- + +Enable director for the wanted login services by telling them to connect to +director socket instead of the default login socket: + +---%<------------------------------------------------------------------------- +service imap-login { + executable = imap-login director +} +service pop3-login { + executable = pop3-login director +} +---%<------------------------------------------------------------------------- + +Consistent hashing should be enabled for new director clusters. It's not +possible to change this setting without a complete director cluster shutdown. +Using it reduces users being moved around when doing backend changes. + +---%<------------------------------------------------------------------------- +director_consistent_hashing = yes # Supported by v2.2.16+ +---%<------------------------------------------------------------------------- + +If you want to enable director for LMTP, also set: + +---%<------------------------------------------------------------------------- +# LMTP first does a passdb lookup to to see if there's a proxy field returned. +# If not, it fallbacks to doing userdb lookup. +lmtp_proxy = yes + +protocol lmtp { + # NOTE: director-userdb socket is actually used only for passdb lookups, not +userdb lookups + auth_socket_path = director-userdb +} + +# If you want lmtp-proxy listening on the network, uncomment the following: +#service lmtp { +# inet_listener lmtp { +# port = 24 +# } +#} +---%<------------------------------------------------------------------------- + +By default LMTP proxy connects to the same port in backend as what was used for +the incoming connection. + +Other settings +-------------- + +Directors redirect a user to the same server always the user has active +connections. The redirection is also done for a while after the last connection +already disconnected. This is mainly to avoid trouble with NFS caches that +haven't yet expired. You can configure this setting from: + +---%<------------------------------------------------------------------------- +director_user_expire = 15 min +---%<------------------------------------------------------------------------- + +'doveadm director kick' and 'doveadm director move' need to be able to connect +to the 'ipc' socket. Make sure the director process can do it: + +---%<------------------------------------------------------------------------- +service ipc { + unix_listener ipc { + user = dovecot # This is already the default in v2.3.1+ + } +} +---%<------------------------------------------------------------------------- + +Passdb configuration +-------------------- + +Your passdb must return "proxy" +[PasswordDatabase.ExtraFields.txt], otherwise director doesn't do anything. + +Director works by adding a "host" extra field to the auth reply, which contains +the temporary destination mail server. This "host" field isn't added if the +passdb lookup already returns "host". This allows configuring some users to be +always proxied to a specific server. + +If the backend servers verify password, you can use static passdb for director: + +---%<------------------------------------------------------------------------- +passdb { + driver = static + args = proxy=y nopassword=y +} +---%<------------------------------------------------------------------------- + +Note that while this is the simplest director configuration, users will be +assigned to a backend before they have been authenticated. A director +configured this way can be attacked by sending it a large number of unknown +users. To prevent this, the director should be configured to authenticate the +user and might make use of a master password to log into the backend servers. + +Doveadm server +-------------- + +Use these settings for both director and backends: + +---%<------------------------------------------------------------------------- +service doveadm { + inet_listener { + # any port you want to use for this: + port = 24245 + } +} + +local 10.10.10.0/24 { + # password to use for client authentication + doveadm_password = secret + # allow client to only use specified list of commands (default is all): + #doveadm_allowed_commands = +} +---%<------------------------------------------------------------------------- + +The director also needs the following configuration: + +---%<------------------------------------------------------------------------- +# same port as doveadm's inet_listener +doveadm_proxy_port = 24245 + +protocol doveadm { + # NOTE: director-userdb socket is actually used only for passdb lookups, not +userdb lookups + auth_socket_path = director-userdb +} +---%<------------------------------------------------------------------------- + +Now you can run doveadm commands on the director, and it'll run them +automatically on the correct backend server. + +Health monitoring of backend servers +------------------------------------ + +Brad Davidson has written a small daemon for monitoring backend servers, and +disable/enable them on demand. +Ref:https://dovecot.org/list/dovecot/2010-August/051946.html + +Forcefully moving users to a different backend +---------------------------------------------- + +This is useful if you need to do maintenance on one of the backend servers and +want (active) clients to move to a different backend: + + 1. Disable any watchdog system that will undo changes you make to backend + server weights, such as poolmon. + * Not needed if the watchdog is new enough to use HOST-UP/HOST-DOWN + commands rather than change weights. + 2. Set the weight of the backend server to be worked on to 0: 'doveadm + director add 0' + 3. Flush current assignments to disable new connections to this backend: + 'doveadm director flush ' + * This will also kick the existing connections to the backend in v2.2.19+. + +Most IMAP clients will silently just reconnect to the (new backend) server +after being kicked (at least Apple Mail 6.0 and Thunderbird 14.0). + +For moving specific users to other servers (e.g. because there are too many +"heavy users" assigned to the same backend), you can use 'doveadm director +move' command in v2.0.14+. This requires the ipc permissions to be configured +correctly (see above). + +Tags +---- + +(Requires v2.2.16+) + +*WARNING*: This feature isn't working perfectly in v2.2.26.1 and older. If two +users with different tags have the same 32bit hash, they may end up going to +the wrong tag's backend. + +With tags you can use a single director ring to serve multiple backend +clusters. Each backend cluster is assigned a tag name, which can be anything +you want. By default everything has an empty tag. A passdb lookup can return +"director_tag" field containing the wanted tag name. If there aren't any +backend servers with the wanted tag, it's treated the same as if there aren't +any backend servers available (= wait for 30 secs for a backend and then return +temporary failure). + +Tags can be added to configuration by adding @tag suffix to IPs/hosts. For +example: + +---%<------------------------------------------------------------------------- +director_mail_servers = 10.0.0.100-10.0.0.110@name1 10.0.0.120@name2 +---%<------------------------------------------------------------------------- + +"doveadm director add" can also add tags either with @tag suffix or with -t +parameter. "doveadm director status user@domain" requires giving the user's +correct tag with -t parameter or the results won't be correct (empty tag's +results are shown). Tags can't currently be changed for an existing host +without removing it first. + +Director and Backend in same server (broken) +-------------------------------------------- + +NOTE: This feature never actually worked. It would require further development +to fix (director would need to add "proxy" field to extra fields and notify +auth that the auth_request can be freed). + +Have the passdb lookup return 'director_proxy_maybe=y'. LMTP however doesn't +currently support mixing recipients to both being proxied and store locally. + +Flush socket +------------ + +(Requires v2.2.26+) + +This allows calling a script for each user (hash) that is moved between +backends. This is triggered by "doveadm director move" and "doveadm director +flush" commands. What happens is: + + * User's connections are kicked from the director cluster + * Flush socket is called and waited on. + * User logins are delayed until the flush socket is finished, or the user move + times out after 30 seconds (hardcoded). + * Only the director that initiated the doveadm command will call the flush + socket. + * director_user_kick_delay is ignored by the initiating director, but used + by the other directors. + +Configuration: + +---%<------------------------------------------------------------------------- +director_flush_script = user-flush +service user-flush { + executable = script /usr/local/bin/user-flush.sh + unix_listener user-flush { + user = dovecot + } +} +---%<------------------------------------------------------------------------- + +The user-flush.sh will receive as parameters: + + * "FLUSH" + * username hash + * old host's IP + * new host's IP + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/DomainLost.txt b/doc/wiki/DomainLost.txt new file mode 100644 index 0000000000..d7d24a579a --- /dev/null +++ b/doc/wiki/DomainLost.txt @@ -0,0 +1,60 @@ +Domain (%d) is empty +==================== + +IMAP or POP3 protocol doesn't have explicit support for domains. The usernames +are commonly in user@domain format, and that is also where Dovecot gets the +domain from. If the username doesn't have @domain, then the domain is also +usually empty (unless 'auth_default_realm' setting is used). + +If you login as user@domain, but the %d is still empty, the problem is that +your configuration lost the domain part by changing the username. Dovecot +doesn't keep track of the domain separately from username, so if something +changes username from "user@domain" to just plain "user", the domain is lost +and %d returns nothing. If you have 'auth_debug=yes', this shows up in logs +like: + +---%<------------------------------------------------------------------------- +Info: auth(user@domain.org): username changed user@domain.org -> user +---%<------------------------------------------------------------------------- + +Below are some of the most common reasons for this. + +Settings +-------- + +'auth_username_format = %Ln' lowercases the username but also drops the domain. +Use 'auth_username_format = %Lu' instead. + +'auth_username_format' changes the username permanently, currently it's not +possible to make it affect only the authentication part. + +SQL +--- + +'password_query' gets often misconfigured to drop the domain if username and +domain are stored separately. For example: + +---%<------------------------------------------------------------------------- +# BROKEN: +password_query = SELECT username AS user, password FROM users WHERE username = +'%n' AND domain = '%d' +---%<------------------------------------------------------------------------- + +The "username AS user" changes the username permanently and the domain is +dropped. You can instead use: + +---%<------------------------------------------------------------------------- +# MySQL: +password_query = SELECT concat(username, '@', domain) AS user, password FROM +users WHERE username = '%n' AND domain = '%d' +---%<------------------------------------------------------------------------- + +Or you can return username and domain fields separately and Dovecot will merge +them into a single user field: + +---%<------------------------------------------------------------------------- +password_query = SELECT username, domain, password FROM users WHERE username = +'%n' AND domain = '%d' +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Errors.ChgrpNoPerm.txt b/doc/wiki/Errors.ChgrpNoPerm.txt new file mode 100644 index 0000000000..721c672336 --- /dev/null +++ b/doc/wiki/Errors.ChgrpNoPerm.txt @@ -0,0 +1,28 @@ +Operation Not Permitted +======================= + +---%<------------------------------------------------------------------------- +imap(user): Error: chown(/home/user/mail/.imap/INBOX, group=12(mail)) failed: +Operation not permitted (egid=1000(user), group based on /var/mail/user - see +http://wiki2.dovecot.org/Errors/ChgrpNoPerm) +---%<------------------------------------------------------------------------- + +This means that Dovecot tried to copy '/var/mail/user' file's group (mail) to +the index file directory it was creating ('/home/user/mail/.imap/INBOX'), but +the process didn't belong to the mail group, so it failed. This is important +for preserving access permissions with +[SharedMailboxes.txt]. Group copying is done only when it actually changes the +access permissions; for example with 0600 or 0666 mode the group doesn't matter +at all, but with 0660 or 0640 it does. + +To solve this problem you can do only one of two things: + + 1. If the group doesn't actually matter, change the permissions so that the + group isn't copied (e.g.'chmod 0600 /var/mail/*', see + ) + 2. Give the mail process access to the group (e.g. 'mail_access_groups=mail' + setting). However, this is dangerous.It allows users with shell access to + read other users' INBOXes + [http://dovecot.org/list/dovecot-news/2008-March/000060.html]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Events.txt b/doc/wiki/Events.txt new file mode 100644 index 0000000000..39e0a5a623 --- /dev/null +++ b/doc/wiki/Events.txt @@ -0,0 +1,288 @@ +Events +====== + +List of all events that can be used in and elsewhere. Note +that in v2.3 these are added on version basis, so not all events are available +in all v2.3 releases. + +Contents + + + 1. Events + + 1. Authentication Server (v2.3.6) + + 2. Authentication Client (v2.3.6) + + 3. Connection + + 4. HTTP + + 5. IMAP + + 6. DNS + + 7. SQL + +Authentication Server (v2.3.6) +------------------------------ + +These events are generated by authentication in auth process(es). + +*auth_request_finished* + + * user: full username + * original_username: original username used + * translated_username: username after translations + * login_user: when doing login using master_user, this is the user we are + logging in as + * master_user: master username + * error: set when error happens + * successful: yes, when succeeded + * transport: insecure, trusted, TLS + * mechanism: name of used mechanism + * credentials_scheme: type of credential (SHA256-CRYPT, PLAIN, ...) + * policy_penalty: time of penalty added by policy server + * policy_result: ok, delayed, refused + +*auth_passdb_request_started* + + * passdb_name: name of passdb + * passdb: driver + +*auth_passdb_request_finished* + + * passdb_name: name of passdb + * passdb: driver + * user: full username + * master_user: master user name + * username: username without domain + * domain: domain if present + * result: authentication result + +*auth_userdb_request_started* + + * userdb_name: name of passdb + * userdb: driver + +*auth_userdb_request_finished* + + * userdb_name: name of passdb + * userdb: driver + * user: full username + * master_user: master user name + * username: username without domain + * domain: domain if present + * result: authentication result + +*auth_policy_request_finished* + + * mode: allow, report + * policy_response: number + +Authentication Client (v2.3.6) +------------------------------ + +These events are generated by authentication clients (lib-auth). + +*auth_client_request_started* + + * id: event id + +*auth_client_request_continue* + + * id: event id + +*auth_client_request_finished* + + * id: event id + * error: reason + +*auth_client_request_challenged* + + * id: event id + +*auth_client_userdb_lookup_started* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + +*auth_client_userdb_lookup_finished* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + * error: error if occured + +*auth_client_passdb_lookup_started* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + +*auth_client_passdb_lookup_finished* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + * error: error if occured + +*auth_client_userdb_list_started* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + +*auth_client_userdb_list_finished* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + * error: error if occured + +*auth_client_cache_flush_started* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + +*auth_client_cache_flush_finished* + + * service: name of service (smtp, imap, lmtp...) + * local_ip: local ip + * local_port: local port + * remote_ip: remote ip + * remote_port: remote port + * user: full username + * error: error if occured + +Connection +---------- + +These events apply only for connections using the "connection API". +Unfortunately not all connections currently use this, so these events work for +some types of connections, but not for others. + +*client_connection_connected* + + * client_ip: source IP + * client_port: source port + * ip: target IP + * port: target port + +*client_connection_disconnected* + + * fields from *client_connection_connected* + * bytes_in: amount of data read + * bytes_out: amount of data written + * reason: disconnection reason + +*server_connection_connected* + + * client_ip: source IP + * client_port: source port + * ip: target IP + * port: target port + * bytes_in: amount of data read + * bytes_out: amount of data written + +*server_connection_disconnected* + + * fields from *server_connection_connected* + * bytes_in: amount of data read + * bytes_out: amount of data written + * reason: disconnection reason + +HTTP +---- + +*http_request_finished* + + * status_code: HTTP result status code + * attempts: Number of individual HTTP request attempts that were done (i.e. + number of retries after failures + 1) + * redirects: Number of redirects that were done while processing this request. + + * bytes_in, bytes_out: Number of bytes received/sent for this request. + +*http_request_redirected* + + * Intermediate event sent while HTTP request is being redirected. The + *http_request_finished* is still being sent as well. + * Same fields as *http_request_finished* + +*http_request_retried* + + * Intermediate event sent while HTTP request is being retried. The + *http_request_finished* is still being sent as well. + * Same fields as *http_request_finished* + +IMAP +---- + +*imap_command_finished* + +NOTE: These are currently not sent for pre-login IMAP commands. + + * tag: Command tag + * name: Command name + * args: Command's full parameters + * human_args: Command's parameters as more human-readable output + * tagged_reply_state: OK, NO, BAD + * tagged_reply: Full tagged reply, e.g. "OK SELECT finished." + * last_run_time: Timestamp when the command was running last time (it's + followed by "mailbox sync" that can take some time) + * running_usecs: How many usecs this command itself has spent running + * lock_wait_usecs: How many usecs this command itself has spent waiting for + locks. + * bytes_in, bytes_out: How many bytes of client input/output command has used. + +DNS +--- + +*dns_worker_request_finished* and *dns_request_finished* + + * error: Human readable error + * error_code: Error code usable with net_gethosterror() + +*dns_worker_request_started* and *dns_request_started* + +SQL +--- + +*sql_query_finished* + + * error: Human readable error + * error_code: Error code (if available) + * query_first_word: First word of the query, such as SELECT + +*sql_transaction_finished* + + * error: Human readable error + * error_code: Error code (if available) + +*sql_connection_finished* + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/FindMailLocation.txt b/doc/wiki/FindMailLocation.txt new file mode 100644 index 0000000000..ed8d62c024 --- /dev/null +++ b/doc/wiki/FindMailLocation.txt @@ -0,0 +1,65 @@ +Finding Your Mail +================= + +Before configuring Dovecot, you'll need to know where your mails are located. +You should already have an SMTP server installed and configured to deliver +mails somewhere, so the easiest way to make Dovecot work is to just use the +same location. Otherwise you could create '~/Maildir' directory and configure +your SMTP server to use the Maildir format. + +First send a test mail to yourself (as your own non-root user): + +---%<------------------------------------------------------------------------- +echo "Hello me" | mail -s "Dovecot test" $USER +---%<------------------------------------------------------------------------- + +Now, find where the mail went. Here's a simple script which checks the most +common locations: + +---%<------------------------------------------------------------------------- +for mbox in /var/mail/$USER /var/spool/mail/$USER ~/mbox ~/mail/* ~/*; do + grep -q "Dovecot test" "$mbox" && echo "mbox: $mbox" +done +grep -q "Dovecot test" ~/Maildir/new/* 2>/dev/null && echo "Maildir: ~/Maildir" +---%<------------------------------------------------------------------------- + +mbox +---- + +In most installations your mail went to '/var/mail/username' file. This file is +called *INBOX* in IMAP world. Since IMAP supports multiple mailboxes, you'll +also have to have a directory for them as well. Usually '~/mail' is a good +choice for this. For installation such as this, the mail location is specified +with (typically in 'conf.d/10-mail.conf'): + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=/var/mail/%u +---%<------------------------------------------------------------------------- + +Where '%u' is replaced with the username that logs in. Similarly if your INBOX +is in '~/mbox', use: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=~/mbox +---%<------------------------------------------------------------------------- + +Maildir +------- + +Maildir exists almost always in '~/Maildir' directory. The mail location is +specified with (typically in 'conf.d/10-mail.conf'): + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir +---%<------------------------------------------------------------------------- + +Problems? +--------- + +If you can't find the mail, you should check your SMTP server logs and +configuration to see where it went or what went wrong. + +If you can find the mail, but it's in more exotic location, see if + can help you to configure it. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/FinishBasicConfiguration.txt b/doc/wiki/FinishBasicConfiguration.txt new file mode 100644 index 0000000000..42a63ad0f9 --- /dev/null +++ b/doc/wiki/FinishBasicConfiguration.txt @@ -0,0 +1,12 @@ +Finishing Basic Configuration +============================= + +Unless you're going to have only virtual users and you don't care about their +passwords,*switch back to disable_plaintext_only = yes* and +[SSL.txt]. + +Plaintext authentication is still allowed from localhost, so you can have your +webmail application to connect to Dovecot without using SSL or even having to +configure it. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HAProxy.txt b/doc/wiki/HAProxy.txt new file mode 100644 index 0000000000..e7194c7916 --- /dev/null +++ b/doc/wiki/HAProxy.txt @@ -0,0 +1,90 @@ +HAProxy Support +=============== + +For high availability, client connections can be spread across multiple backend +servers using HAProxy [http://www.haproxy.org/]. This solution is often +employed for large Dovecot installations as a replacement for a hardware load +balancer. A common problem with such a reverse proxy +[http://en.wikipedia.org/wiki/Reverse_proxy] is that Dovecot does not talk to +the client directly. This means that the connection information (IP:port) that +Dovecot uses for logging and authentication purposes points to the proxy, +rather than the real client. To solve this, the proxy needs to convey the +connection information to the backend somehow, so that that information is +available there. The developers of HAProxy defined a custom Proxy Protocol +[http://blog.haproxy.com/haproxy/proxy-protocol/] for this purpose. By this +protocol, the proxy sends the connection information immediately after +connection setup in a special initial header. Note that this isn't normally +needed after the initial Dovecot proxies, because Dovecot internally uses +IMAP/POP3/LMTP extensions to forward the original IP address. Dovecot supports +both versions of the Proxy Protocol since Dovecot version 2.2.19. + +Dovecot Configuration +--------------------- + +The following global settings relate to HAProxy: + +haproxy_trusted_networks = : + A space-separated list of trusted network ranges for HAProxy connections. + Connections from networks outside these ranges to ports that are configured + for HAProxy are aborted immediately. + +haproxy_timeout = 3 : + The time in seconds after which a HAPRoxy connection is aborted when no + complete header is received. + +The HAPRoxy protocol can be enabled for specific +[Services.txt]. This way, a service such as IMAP or POP3 can accept both normal +and HAProxy connections. A TCP listener is configured for HAProxy by setting +'haproxy=yes' for that listener. If 'haproxy=yes' is set for a listener, its +use is mandatory on that port; i.e., if the client is not a proper proxy (its +omits the PROXY header), the connection will be aborted. + +For example, to enable normal IMAP connections on port 143, SSL connections on +port 993 and HAProxy connections on port 10143, the 'imap-login' service is +configured as follows: + +---%<------------------------------------------------------------------------- +service imap-login { + inet_listener imap { + port = 143 + } + inet_listener imaps { + port = 993 + ssl = yes + } + inet_listener imap_haproxy { + port = 10143 + haproxy = yes + } +} +---%<------------------------------------------------------------------------- + +HAProxy Configuration +--------------------- + +The documentation of this feature on the HAProxy side is a bit fragmented +between the HAProxy Configuration Manual +[http://www.haproxy.org/download/1.6/doc/configuration.txt] and the Proxy +Protocol documentation +[http://www.haproxy.org/download/1.6/doc/proxy-protocol.txt]. In summary, it is +enabled by including the 'send-proxy' setting in the 'server' lines. It is +therefore only enabled on a per-server basis. + +For example, the 'frontend' and 'backend' configuration of HAProxy could look +as follows: + +---%<------------------------------------------------------------------------- +frontend ft_imap + bind :143 + mode tcp + default_backend bk_imap + +backend bk_imap + mode tcp + balance leastconn + stick store-request src + stick-table type ip size 200k expire 30m + server s1 backend.example.com:10143 send-proxy-v2 +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.AntispamWithSieve.txt b/doc/wiki/HowTo.AntispamWithSieve.txt new file mode 100644 index 0000000000..e8aea0694e --- /dev/null +++ b/doc/wiki/HowTo.AntispamWithSieve.txt @@ -0,0 +1,270 @@ +Replacing antispam plugin with IMAPSieve +======================================== + +Contents + + + 1. Replacing antispam plugin with IMAPSieve + + 1. Caveats and possible pitfalls + + 2. Dovecot configuration + + 3. Sieve scripts + + 4. Shell scripts + + 1. For spamassassin + + 2. For dspam + + 3. For rspamd + + 5. Debugging + + 6. RoundCube + +You will need at least pigeonhole v0.4.14 for this. If you have already +configured sieve, please adjust the following to match your setup. + +Caveats and possible pitfalls +----------------------------- + + * INBOX name is case-sensitive + * [Pigeonhole.Sieve.Plugins.IMAPSieve.txt] will *only* apply to + IMAP. It *will not* apply to LDA or LMTP. Use [Pigeonhole.Sieve.txt] + normally for LDA/LMTP. + * With this configuration, moving mails will slow down due to learn being done + per email. If you want to avoid this, you need to think of something else. + Probably piping things into a FIFO or perhaps using a socket based worker + might work better. + * Please read and + to understand sieve configuration better. + * Please read for more information about sieve + extensions. + * If you run Spamassassin trough Amavis and you use a virtual users setup, you + should instead configure Spamassassin to use MySQL/PostgreSQL as a backend, + unless you want a headache with file permissions and lock files. You can + find instructions here + [http://www.iredmail.org/docs/store.spamassassin.bayes.in.sql.html]. In this + case, the '-u' parameter passed to 'sa-learn' (and the relevant sieve + variables) is obsolete and can be safely removed. + * Reloading dovecot doesn't activate changes in this configuration, you'll + need to perform a full restart. + +Changes: + + * 2017/11/20 - Possibility of using spamc with to mitigate + multi-message delays + * 2017/05/05 - Recommendation about Virtual Users and using an SQL Backend. + Added brief info about . + * 2017/04/01 - Pass imap user to scripts. + * 2017/03/19 - Added rspamd scripts and mention about sieve plugins. + * 2017/02/13 - Improved documentation and added instructions for Spam->Trash. + (Thanks for everyone who commented on mailing list) + * 2017/02/10 - Removed imap_stats (it's not needed). + * 2018/04/11 - Added notes about sa-learn/spamc and warning about sieve script + location. + +Dovecot configuration +--------------------- + +---%<------------------------------------------------------------------------- +protocol imap { + mail_plugins = $mail_plugins imap_sieve +} + +plugin { + sieve_plugins = sieve_imapsieve sieve_extprograms + + # From elsewhere to Spam folder + imapsieve_mailbox1_name = Spam + imapsieve_mailbox1_causes = COPY + imapsieve_mailbox1_before = file:/usr/lib/dovecot/sieve/report-spam.sieve + + # From Spam folder to elsewhere + imapsieve_mailbox2_name = * + imapsieve_mailbox2_from = Spam + imapsieve_mailbox2_causes = COPY + imapsieve_mailbox2_before = file:/usr/lib/dovecot/sieve/report-ham.sieve + + sieve_pipe_bin_dir = /usr/lib/dovecot/sieve + + sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment +} +---%<------------------------------------------------------------------------- + +Sieve scripts +------------- + +*You cannot run scripts anywhere you want* + +Sieve allows you to only run scripts under sieve_pipe_bin_dir. You can't use +/usr/local/bin/my-sieve-filter.sh, you have to put the script under +sieve_pipe_bin_dir and use my-sieve-filter.sh instead. + +Create directory /usr/lib/dovecot/sieve and put following files to that: + +report-spam.sieve + +---%<------------------------------------------------------------------------- +require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; + +if environment :matches "imap.user" "*" { + set "username" "${1}"; +} + +pipe :copy "sa-learn-spam.sh" [ "${username}" ]; +---%<------------------------------------------------------------------------- + +report-ham.sieve + +---%<------------------------------------------------------------------------- +require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", "variables"]; + +if environment :matches "imap.mailbox" "*" { + set "mailbox" "${1}"; +} + +if string "${mailbox}" "Trash" { + stop; +} + +if environment :matches "imap.user" "*" { + set "username" "${1}"; +} + +pipe :copy "sa-learn-ham.sh" [ "${username}" ]; +---%<------------------------------------------------------------------------- + +Shell scripts +------------- + +For spamassassin +---------------- + +*Untested* + +spamc interaction scripts are not tested yet. + +sa-learn-spam.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +# you can also use tcp/ip here, consult spamc(1) 2 +exec /usr/bin/spamc -u ${1} -L spam -C report 3 +---CodeArea------------------------------------------------------------------- + +sa-learn-ham.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +# you can also use tcp/ip here, consult spamc(1) 2 +exec /usr/bin/spamc -u ${1} -L ham -C report 3 +---CodeArea------------------------------------------------------------------- + +You can also use sa-learn. + +Note that using sa-learn often incurs significant start-up time for every +message. This can cause "lockout" of the user until all the processes +sequentially complete, potentially tens of seconds or minutes. If spamd is +being used and the administrator is willing to accept the potential security +issues of allowing unauthenticated learning of spam/ham, spamd can be envoked +with the --allow-tell option and spamc with the --learntype= option. Please +consult the man pages of spamd and spamc for further details. + +sa-learn-spam.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/sa-learn -u ${1} --spam 2 +---CodeArea------------------------------------------------------------------- + +sa-learn-ham.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/sa-learn -u ${1} --ham 2 +---CodeArea------------------------------------------------------------------- + +For dspam +--------- + +sa-learn-spam.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/dspam --client --user ${1} --class=spam --source=error 2 +---CodeArea------------------------------------------------------------------- + +sa-learn-ham.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/dspam --client --user ${1} --class=innocent --source=error 2 +---CodeArea------------------------------------------------------------------- + +*CRLF handling* + +dspam may fail to read the mail if it contains CRLF line endings, add the +*Broken lineStripping* option in dspam.conf if needed. + +For rspamd +---------- + +By default, rspamd does global learning. If you want per-user classification, +or something more complex, see +https://rspamd.com/doc/configuration/statistic.html + +Alternative scripts can be found from +https://github.com/darix/dovecot-sieve-antispam-rspamd/ + +sa-learn-spam.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/rspamc -h /run/rspamd/worker-controller.socket -P +learn_spam 2 +---CodeArea------------------------------------------------------------------- + +sa-learn-ham.sh + +---CodeArea------------------------------------------------------------------- +#!/bin/sh 1 +exec /usr/bin/rspamc -h /run/rspamd/worker-controller.socket -P +learn_ham 2 +---CodeArea------------------------------------------------------------------- + +Before running following commands, make sure dovecot.conf has all the sieve +configuration you want. Then run following commands: + +---%<------------------------------------------------------------------------- +sievec /usr/lib/dovecot/sieve/report-spam.sieve +sievec /usr/lib/dovecot/sieve/report-ham.sieve +chmod +x /usr/lib/dovecot/sieve/sa-learn-ham.sh +/usr/lib/dovecot/sieve/sa-learn-spam.sh +---%<------------------------------------------------------------------------- + +Now your learn scripts should be invoked when you move mails between folders. + +Debugging +--------- + +To debug, you need to import "vnd.dovecot.debug" extension. Then you can put, +when required + +---%<------------------------------------------------------------------------- +debug_log "something" +---%<------------------------------------------------------------------------- + +variables are supported in this. + +RoundCube +--------- + +Recent versions of RoundCube [https://roundcube.net/] include a markasjunk2 +plugin [https://plugins.roundcube.net/packages/johndoh/markasjunk2] for +allowing users to mark Spam/Ham in a convenient way. Please make sure the +Junk/Spam folder matches your configuration. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.EximAndDovecotSASL.txt b/doc/wiki/HowTo.EximAndDovecotSASL.txt new file mode 100644 index 0000000000..99d178ce5e --- /dev/null +++ b/doc/wiki/HowTo.EximAndDovecotSASL.txt @@ -0,0 +1,54 @@ +Exim and Dovecot SASL +===================== + +Exim v4.64+ users can use Dovecot SASL instead of Cyrus SASL for authenticating +SMTP clients. + +conf.d/10-master.conf +--------------------- + +---%<------------------------------------------------------------------------- +service auth { +... +#SASL + unix_listener auth-client { + mode = 0660 + user = mail + } +... +} +---%<------------------------------------------------------------------------- + +conf.d/10-auth.conf +------------------- + +---%<------------------------------------------------------------------------- +auth_mechanisms = plain login +---%<------------------------------------------------------------------------- + +exim.conf +--------- + +Create authenticators for Dovecot: + +---%<------------------------------------------------------------------------- +dovecot_login: + driver = dovecot + public_name = LOGIN + server_socket = /var/run/dovecot/auth-client +# setting server_set_id might break several headers in mails sent by +authenticated smtp. So be careful. + server_set_id = $auth1 + +dovecot_plain: + driver = dovecot + public_name = PLAIN + server_socket = /var/run/dovecot/auth-client + server_set_id = $auth1 +---%<------------------------------------------------------------------------- + +If you are having problems with this not working ensure that you are using +version 4.72 or greater of exim. Previous versions of exim have trouble with +the version of the protocol used in Dovecot 2 + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.ImapcProxy.txt b/doc/wiki/HowTo.ImapcProxy.txt new file mode 100644 index 0000000000..805c989230 --- /dev/null +++ b/doc/wiki/HowTo.ImapcProxy.txt @@ -0,0 +1,174 @@ +Dovecot imapc proxy +=================== + +Using Dovecot as a secure IMAP Proxy in front of Exchange, using Exchange +Authentication and IMAPC. This requires Dovecot 2.1.rc1 or newer. + +Many thanks to Timo on the Dovecot mailing list for all his help! + +This " " is based on already having Dovecot already compiled and +installed. + + 1. Create an unprivileged, non-system account user and group for the proxy, + with a home directory. This needs to have a writable home directory, but no + other privileges. + + ---%<--------------------------------------------------------------------- + [root@localhost]# useradd imapproxy + ---%<--------------------------------------------------------------------- + + 2. Verify that the user can not login: + + ---%<--------------------------------------------------------------------- + [root@localhost]# grep imapproxy /etc/shadow + ---%<--------------------------------------------------------------------- + + You should see something like: + + ---%<--------------------------------------------------------------------- + imapproxy:!!:nnnn:0:nn:n::: + ---%<--------------------------------------------------------------------- + + The important part is the "!!". This indicates that the account is locked. + If you don't see this, lockout the account (check man passwd) + 3. Create '/etc/dovecot/dovecot.conf' or + ('/usr/local/etc/dovecot/dovecot.conf') as appropriate: + + ---%<--------------------------------------------------------------------- + ## Dovecot configuration file + + mail_uid = imapproxy + mail_gid = imapproxy + + protocols = imap + + listen = *, :: + + mail_location = imapc:~/imapc + # Change the line below to reflect the IP address of your Exchange Server. + imapc_host = 10.1.2.3 + imapc_port = 143 + + passdb { + driver = imap + # Change the line below to reflect the IP address of your Exchange + Server. + args = host=10.1.2.3 + default_fields = userdb_imapc_user=%u userdb_imapc_password=%w + } + userdb { + driver = prefetch + } + + # /home/imapproxy is the home directory for the imapproxy user, and + # %u is a subdir that will be automatically created for each IMAP user when + they connect + + mail_home = /home/imapproxy/%u + + auth_mechanisms = plain login + + # This is the auth service used by Postfix to do dovecot auth. + service auth { + unix_listener auth-userdb { + } + inet_listener { + port = 12345 + } + } + + ## + ## SSL settings + ## + + # These will need to ba adjusted to point to *your* certificates, not mine + 8-) + # The ssl_ca line refers to the intermediate certificate bundle which may + or may not be required by your SSL provider + + ssl_cert = or . + +Problems with POP-before-SMTP +----------------------------- + + * *Shared IP addresses* are in widespread use. You are opening your server not + only to your user, but to anyone else who might be sharing the same IP + address, other users, other computers in the same NAT. If you lose the + connection, the next one who is assigned your IP also inherits your relay + permit. This might include virus-infected spambot machines. Or consider a + public wireless hotspot or an Internet cafe: both types of establishments + are known to be frequented by spammers. + * *Not properly implemented* in all mail clients: it only works right if the + client checks for new mail immediately before attempting to send. And it can + be very unsafe if longer timeouts are used, such that the user has time to + write an email. + * Probably others. [RobMcGee.txt] ( [RobMcGee.txt]) just + thought it was wrong to have a HOWTO page here without a warning about why + /not/ to. Know what you are doing. If you are setting up a new mail service + from scratch, by all means, do it right! + +Advantages of POP-before-SMTP over SMTP AUTH +-------------------------------------------- + + * Likely to be relatively easier to implement in your mail submission agent. + What's easier is a matter of opinion, and it varies, of course, but probably + all MTA/MSA servers support some form of access lists without patching or + recompiling. + * Simple non-technical instructions for users: /"Remember to check for new + mail before you try to send mail."/ + +Pop-before-smtp.pl +================== + +If you want to use pop-before-smtp.pl (from http://popbsmtp.sourceforge.net/) +together with Dovecot, you can use this regular expression to match successful +POP3 and IMAP logins: + +---%<------------------------------------------------------------------------- +$pat = '^(... .. ..:..:..) \S+ (?:pop3|imap)-login: Login: .+ +\[(\d+\.\d+\.\d+\.\d+)\]'; +---%<------------------------------------------------------------------------- + +v1.0RC2 seems to need this format to work properly: + +---%<------------------------------------------------------------------------- +$pat = '^dovecot: (... .. ..:..:..) \S+ (?:pop3|imap)-login: Login: \S+ \S+ \S+ +lip=(\d+\.\d+\.\d+\.\d+)'; +---%<------------------------------------------------------------------------- + +Note: This only works with IPv4, anyone who wants to fix it for IPv6, please do +so:) + +worked for me on Fedora: + +---%<------------------------------------------------------------------------- +$pat = '(?:pop3|imap)-login: (... .. ..:..:..) Info: Login: \S+ +\[(\d+\.\d+\.\d+\.\d+)\]'; +---%<------------------------------------------------------------------------- + +With v1.0 Alpha 4, the following pattern works: + +---%<------------------------------------------------------------------------- +$pat = '^(... .. ..:..:..) \S+ (?:dovecot: )?(?:imap|pop3)-login: Login: \S+ +\S+ rip=(\d+\.\d+\.\d+\.\d+)' +---%<------------------------------------------------------------------------- + +This works with RHEL 4.3 (at least until IPv6 really catches): + +---%<------------------------------------------------------------------------- +$pat = '(?:pop3|imap)-login: (... .. ..:..:..) Info: Login: \S+ +\[::ffff:(\d+\.\d+\.\d+\.\d+)\]'; +---%<------------------------------------------------------------------------- + +DRAC +==== + +The DRAC historical plugin for Dovecot 1.x, located here +[http://mail.cc.umanitoba.ca/drac/], doesn't work with Dovecot 2.x, since it +relies on the "IP" environment variable, not set anymore by Dovecot 2.x + +a more recent version of this plugin is available here: DRAC Plugin for Dovecot +2.x [http://sourceforge.jp/projects/dovecot2-drac/]. The README file explains +how to compile it. Change the path to your Dovecot 2.x source code into the +Makefile to compile it. + +DRAC runs as a separate daemon, maintaining a BerkeleyDB database of IPs that +have successfully authenticated via POP3 or IMAP, expiring them after 30 +minutes. Installing it therefore requires that both your POP3/IMAP server and +your SMTP daemon (Postfix/Sendmail/qmail) be set up to support it. +DRAC-PLUGIN.c is a small C program, and accessing BerkeleyDB databases is +efficient so it works pretty well. + +By following the instructions you will install a file drac_plugin.so in your +dovecot 'lib/' directories for IMAP and/or POP3 loadable modules. + +To turn on the new DRAC plugin in dovecot, you must set up these lines in your +dovecot.conf. There is a separate section for ''protocol imap'' and another +under ''protocol pop3''; make sure you enable both. + +---%<------------------------------------------------------------------------- + # Support for dynamically loadable modules + mail_plugin_dir = /usr/lib/dovecot/imap # not mandatory + mail_plugins = drac # provide a list of all +plugins you want to load here +---%<------------------------------------------------------------------------- + +Permissions note: the directory containing the drac_plugin.so file has to be +readable by ordinary users. Check your Dovecot error log for help. + +To get DRAC working on your machine, download the main DRAC +[http://mail.cc.umanitoba.ca/drac/] daemon, edit the makefile as directed in +the instructions, and make and install it. You will also want to ensure that +you register the rpcs by executing rpcgen. See the Makefile for more details. + +SQL +=== + +Advantage: you do not have a multi-megabyte Perl daemon reading your logs + +Disadvantage: for each login you need the time and space to execute this script + + 1. tell your MTA to look up IPs authorized to relay in an SQL table + 2. delete old IPs from the table regularly (cron job for example, or a + modification to the script below) + 3. tell dovecot to update the SQL table upon successful login + +Dovecot 1.0 (and probably 0.99) can update a SQL table with the script below. + +/!\ *Note* that *you* must set up a script that deletes old IPs separately, and +*you* also must configure your MTA properly. The script *only* performs the +'update on successful login' step, which alone is insecure without expiring +older IPs!/Add your working examples to this section. This Wiki depends on your +help!/ + +---%<------------------------------------------------------------------------- +#!/bin/sh +# This script created 2005-08-21 by Lorens Kockum +# Released into the Public Domain +# Changes: +# 2006-06-06 Matthias Andree +# - changed $* to "$@" for more robust argument quoting +# Action: when called by dovecot 1.0 as described below, updates an SQL table +# with logged-in IP and current time, and then executes the relevant process. +# Output: normally nothing +# dovecot.conf should be modified with these lines (where +# /usr/lib/dovecot/popbsmtp.sh represents this script): +# protocol pop3 { +# mail_executable = /usr/lib/dovecot/popbsmtp.sh /usr/lib/dovecot/pop3 +# } +# protocol imap { +# mail_executable = /usr/lib/dovecot/popbsmtp.sh /usr/lib/dovecot/imap +# } +# The HOME= lines are necessary to find $HOME/.my.cnf containing login info, +# because mail_executable is executed as root, but without a home directory. +# Of course this script must not be writable by anyone else than root. +( + # drop out IPs from local networks that can relay anyway + IP=`echo $IP | grep -v '^192\.168\.'` + if [ -n "$IP" ] + then + export HOME=/root/ + echo "replace into popbsmtp VALUES('$IP',now());" | mysql mail + export HOME=/ + fi +) >> /var/log/dovecot3 2>&1 +exec "$@" +---%<------------------------------------------------------------------------- + +Example for postgresql, postfix +------------------------------- + +/usr/lib/dovecot/popbsmtp.sh + +---%<------------------------------------------------------------------------- +#!/bin/sh +( + if [ -n "$IP" ] + then + /usr/bin/psql -U popbsmtp -d popbsmtp -c "begin;update auth set +accessed=now() where host=substring('$IP' from 8);commit;insert into auth(host, +accessed) values(substring('$IP' from 8),now());" + fi + +) >> /var/log/dovecot3 2>&1 +exec "$@" +---%<------------------------------------------------------------------------- + +The substring call was necessary because $IP has '::ffff:' or something like +that in front of the IP address on my system. The update followed by an insert, +with the update in a transaction is necessary to replicate mysql's REPLACE INTO +functionality. The INSERT will produce an error if the IP already exists but it +doesn't matter as the UPDATE will have committed by then. + +/etc/postfix/main.cf + +---%<------------------------------------------------------------------------- +smtpd_recipient_restrictions = + permit_mynetworks + permit_sasl_authenticated + permit_tls_clientcerts + check_client_access pgsql:/etc/postfix/popbsmtp.cf + reject_unauth_destination + check_policy_service unix:private/policy +---%<------------------------------------------------------------------------- + +/etc/postfix/popbsmtp.cf + +---%<------------------------------------------------------------------------- +hosts = localhost +user = username +password = secret +dbname = popbsmtp +query = SELECT 'OK' as result FROM auth WHERE host = '%s' +---%<------------------------------------------------------------------------- + +/etc/cron.hourly/popbsmtp_purge + +---%<------------------------------------------------------------------------- +#!/bin/bash +/usr/bin/psql -U popbsmtp -d popbsmtp -c "DELETE FROM auth WHERE (now() - +accessed) > '30 minutes'::interval" +---%<------------------------------------------------------------------------- + +Example for MySQL, postfix +-------------------------- + +Note that you can use this even if pop/imap and smtp are not on the same host +as it is the case in my setup. + +First you have to create a table (in this example named "popbsmtp") with 2 +fields: + + * address (varchar 39, primary) + * last_seen (datetime) + +varchar size 39 is for IPv6 addresses.You should definitely consider adding +IPv6 support to your popbsmtp solution because postfix and dovecot do well with +IPv6. + +/!\ *address field* must be *primary* for "REPLACE into" to work. + +/opt/dovecot-popbsmtp.sh + +---%<------------------------------------------------------------------------- +#!/bin/sh +( + if [ -n "$IP" ] + then + echo "REPLACE INTO virtual_mail.popbsmtp (address,last_seen) +VALUES ('$IP', NOW( ))" \ + | mysql -u user -p secret -h host > /dev/null 2>&1 + fi +) +exec "$@" +---%<------------------------------------------------------------------------- + +mail_executable in dovecot.conf looks something like this: + +---%<------------------------------------------------------------------------- +mail_executable = /opt/dovecot-popbsmtp.sh /usr/libexec/dovecot/imap +---%<------------------------------------------------------------------------- + +postfix map (/etc/postfix/mysql_popbsmtp_access_maps.cf): + +---%<------------------------------------------------------------------------- +hosts = mysqlhost +user = user +password = secret +dbname = virtual_mail +query = SELECT 'OK' FROM popbsmtp WHERE last_seen >= DATE_SUB(NOW(),INTERVAL 30 +MINUTE) AND address = '%s' +---%<------------------------------------------------------------------------- + +In postfix main.cf add the following access map to your recipient restrictions +(/!\ *before* "reject_unauth_destination"): + +---%<------------------------------------------------------------------------- +check_client_access mysql:$config_directory/mysql_popbsmtp_access_maps.cf +---%<------------------------------------------------------------------------- + +The 30 minute relay access period is handled by the INTERVAL in DATE_SUB. So +it's safe anyway, but you should definitely run a cron job daily that deletes +older records. That's to keep the table clean and speed up lookups. You might +also need to run "OPTIMIZE TABLE" via the cron job to free up allocated space. + +relay-ctrl +========== + +relay-ctrl [http://untroubled.org/relay-ctrl/] consists of a few small programs +designed to fit in qmail-like command chains. The most important: + + * 'relay-ctrl-allow' runs after a successful POP/IMAP login, recording the + client IP and timestamp + * 'relay-ctrl-check' runs before the SMTP server, enabling relaying if the + client IP has authenticated recently + +'relay-ctrl-allow' expects to find the client IP in the environment as +'$TCPREMOTEIP'. Dovecot provides it as '$IP', so you'll need this tiny +'dovecot-settcpremoteip' wrapper script: + +---%<------------------------------------------------------------------------- +#!/bin/sh +# +# Wrapper for relay-ctrl-allow that sets TCPREMOTEIP. +TCPREMOTEIP="${IP}"; export TCPREMOTEIP +exec "$@" +---%<------------------------------------------------------------------------- + +Edit 'dovecot.conf' and set 'mail_executable' appropriately, e.g., for IMAP +(this is one long line): + +---%<------------------------------------------------------------------------- +mail_executable = /usr/local/bin/envdir /etc/relay-ctrl +/usr/local/bin/relay-ctrl-chdir /usr/local/bin/dovecot-settcpremoteip +/usr/local/bin/relay-ctrl-allow /usr/local/libexec/dovecot/imap +Dove +---%<------------------------------------------------------------------------- + +Restart Dovecot. Verify that your IMAP client still works. Verify that +relay-ctrl has recorded your client IP. Hook 'relay-ctrl-check' into your SMTP +service, as documented in the relay-ctrl README, and you're done. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.PopRelay.txt b/doc/wiki/HowTo.PopRelay.txt new file mode 100644 index 0000000000..e8e9532357 --- /dev/null +++ b/doc/wiki/HowTo.PopRelay.txt @@ -0,0 +1,139 @@ +Poprelay is a service that allows your Dovecot pop users to also send e-mails +through your sendmail SMTP server on the same machine.When they check their +e-mails, their IP address is logged in a database. Once the same user wants to +send an e-mail using your SMTP port, then sendmail checks this database and if +the IP is present, then the relay is allowed. This makes it completely +transparent to the user and the server will still be very secure and blocking +all other IP addresses for relaying.The IP address is kept in the database for +about 15 minutes only, so there is no chance for anyone else to use the same IP +address and server. + +Dated 2005-08-19 you can now use this poprelay service with Dovecot. + +The main page of poprelayd is the following: +http://sourceforge.net/projects/poprelay + +Download the latest source there. You have an install script that does almost +everything for you. You still need some manual intervention to add some +required lines in your sendmail.mc file. This is described in the install +documentation.Once this is done, you will have to replace your +/etc/mail/poprelay.conf file by the lines below.Once this is done restart the +poprelay service and it should be working. + +Please report any problem on the poprelay forum. You will need to leave the +v6-mapped ip4 address in the /etc/mail/maillog for the moment (e.g. you should +see '[::ffff:IP_ADDRESS]' in the pop user's login logs). + ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +You might have to adapt the first configuration lines if you are using +something else than CentOS 4. + +OpenBSD (using ip4 addresses) produces syslog lines such as: + +Jun 7 09:54:53 mail dovecot: pop3-login: Login: user=, method=PLAIN, +rip=[IP Address], lip=[IP Address] + +To fix the extra word in the split use this line: + +($crap, $info, $string)=split(/\: /,$line); + +To accommodate ip4 address in the form rip=w.x.y.z, use this line: + +($ip) = $line=~/\, rip\=(\d+\.\d+\.\d+\.\d+)/; + +---%<------------------------------------------------------------------------- +#This file is interpreted by perl +#you can do a quick syntax check by doing "perl poprelay.conf" + +#=======================Standard configuration options===================== + +# where POP3/IMAP daemon connections get logged +$logfile = "/var/log/maillog"; + +# Where we put our PID. (dieing output +# will be dumped here too) +$pidfile = "/var/run/poprelayd.pid"; + +# Sendmail map to update. +$dbfile = "/etc/mail/popip.db"; + +#Change this to match the type of db file sendmail needs +#Your perl must support that type of file as well +$dbtype = "DB_HASH"; + +# Minutes an entry lasts. (3000 = ~ 2 days) +# IMAP connections can last a very long time so I like to keep this long. +# The odds that someone will hop onto one of your valid user's old IP's and +# spam from it are so small I wouldn't worry about it. I recommend making +# this long to avoid complications. +$timeout_minutes = 15; + +# Number of seconds to sleep between checks +$log_wait_interval = 5; + +#=======================Advanced configuration options===================== + +#Alternate log line parsers: + +#There can be only one log parser. +#the standard one should work for most systems. The other +#ones may be slightly out of date. I don't have any systems +#that run these servers so I can't update or test the routines. +#If you fix anything with them please let me know and I will +#roll the changes into the main version. +# +#$log_parser = \&log_parse_standard; +#$log_parser = \&log_parse_berkeley; +#$log_parser = \&log_parse_qpopper; +#$log_parser = \&log_parse_qpopper_old; +#$log_parser = \&log_parse_cucipop; +$log_parser = \&log_parse_custom; + +#Custom log line parsing scripts: + +#If you want to create your own log parsing routine, do it here in +#the config file so you can update the poprelayd without losing your +#custom parsing routine. The routine below does the same thing as +#log_parse_standard. It should be a good starting point for any +#customization. It parses lines in many stages so it can be easily +#customized. It will even do dns lookups of hostnames using +#gethostbyname if the program logs the hostname instead of the ip. +# +#If you get something working post it to the forums at +#http://sourceforge.net/projects/poprelay so the next guy doesn't have +#to go through the same headache. I'll try and roll new routines into +#the main program so that poprelayd can work out of the box for all +#the log formats. + +# Dovecot maillog parser: +sub log_parse_custom ($) { + my $s = $_[0]; + my @paddrs; # Packed IP addresses. + my @addrs; # ASCII addresses. + my ($junk,$info,$string,$service,$ip,$host); + ($info, $string)=split(/\: /,$line); + ($service) = $info=~/(\S+)$/; + $service=~s/\[\d+\]//; + return () unless $service=~/^(pop2|pop3|imap)-login$/; + return () unless $string=~/^(Login|Authenticated)/; + ($ip) = $line=~/.*\:\:ffff\:(\d+\.\d+\.\d+\.\d+)\]/; + if ($ip) { + print "$service: $ip\n"; + return ($ip); + } else { + ($host) = $string=~/^(\S+)/; + print "$service: $host\n"; + ($junk, $junk, $junk, $junk, @paddrs) = gethostbyname($host); + while (@paddrs) { + push(@addrs, join('.', unpack('C4', shift(@paddrs)))); + } + return (@addrs); + } +} + +#leave this alone: +1; +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.PostfixAndDovecotSASL.txt b/doc/wiki/HowTo.PostfixAndDovecotSASL.txt new file mode 100644 index 0000000000..48aaf156e7 --- /dev/null +++ b/doc/wiki/HowTo.PostfixAndDovecotSASL.txt @@ -0,0 +1,113 @@ +Postfix and Dovecot SASL +------------------------ + +Since version 2.3, Postfix supports SMTP AUTH through [Sasl.txt] +as introduced in the Dovecot 1.0 series. If using Postfix obtained from a +binary (such as a .rpm or .deb file), you can check if Postfix was compiled +with support for Dovecot SASL by running the command: + +---%<------------------------------------------------------------------------- +postconf -a +---%<------------------------------------------------------------------------- + +Once you have verified that your installation of Postfix supports Dovecot SASL, +it's very simple to configure: + +Example conf.d/10-master.conf excerpt +------------------------------------- + +---%<------------------------------------------------------------------------- +service auth { +... + unix_listener /var/spool/postfix/private/auth { + mode = 0660 + # Assuming the default Postfix user and group + user = postfix + group = postfix + } + ... +} + +# Outlook Express and Windows Mail works only with LOGIN mechanism, not the +standard PLAIN: +auth_mechanisms = plain login +---%<------------------------------------------------------------------------- + +Example Postfix main.cf excerpt +------------------------------- + +---%<------------------------------------------------------------------------- +smtpd_sasl_type = dovecot + +# Can be an absolute path, or relative to $queue_directory +# Debian/Ubuntu users: Postfix is setup by default to run chrooted, so it is +best to leave it as-is below +smtpd_sasl_path = private/auth + +# On Debian Wheezy path must be relative and queue_directory defined +#queue_directory = /var/spool/postfix + +# and the common settings to enable SASL: +smtpd_sasl_auth_enable = yes +# With Postfix version before 2.10, use smtpd_recipient_restrictions +smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, +reject_unauth_destination +---%<------------------------------------------------------------------------- + +Using SASL with Postfix submission port +--------------------------------------- + +When Dovecot is used as the authentication backend for Postfix it is good +practice to use a dedicated submission port for the MUAs (TCP 587). Not only +can you specify individual parameters in *master.cf* overriding the global ones +but you will not run into internet mail rejection while the Dovecot Auth +Mechanism is unavailable. In this example Postfix is configured to accept TLS +encrypted sessions only, along with several other sanity checks: + + * Verification of alias ownership via Login Maps + * Domainname and recipient plausibility + +'master.cf' + +---%<------------------------------------------------------------------------- +submission inet n - n - - smtpd + -o smtpd_tls_security_level=encrypt + -o smtpd_sasl_auth_enable=yes + -o smtpd_sasl_type=dovecot + -o smtpd_sasl_path=private/auth + -o smtpd_sasl_security_options=noanonymous + -o smtpd_sasl_local_domain=$myhostname + -o smtpd_client_restrictions=permit_sasl_authenticated,reject + -o smtpd_sender_login_maps=hash:/etc/postfix/virtual + -o smtpd_sender_restrictions=reject_sender_login_mismatch + -o +smtpd_recipient_restrictions=reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_sasl_authenticated,reject +---%<------------------------------------------------------------------------- + +Dovecot authentication via TCP +------------------------------ + +If Postfix and Dovecot are running on separate servers, you can also +authenticate via TCP. For Dovecot set up an inet_listener: + +---%<------------------------------------------------------------------------- +service auth { + inet_listener { + port = 12345 + } +} +---%<------------------------------------------------------------------------- + +And configure Postfix to use it: + +---%<------------------------------------------------------------------------- +smtpd_sasl_path = inet:dovecot.example.com:12345 +smtpd_sasl_type = dovecot +---%<------------------------------------------------------------------------- + +See also: +--------- + + * http://www.postfix.org/SASL_README.html#server_dovecot + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.Rootless.txt b/doc/wiki/HowTo.Rootless.txt new file mode 100644 index 0000000000..b9f23f7c0a --- /dev/null +++ b/doc/wiki/HowTo.Rootless.txt @@ -0,0 +1,119 @@ +Rootless Installation +===================== + +It's possible to make Dovecot run under a single system user without requiring +root privileges at any point. This shouldn't be thought of as a security +feature, but instead simply as a way for non-admins to run Dovecot in their +favorite mail server. It's also useful if you just wish to test Dovecot without +messing up your system. + +If you think of this as a good way to achieve security, ask yourself which is +worse: + +a) + + * A very small possibility to get root privileges through Dovecot. + * A small possibility without logging in to get into system as a + non-privileged *dovecot* user, chrooted into an empty directory. + * A small possibility to get user's privileges after logging in, but no + possibility to read others' mails since they're saved with different UIDs + (plus you might also be chrooted to your own mailbox). + +b) + + * Absolutely zero possibility to get root privileges through Dovecot. + * A small possibility to get into system as a mail user, possibly even without + logging in, and being able to read everyone's mail (and finally gaining + roots by exploiting some just discovered local vulnerability, unless you + bothered to set up a special chrooted environment). + +Installation +------------ + +Install somewhere under home directory: + +---%<------------------------------------------------------------------------- +./configure --prefix=$HOME/dovecot +make +make install +---%<------------------------------------------------------------------------- + +Dovecot is then started by running '~/dovecot/sbin/dovecot'. The example +configuration file exists in '~/dovecot/share/doc/dovecot/example-config/' and +needs to be copied to '~/dovecot/etc/dovecot/'. + +Configuration +------------- + +The important settings to change for rootless installation are: + + * Set usernames and group to the user which dovecot will be run under: + + ---%<---------------------------------------------------------------------- + default_internal_user = user + default_login_user = user + default_internal_group = group + ---%<---------------------------------------------------------------------- + + * Remove default chrooting from all services: + + ---%<---------------------------------------------------------------------- + service anvil { + chroot = + } + service imap-login { + chroot = + } + service pop3-login { + chroot = + } + ---%<---------------------------------------------------------------------- + + * Change listener ports: + + ---%<---------------------------------------------------------------------- + service imap-login { + inet_listener imap { + port = 10143 + } + inet_listener imaps { + port = 10993 + } + } + service pop3-login { + inet_listener pop3 { + port = 10110 + } + inet_listener pop3s { + port = 10995 + } + } + ---%<---------------------------------------------------------------------- + + * Change logging destination: + + ---%<---------------------------------------------------------------------- + log_path = /home/user/dovecot.log + ---%<---------------------------------------------------------------------- + + * Instead of [PasswordDatabase.PAM.txt] use for example + [AuthDatabase.PasswdFile.txt]: + + ---%<---------------------------------------------------------------------- + passdb { + driver = passwd-file + args = /home/user/dovecot/etc/passwd + } + userdb { + driver = passwd + } + ---%<---------------------------------------------------------------------- + + Where the 'passwd' file contains the username and password for your login + user: + + ---%<---------------------------------------------------------------------- + user:{PLAIN}pass + ---%<---------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.SimpleVirtualInstall.txt b/doc/wiki/HowTo.SimpleVirtualInstall.txt new file mode 100644 index 0000000000..cdc21fe507 --- /dev/null +++ b/doc/wiki/HowTo.SimpleVirtualInstall.txt @@ -0,0 +1,188 @@ +Simple Virtual User Installation +================================ + + * Virtual users configured in '/etc/dovecot/passwd' file + * Assuming an unmodified Dovecot v2.x installation + * Assuming you're not using NFS. See for problems related to it. + +Contents + + + 1. Simple Virtual User Installation + + 2. System configuration + + 3. dovecot.conf + + 4. /etc/dovecot/passwd + + 1. Passwords + + 5. SMTP server configuration + + 1. Delivering mails + + 2. SMTP AUTH + + 6. Quota + +System configuration +==================== + + * Create *dovecot* and *dovenull* users and groups if they don't exist yet. + These are unprivileged users for Dovecot's internal use. They doen't need a + home directory or a shell. + * Create *vmail* user and *vmail* group. This is the user/group that's used to + access the mails. + * Create '/home/vmail' directory owned by vmail:vmail. The mails for all users + are stored under this directory. + * Create '/var/log/dovecot.log' and '/var/log/dovecot-info.log' files owned by + vmail:vmail, so that [LDA.txt] can write to them. + +dovecot.conf +============ + +Below is a fully working 'dovecot.conf' file. You can use it directly, but it +might be better to instead use the included example-config as the base and make +the same modifications to it. + +If you want to configure SSL, see . + +---%<------------------------------------------------------------------------- +protocols = imap pop3 + +# It's nice to have separate log files for Dovecot. You could do this +# by changing syslog configuration also, but this is easier. +log_path = /var/log/dovecot.log +info_log_path = /var/log/dovecot-info.log + +# Disable SSL for now. +ssl = no +disable_plaintext_auth = no + +# We're using Maildir format +mail_location = maildir:~/Maildir + +# If you're using POP3, you'll need this: +pop3_uidl_format = %g + +# Authentication configuration: +auth_verbose = yes +auth_mechanisms = plain +passdb { + driver = passwd-file + args = /etc/dovecot/passwd +} +userdb { + driver = static + args = uid=vmail gid=vmail home=/home/vmail/%u +} +---%<------------------------------------------------------------------------- + +/etc/dovecot/passwd +=================== + +See for the full file format. Here we're +interested only having usernames and passwords in it. Below's an example file: + +---%<------------------------------------------------------------------------- +test:{PLAIN}pass:::::: +bill:{PLAIN}secret:::::: +timo@example.com:{PLAIN}hello123:::::: +dave@example.com:{PLAIN}world234:::::: +joe@elsewhere.org:{PLAIN}whee:::::: +jane@elsewhere.org:{PLAIN}mypass:::::: +---%<------------------------------------------------------------------------- + +As you can see, you can use multiple domains in the file, or no domains at all. +Dovecot doesn't care about domains. The extra colons are needed for +[UserDatabase.txt] passwd-file format, and can be omitted if you are using the +static user database in the example above. + +Users can be added by editing this file. Dovecot automatically notices the new +users immediately after they're added. It also creates their home directories +when the user logs in. + +Passwords +--------- + +The passwords in the example passwd file are listed using plaintext scheme. +It's possible to use other +[Authentication.PasswordSchemes.txt] as well. For example SSHA256 would be a +pretty strong scheme. You can create them using 'doveadm pw' utility, for +example: + +---%<------------------------------------------------------------------------- +doveadm pw -s ssha256 +Enter new password: foo +Retype new password: foo +{SSHA256}ZpgszeowIcHdoxe3BNqvUTtPxFd6fMsyQxEWyY0Qlobaacjk +---%<------------------------------------------------------------------------- + +Note that you won't get the same output after {SSHA256} as above, because +Dovecot uses random salts when creating the SSHA256 hash. This means that even +if multiple users have the same password, you won't know that because their +hashes are different. + +The passwd file entry would be: + +---%<------------------------------------------------------------------------- +{SSHA256}ZpgszeowIcHdoxe3BNqvUTtPxFd6fMsyQxEWyY0Qlobaacjk +---%<------------------------------------------------------------------------- + +Joe would now have "foo" as his password. + +SMTP server configuration +========================= + +Delivering mails +---------------- + +You can configure the SMTP server to deliver mails internally, or you can use + [LDA.txt]. Using dovecot-lda gives you better performance because +it updates Dovecot's index files while saving the mails. See for how +to configure this. Alternatively you can also use . In config you +should have: + +---%<------------------------------------------------------------------------- +protocol lda { + postmaster_address = postmaster@example.com +} +---%<------------------------------------------------------------------------- + +SMTP AUTH +--------- + +If you're using Postfix v2.3+ or Exim v4.64+ you can use Dovecot SASL instead +of Cyrus SASL. + + * [HowTo.PostfixAndDovecotSASL.txt] + * [HowTo.EximAndDovecotSASL.txt] + +Quota +===== + +If you need to have quota, add this to 'dovecot.conf': + +---%<------------------------------------------------------------------------- +mail_plugins = $mail_plugins quota +protocol imap { + mail_plugins = $mail_plugins imap_quota +} +plugin { + quota = maildir +} +---%<------------------------------------------------------------------------- + +Then configure quota by adding 'userdb_quota_rule' +[UserDatabase.ExtraFields.txt] to '/etc/dovecot/passwd', for example: + +---%<------------------------------------------------------------------------- +joe:{PLAIN}pass::::::userdb_quota_rule=*:storage=100M +jane:{PLAIN}pass::::::userdb_quota_rule=*:storage=200M +---%<------------------------------------------------------------------------- + +Joe has now 100MB quota and Jane has 200MB quota. See for more +information about quota settings. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.WriteConfiguration.txt b/doc/wiki/HowTo.WriteConfiguration.txt new file mode 100644 index 0000000000..72960b8d24 --- /dev/null +++ b/doc/wiki/HowTo.WriteConfiguration.txt @@ -0,0 +1,56 @@ +local / remote blocks +===================== + +Use 'local' and 'remote' blocks in this order: + +---%<------------------------------------------------------------------------- +local 127.0.0.1 { +# next block is used only, if TLS SNI extension is used. It expands to the TLS +SNI hostname. +# Typically this is only used to configure per-host TLS certificates. + local_name foo { + remote 127.0.0.1 { + protocol imap { + } + } + } +} +---%<------------------------------------------------------------------------- + +The protocol block is the innermost block always, you can leave some blocks +out. + +Change a Setting +================ + +Typically you can change each setting in another block, like so: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir +mail_plugins = quota +protocol imap { + # overwrites a setting + mail_location = mbox:~/mbox + # adds other values to the setting + mail_plugins = $mail_plugins imap_quota +} +---%<------------------------------------------------------------------------- + +This way, you can keep all settings for a feature, e.g. ACL in one .conf file. + +Unfortunately, you cannot access variables from the plugin section, e.g.: + +---%<------------------------------------------------------------------------- +plugin { + sieve_plugins = extdata +} + +# other file or later + +plugin { + # this won't work !!! + sieve_plugins = $sieve_plugins extdata +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/HowTo.txt b/doc/wiki/HowTo.txt new file mode 100644 index 0000000000..d2be24c536 --- /dev/null +++ b/doc/wiki/HowTo.txt @@ -0,0 +1,93 @@ +HOWTOs / Examples / Tutorials +============================= + + [VirtualUsers.txt] with +[AuthDatabase.PasswdFile.txt]: + + * [HowTo.SimpleVirtualInstall.txt] + + * [HowTo.VirtualhostingWithExim.txt] + * [HowTo.VirtualUserFlatFilesPostfix.txt] + + [VirtualUsers.txt] with [AuthDatabase.LDAP.txt]: + + * [HowTo.DovecotOpenLdap.txt] ( + [HowTo.DoveLdapCheatSheet.txt]) + * Postfix and Active Directory or OpenLDAP [http://www.linuxmail.info] + + [SystemUsers.txt] and/or [VirtualUsers.txt] with + [AuthDatabase.LDAP.txt]: + + * Dovecot, ManageSieve, Exim, OpenLDAP and getmail + [http://www.effinger.org/blog/2009/03/22/dovecot-exim-openldap-und-getmail-unter-ubuntu-1-openldap/] + (Instructions in German) - LDAP users (can be both + [SystemUsers.txt] and [VirtualUsers.txt] depending on how + you use [AuthDatabase.LDAP.txt]) with the possibility to add + subaccounts for each user. For example if you have a LDAP user named peter, + you can add a separate subordinate mailbox to retrieve mail from an external + mail account like peter[at]gmail.com + + [VirtualUsers.txt] with [AuthDatabase.SQL.txt]: + + * MySQL + * [HowTo.DovecotXAMS.txt] + * + [HowTo.DovecotLDAPostfixAdminMySQL.txt] + * MySQL, Exim, SpamAssassin and ClamAV + [http://struction.de/projects/HOWTO_VirtualMail_Exim-MySQL-Spamassassin-ClamAV-Dovecot/] + * Postfix and Dovecot with MySQL and TLS/SSL, Postgrey and DSPAM + [http://johnny.chadda.se/2007/04/15/mail-server-howto-postfix-and-dovecot-with-mysql-and-tlsssl-postgrey-and-dspam/] + * ISP-style Email Server with Debian-Etch and Postfix (MySQL, Dovecot, + Postfix etc.) [http://workaround.org/ispmail] + * PostgreSQL + * [HowTo.DovecotPostgresql.txt] + * PostgreSQL, Postfix (Dovecot LMTP and Dovecot SASL), Dovecot and vmm + (command line tool) [http://vmm.localdomain.org/] + * Installing a fully fledged, ready to use mailserver on Centos 6 with + Postfix, PostgreSQL, Amavis, ClamAV, Spamassassin and Dovecot + [http://shisaa.jp/postset/mailserver-1.html] + * SQLite + * Postfix+Dovecot with SQLite3 backend [http://rob0.nodns4.us/howto/] (also + implements system users) + +Others: + + * SMTP AUTH + * [HowTo.PostfixAndDovecotSASL.txt] + * [HowTo.EximAndDovecotSASL.txt] + * With HALON [http://wiki.halon.se/SASL] + * [HowTo.ChasquidAndDovecotSASL.txt] + * + [HowTo.DebianStable.txt] + * = 2.1)> [HowTo.ImapcProxy.txt] + * [HowTo.CRAM-MD5.txt] + * [HowTo.Rootless.txt] + * [HowTo.PopBSMTPAndDovecot.txt] + * [HowTo.PopRelay.txt] + * [HowTo.RefilterMail.txt] + * [HowTo.TriggerGetmailOnIMAPAccess.txt] + * [HowTo.BynariConnector.txt] + * [HowTo.Fail2Ban.txt] + * [HowTo.VMailMgr.txt] + * + [HowTo.ActiveDirectoryNtlm.txt] + * [HowTo.PostfixDovecotLMTP.txt] + * Apple Discussion Forum: Mail Services in Mac OS X Server v10.6 Snow Leopard + [http://discussions.apple.com/forum.jspa?forumID=1350] + * + [HowTo.Virtual+Postfix+Dspam+Dovecot.txt] + * Postfix + Dovecot2.0.13 + MySQL virtual_users as proxy to DBMail + [http://content.fens.org/index.php?q=admin-howto/mail/dovecot2dbmail-proxy] + * [HowTo.WriteConfiguration.txt] + * [HowTo.AntispamWithSieve.txt] + * Debian + Dovecot + chasquid how-to + [https://blitiri.com.ar/git/r/chasquid/b/master/t/docs/f=howto.md.html] + * The ultimate mail server setup + [https://www.ohreally.nl/2018/11/19/mail-server/]: Dovecot (IMAP + SASL + + LDA + Pigeonhole) + Postfix + Let's Encrypt + Greylisting + SPF + DKIM + + Bogofilter + Clamav + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/IMAPServer.Hibernation.txt b/doc/wiki/IMAPServer.Hibernation.txt new file mode 100644 index 0000000000..21d4a3ffea --- /dev/null +++ b/doc/wiki/IMAPServer.Hibernation.txt @@ -0,0 +1,51 @@ +Hibernation +=========== + +This is not supported on kqueue based systems currently, such as FreeBSD. + +Dovecot supports moving connections that have issued IDLE to a special holding +process, called imap-hibernate. This process is responsible for holding the +idle processes until they need to be thawed. + +Configuration +------------- + +'imap_hibernate_timeout' specifies the delay before moving users to +'imap-hibernate' process. This requires inter-process communication between +'imap' and 'imap-hibernate' process. + +---%<------------------------------------------------------------------------- +imap_hibernate_timeout = 5s + +service imap { + # Note that this change will allow any process running as + # $default_internal_user (dovecot) to access mails as any other user. + # This may be insecure in some installations, which is why this isn't + # done by default. + unix_listener imap-master { + user = $default_internal_user + } +} + +# The following is the default already in v2.3.1+: +service imap { + extra_groups = $default_internal_group +} +service imap-hibernate { + unix_listener imap-hibernate { + mode = 0660 + group = $default_internal_group + } +} +---%<------------------------------------------------------------------------- + +How it works +------------ + +When client issues IDLE, the connection socket is moved to the hibernation +process. This process is responsible for keeping all connections that are +idling, until they issue some command that requires them to be thawed into a +imap process. This way, memory and CPU resources are saved, since there is only +one hibernation process. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/IMAPServer.txt b/doc/wiki/IMAPServer.txt new file mode 100644 index 0000000000..8681cfef08 --- /dev/null +++ b/doc/wiki/IMAPServer.txt @@ -0,0 +1,20 @@ +Dovecot as an IMAP server +========================= + +Dovecot was optimized since the beginning to work as an efficient IMAP server. +Dovecot supports a lot of IMAP extensions [http://imapwiki.org/Specs]. + +Some of the extensions need to be explicitly enabled: + + * [ImapMetadata.txt] + * [Plugins.Compress.txt] + * [Plugins.FTS.txt] + * [MailboxSettings.txt] + * NOTIFY: Set mailbox_list_index=yes + * URLAUTH: Set imap_urlauth_host and mail_attribute_dict + +Other features: + + * [IMAPServer.Hibernation.txt] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/ImapMetadata.txt b/doc/wiki/ImapMetadata.txt new file mode 100644 index 0000000000..858341553c --- /dev/null +++ b/doc/wiki/ImapMetadata.txt @@ -0,0 +1,25 @@ +IMAP METADATA +============= + +Dovecot supports the IMAP METADATA extension (RFC 5464) +[https://tools.ietf.org/html/rfc5464], which allows per-mailbox, per-user data +to be stored and accessed via IMAP commands. + +To activate metadata storage, a [Dictionary.txt] needs to be +configured in the Dovecot configuration using the 'mail_attribute_dict' option. + +To activate the IMAP METADATA commands, the 'imap_metadata' option needs to be +activated. + +Example: + +---%<------------------------------------------------------------------------- +# Store METADATA information within user's Maildir directory +mail_attribute_dict = file:%h/Maildir/dovecot-attributes + +protocol imap { + imap_metadata = yes +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/IndexFiles.txt b/doc/wiki/IndexFiles.txt new file mode 100644 index 0000000000..88ff012968 --- /dev/null +++ b/doc/wiki/IndexFiles.txt @@ -0,0 +1,192 @@ +Dovecot's index files +===================== + +The basic idea behind Dovecot's index files is that it makes reading the +mailboxes a lot faster. The index files consist of the following files: + + * dovecot.index: Main index file + * dovecot.index.cache: Cached mailbox data + * dovecot.index.log: Transaction log file + * dovecot.index.log.2: .log file is rotated to .log.2 file when it grows too + large. + * dovecot.list.index*: Mailbox list index files + +Each mailbox has its own separate index files. If the index files are disabled, +the same structures are still kept in the memory, except cache file is disabled +completely (because the client probably won't fetch the same data twice within +a connection). + +If index files are missing, Dovecot creates them automatically when the mailbox +is opened. If at any point creating a file or growing a file gives "not enough +disk space" error, the indexes are transparently moved to memory for the rest +of the session. This isn't done with mailbox formats that rely on index files +(e.g. dbox). + +See for more technical information how the index files are +handled. + +Main index +---------- + +The main index contains the following information for each message: + + * IMAP UID + * Current flags and keywords + * Pointer to cache file + * mbox-only: mbox file offset + * mbox-only: MD5 sum of some of the message headers, intended to help find the + message when its X-UID: header hasn't yet been written + * Other extensions in Dovecot v1.1+, such as mailbox sorting data + +This is the same information that most other IMAP servers keep in memory while +the mailbox is open, but Dovecot has the advantage of keeping the information +permanently stored so it's easy to get it when opening the mailbox. + +The index file's header also contains some summary information, such as how +many messages exist, how many of them are unseen and how many are marked with +\Deleted flag. Opening mailboxes and answering to STATUS IMAP commands can be +usually done simply by getting the required information from the index file's +header. This is why these operations are extremely fast with Dovecot compared +to other servers that don't use an equivalent index file. + +Mailbox synchronization +----------------------- + +The main index's header also contains mailbox syncing state: + + * Maildir: cur/ and new/ directories' timestamps + * mbox: mbox file's mtime and size + +The index file is synchronized against mailbox only if the syncing information +changes. + +Cache file +---------- + +Cache file may contain the following information for messages: + + * Message headers (some, not all) + * Sent date (parsed Date: header) + * Received date (IMAP's INTERNALDATE field) + * Physical and virtual message sizes + * Message's parsed MIME structure, allowing to quickly read only a specific + MIME part (IMAP's FETCH BODY[1.2.3] command) + * IMAP's BODY and BODYSTRUCTURE fields + * If both are used, only BODYSTRUCTURE is saved, since BODY can be + generated from it + * IMAP's ENVELOPE isn't cached currently. Instead the headers used to build it + are cached directly. + +IMAP clients can work in many different ways. There are basically 2 types: + + 1. Online clients that ask for the same information multiple times (eg. + webmails, Pine) + 2. Offline clients that usually download first some of the interesting message + headers and only after that the message bodies (possibly automatically, or + possibly only when the user opens the mail). Most IMAP clients behave like + this. + +Cache file is extremely helpful with the type 1 clients. The first time that +client requests message headers or some other metadata they're stored into the +cache file. The second time they ask for the same information Dovecot can now +get it quickly from the cache file instead of opening the message and parsing +the headers. + +For type 2 clients the cache file is helpful if they use multiple clients or if +the data was cached while the message was being saved (Dovecot v1.1+ can do +this). Some of the information is helpful in any case, for example it's +required to know the message's virtual size when downloading the message. +Without the virtual size being in cache Dovecot first has to read the whole +message to calculate it. + +Only the mailbox metadata that client(s) have asked for earlier are stored into +cache file. This allows Dovecot to be adaptive to different clients' needs and +still not waste disk space (and cause extra disk I/O!) for fields that client +never needs. + +Dovecot can cache fields either permanently or temporarily. Temporarily cached +fields are dropped from the cache file after about a week. Dovecot uses two +rules to determine when data should be cached permanently instead of +temporarily: + + 1. Client accessed messages in non-sequential order within this session. This + most likely means it doesn't have a local cache. + 2. Client accessed a message older than one week. + + explains the reasons for these rules. + +Transaction log +--------------- + +All changes to the main index go through transaction log first. This has two +advantages when the mailbox is accessed using multiple simultaneous +connections: + + 1. It allows getting a list of changes quickly so that IMAP clients can be + notified of the changes. An alternative would be to do a comparison of two + index mappings, which is what most other IMAP servers do. + 2. 'mmap_disable=yes' implementation relies on the transaction log. Instead of + re-reading the whole main index file after each change it's necessary to + only read a few bytes from the transaction log. + +In Dovecot v1.1+ the transaction log plays an even more important role. The +main index file is updated only "once in a while" to reduce disk writes, so it +is common to first read the main index and then apply new changes from the +transaction log on top of that. With empty mailboxes (eg. download+delete POP3 +users) it would even be possible to delete the whole main index and keep only +the transaction log (although this isn't done currently). + +List index +---------- + +Mailbox list index file is called dovecot.list.index[.log] and it basically +contains: + + * Header contains ID => name mapping. The name isn't the full mailbox name, + but rather each hierarchy level has its own ID and name. For example a + mailbox name "foo/bar" (with '/' as separator) would have separate IDs for + "foo" and "bar" names. + * The records contain { parent_uid, uid, name_id } field that can be used to + build the whole mailbox tree. parent_uid=0 means root, otherwise it's the + parent node's uid. + * Each record also contains GUID for each selectable mailbox. If a mailbox is + recreated using the same name, its GUID also changes. Note however that the + UID doesn't change, because the UID refers to the mailbox name, not to the + mailbox itself. + * The records may contain also extensions for allowing mailbox_get_status() to + return values directly from the mailbox list index. + * Storage backends may also add their own extensions to figure out if a record + is up to date. + +Settings +-------- + +Since v2.2.34+ you can configure some of the hardcoded optimization-related +settings. It's not recommended to change these settings without fully +understanding the consequences. + + * 'mail_cache_unaccessed_field_drop': Drop fields that haven't been accessed + for n seconds. + * 'mail_cache_record_max_size': If cache record becomes larger than this, + don't add it. + * 'mail_cache_compress_min_size': Never compress the file if it's smaller than + this. + * 'mail_cache_compress_delete_percentage': Compress the file when n% of + records are deleted (by count, not by size). + * 'mail_cache_compress_continued_percentage': Compress the file when n% of + rows contain continued rows. For example 200% means that the record has 2 + continued rows, i.e. it exists in 3 separate segments in the cache file. + * 'mail_cache_compress_header_continue_count': Compress the file when we need + to follow more than n next_offsets to find the latest cache header. + * 'mail_index_rewrite_min_log_bytes', 'mail_index_rewrite_max_log_bytes': + Rewrite the index when the number of bytes that needs to be read from the + .log on refresh is between these min/max values. + * 'mail_index_log_rotate_min_size', 'mail_index_log_rotate_max_size', + 'mail_index_log_rotate_min_age': Rotate transaction log after it's a) + min_size or larger and it was created at least min_age_secs or b) larger + than max_size. + * 'mail_index_log2_max_age': Delete .log.2 when it's older than + log2_stale_secs. Don't be too eager, because older files are useful for + QRESYNC and dsync. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.Exim.txt b/doc/wiki/LDA.Exim.txt new file mode 100644 index 0000000000..1b2ab214f8 --- /dev/null +++ b/doc/wiki/LDA.Exim.txt @@ -0,0 +1,163 @@ +Dovecot LDA with Exim +===================== + +System users +------------ + +Change the localuser router to use dovecot_delivery transport: + +---%<------------------------------------------------------------------------- +localuser: + driver = accept + check_local_user +# local_part_suffix = +* : -* +# local_part_suffix_optional + transport = dovecot_delivery +---%<------------------------------------------------------------------------- + +'check_local_user' is required. It makes Exim execute the transport with the +user's UID and GID and it also sets HOME environment. + +Next create a new transport for dovecot-lda: + +---%<------------------------------------------------------------------------- +dovecot_delivery: + driver = pipe + + # Use /usr/lib/dovecot/dovecot-lda if using Debian's package. + # You may or may not want to add -d $local_part@$domain depending on if you +need a userdb lookup done. + command = /usr/local/libexec/dovecot/dovecot-lda -f $sender_address + + message_prefix = + message_suffix = + log_output + delivery_date_add + envelope_to_add + return_path_add + #group = mail + #mode = 0660 + temp_errors = 64 : 69 : 70: 71 : 72 : 73 : 74 : 75 : 78 +---%<------------------------------------------------------------------------- + +LDA is now running using the local user's UID and GID. The mail is delivered to +the location specified by [MailLocation.txt] setting. Note that +the above configuration doesn't do any [UserDatabase.txt] lookups, so +you can't have any per-user configuration. If you want that, see the virtual +user setup below. + +Virtual users +------------- + +Make sure that 'check_local_user' isn't set in the router. + +Single UID +---------- + +Configure the transport to run as the user you want, for example vmail: + +---%<------------------------------------------------------------------------- +dovecot_virtual_delivery: + driver = pipe + command = /usr/local/libexec/dovecot/dovecot-lda -d $local_part@$domain -f +$sender_address + # v1.1+: command = /usr/local/libexec/dovecot/dovecot-lda -d +$local_part@$domain -f $sender_address -a +$original_local_part@$original_domain + message_prefix = + message_suffix = + delivery_date_add + envelope_to_add + return_path_add + log_output + user = vmail + temp_errors = 64 : 69 : 70: 71 : 72 : 73 : 74 : 75 : 78 +---%<------------------------------------------------------------------------- + +You'll also need to have a master authentication socket and give vmail user +access to it. See for more information. + +List of temp_errors can be found in '/usr/include/sysexits.h'. + +Multiple UIDs +------------- + +If you need multiple uids/gids you'll need to set dovecot-lda setuid root. See + [LDA.txt] for how to do this securely. + +You could alternatively set 'user = root', but this requires that you built +Exim without root being in FIXED_NEVER_USERS list. + +Multiple UIDs, without running dovecot-lda as root +-------------------------------------------------- + +In this mode, dovecot-lda won't be querying Dovecot's master socket, instead +trusting Exim to setup its execution environment. This means you must set up +Exim to get the UID, GID, Home directory from LDAP/SQL/whatever. Here, we're +setting them in the router and the transport automatically inherits them. + +Router configuration +-------------------- + +Insert the following router after your external delivery routers and before +your local system delivery routers. + +This assumes you're using macros set elsewhere to handle your external queries, +as they can quickly become unwieldy to manage. Make sure you adjust it to suit +your installation first! + +---%<------------------------------------------------------------------------- +ldap_local_user: + debug_print = "R: ldap_local_user for $local_part@$domain" + driver = accept + domains = +ldap_local_domains + condition = LDAP_VIRT_COND + router_home_directory = LDAP_VIRT_HOME + user = LDAP_VIRT_UID + group = LDAP_VIRT_GID + #local_part_suffix = +* : -* + #local_part_suffix_optional + transport = dovecot_lda +---%<------------------------------------------------------------------------- + +Transport configuration +----------------------- + +This transport has been tested with Exim 4.69-9 and Dovecot 1:1.2.5-2 +(backported) on Debian Lenny. You also have to set + +---%<------------------------------------------------------------------------- +dovecot_lda: + debug_print = "T: dovecot_lda for $local_part@$domain" + driver = pipe + # Uncomment the following line and comment the one after it if you want +dovecot-lda to try + # to deliver subaddresses into INBOX.{subaddress}. If you do this, uncomment +the + # local_part_suffix* lines in the router as well. Make sure you also change +the separator + # to suit your local setup. + #command = /usr/lib/dovecot/dovecot-lda -e -k -m +"INBOX|${substr_1:$local_part_suffix}" \ + command = /usr/lib/dovecot/dovecot-lda -e -k \ + -f "$sender_address" -a "$original_local_part@$original_domain" + environment = USER=$local_part@$domain + home_directory = /var/mail/home/$domain/$local_part + umask = 002 + message_prefix = + message_suffix = + delivery_date_add + envelope_to_add + return_path_add + log_output + log_defer_output + return_fail_output + freeze_exec_fail + #temp_errors = * + temp_errors = 64 : 69 : 70 : 71 : 72 : 73 : 74 : 75 : 78 +---%<------------------------------------------------------------------------- + +You need to have [VirtualUsers.Home.txt] set to have duplicate +database enabled, among other reasons. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.Indexing.txt b/doc/wiki/LDA.Indexing.txt new file mode 100644 index 0000000000..abde8f8e79 --- /dev/null +++ b/doc/wiki/LDA.Indexing.txt @@ -0,0 +1,47 @@ +LDA Indexing +============ + +LDA's indexing basically does two things while message is being saved: + + 1. It updates the main index file. + * This improves performance with [MailboxFormat.mbox.txt] format, + especially if 'mbox_very_dirty_syncs=no'. + * With [MailboxFormat.Maildir.txt] the benefits of this are + almost irrelevant. + 2. It updates the 'dovecot.index.cache' file. + +Cache file +---------- + +The LDA also updates the cache file, which can be very useful with all mailbox +formats. It means that when an IMAP client wants to fetch the message's +metadata (e.g. some header fields), they're can be retrieved from the cache +file and Dovecot doesn't have to open and parse the message file. There are +some tradeoffs though: + + * LDA indexing wastes disk I/O because it has to open and update index files + * LDA indexing saves disk I/O because it already has the message body in + memory, so it doesn't need to read it from disk. + * IMAP indexing wastes disk I/O because it has to open and read message files + * IMAP indexing may save disk I/O because IMAP process always has index files + opened, and many IMAP clients are configured to download all new message + bodies anyway, so the second time message bodies are read they're already in + memory + +So it depends on IMAP client if it's faster to use LDA or IMAP time indexing. +In any case the user experience is typically faster with LDA indexing, because +the message list metadata can be returned faster when it's pre-indexed. + +See for more information about what the index files contain. + +Non-indexed mail delivery +------------------------- + +Ignoring the benefits of cache file updates, the only thing left is the main +index updates. As mentioned above, with Maildir format these benefits are very +small. This also means that it's perfectly fine to use a non-Dovecot MDA to +deliver mails that doesn't update indexes. Dovecot can efficiently see and +index such new mails without doing anything expensive like "rebuilding +indexes". + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.Postfix.txt b/doc/wiki/LDA.Postfix.txt new file mode 100644 index 0000000000..bf519af175 --- /dev/null +++ b/doc/wiki/LDA.Postfix.txt @@ -0,0 +1,255 @@ +Dovecot LDA with Postfix +======================== + +This page contains only information specific to using LDA with Postfix, see + for more information about using the LDA itself. + +System users +------------ + +If you wish you use 'dovecot-lda' for all system users on a single domain mail +host you can do it by editing 'mailbox_command' parameter in + +'/etc/postfix/main.cf' (postconf(5) [http://www.postfix.org/postconf.5.html]): + +---%<------------------------------------------------------------------------- +mailbox_command = /usr/local/libexec/dovecot/dovecot-lda -f "$SENDER" -a +"$RECIPIENT" +# or +mailbox_command = /usr/libexec/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT" +# or +mailbox_command = /usr/lib/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT" +# or wherever it was installed in your system. +---%<------------------------------------------------------------------------- + +Then run 'postfix reload'. + + * This command doesn't do a [UserDatabase.txt] lookup. If you want + that (e.g. for per-user quota lookups) you need to add '-d "$USER"' + parameter. + * Postfix runs 'mailbox_command' with both the uid and gid of the destination + user. This may not allow 'dovecot-lda' to write a lock file in '/var/mail'. + When this directory is writable by a privileged group (say 'main', see the + option 'mail_privileged_group'), we can use the setgid permission bit on the + 'dovecot-lda' executable: + + ---%<---------------------------------------------------------------------- + # chgrp mail /usr/lib/dovecot/dovecot-lda + # chmod 2755 /usr/lib/dovecot/dovecot-lda + ---%<---------------------------------------------------------------------- + + Alas these permission will disappear if you update dovecot. A more robust + way to do so is to compile a relay program '/etc/postfix/dovecot-lda-relay' + that has the setgid permission and execs the real 'dovecot-lda'. + + ---%<---------------------------------------------------------------------- + # cd /etc/postfix + # cat >dovecot-lda-relay.c < + char *pgm = "/usr/lib/dovecot/dovecot-lda"; /* wherever dovecot-lda is + located */ + int main{int argc, char**argv) { argv[0]=pgm; execv(pgm,argv); return + 10; } + EOF + # gcc -o dovecot-lda-relay dovecot-lda-relay.c + # chown root:mail dovecot-lda-relay + # chmod 2755 dovecot-lda-relay + ---%<---------------------------------------------------------------------- + + Then, simply invoke '/etc/postfix/dovecot-lda-relay' instead of + 'dovecot-lda' in 'mailbox_command'. + * Postfix's 'mailbox_size_limit' setting applies to all files that are written + via dovecot-lda. The default is 50 MB, so dovecot-lda can't write *any* + files larger than that, including mbox files or log files. This shows up + only in Dovecot's logs: + + ---%<---------------------------------------------------------------------- + dovecot-lda(user): write() failed with mbox file /home/user/mail/foo: File + too large (process was started with ulimit -f limit) + ---%<---------------------------------------------------------------------- + + * If you have trouble seeing anything in Dovecot's logs, see + [LDA.txt]. + +Virtual users +------------- + +Dovecot LDA is very easy to use on large scale installations with Postfix +virtual domains support, just add a 'dovecot' service in +'/etc/postfix/master.cf' (master(5) [http://www.postfix.org/master.5.html]): + +---%<------------------------------------------------------------------------- +dovecot unix - n n - - pipe + flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f +${sender} -d ${recipient} +---%<------------------------------------------------------------------------- + +An example using address extensions (ie user+extension@domain.com (don't forget +to define the proper recipient_delimiter in Postfix's main.cf)) to deliver to +the folder 'extension' in your maildir (If you wish to preserve the case of +${extension}, remove the 'hu'flags [http://www.postfix.org/pipe.8.html], and be +sure to utilize [Variables.txt] in your dovecot.conf for mail +locations and other configuration parameters that are expecting lower case): + +---%<------------------------------------------------------------------------- +dovecot unix - n n - - pipe + flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f +${sender} -d ${user}@${nexthop} -m ${extension} + +# or if you have a INBOX/ namespace prefix: +dovecot unix - n n - - pipe + flags=DRhu user=vmail:vmail argv=/usr/local/libexec/dovecot/dovecot-lda -f +${sender} -d ${user}@${nexthop} -m INBOX/${extension} +---%<------------------------------------------------------------------------- + +This example ignores address extensions (ie user+extension@domain.com delivers +just like user@domain.com ), but still shows the original address for Sieve: + +---%<------------------------------------------------------------------------- +dovecot unix - n n - - pipe + flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/dovecot-lda -f ${sender} -a +${original_recipient} -d ${user}@${nexthop} +---%<------------------------------------------------------------------------- + +Replace 'vmail' above with your virtual mail user account. + +Then set 'virtual_transport' to 'dovecot' in '/etc/postfix/main.cf': + +---%<------------------------------------------------------------------------- +dovecot_destination_recipient_limit = 1 +virtual_mailbox_domains = your.domain.here +virtual_transport = dovecot +---%<------------------------------------------------------------------------- + +And remember to run + +---%<------------------------------------------------------------------------- +postfix reload +---%<------------------------------------------------------------------------- + +Virtual users with multiple uids/gids +------------------------------------- + +If you need multiple uids/gids you'll need to set dovecot-lda setuid root or +invoke it through sudo. See [LDA.txt] for how to do this +securely. + +Postfix with a NFS mail store +----------------------------- + +If you are experiencing problems with dovecot-lda processes hanging when +delivering to an NFS mail store, it's likely that the dovecot-lda process is +hanging while waiting for free locks. The occurrence of this can be greatly +reduced, if not eradicated, by forcing Postfix to only deliver to the same +recipient one at a time. + +---%<------------------------------------------------------------------------- +dovecot_destination_concurrency_limit = 1 +---%<------------------------------------------------------------------------- + +Prevent backscatter +------------------- + +To prevent backscatter you should configure Postfix to reject mail for non +existent recipients. + +This is the default behaviour (smtpd_reject_unlisted_recipient = yes) so +there's no need to set "reject_unlisted_recipient" in any of your restriction. +But: Postfix must know if a recipient exists. Depending on how you've +configured Dovecot and Postfix this can be done several ways. + +System users +------------ + +If you only use local system users this is no problem - all valid recipients +can be found in the local password or alias database. + +Virtual users (static) +---------------------- + +When you use virtual users and domains you should maintain a list of valid +recipients. The relevant settings settings are: + +*virtual_alias_maps, virtual_mailbox_maps* + +For static verification you can maintain the content of the files yourself. For +every recipient or alias you need one entry. Example: + +*virtual_alias_maps* + +---%<------------------------------------------------------------------------- +name_recipient@example.com external@example.net +---%<------------------------------------------------------------------------- + +*virtual_mailbox_maps* + +---%<------------------------------------------------------------------------- +name@example.com OK +recipient@example.com available +---%<------------------------------------------------------------------------- + +Don't forget to run "postmap" afterwards. + +*Info:* if you use the Dovecot LDA or LMTP it doesn't matter what you use +behind the recipient address. Use "OK", the full name of the user or else. + +Virtual users (dynamic) +----------------------- + +Do you already use a database (MySQL, PostgreSQL) for Dovecot? Use the same +source for Postfix. You only have to to define a valid sql query for Postfix. +Example: + +---%<------------------------------------------------------------------------- +virtual_mailbox_maps = proxy:mysql:/etc/postfix/virtual_mailbox_maps.cf +---%<------------------------------------------------------------------------- + +*virtual_mailbox_maps.cf* + +---%<------------------------------------------------------------------------- +user = mysql-user +password = mysql-password +hosts = unix:/var/run/mysql/mysqld.sock +dbname = mailserver +query = SELECT name FROM mailbox WHERE email='%s' +---%<------------------------------------------------------------------------- + +This query will return the value of the filed "name" from table "mailbox" if +the email address of the recipient matches the email from the field "email". +This is enough for Postfix because Postfix must only know if the recipient +exists. The value doesn't matter. When you use a database (or LDAP) there's no +need to manually maintain a file with valid recipients. + +*Info:* If you use "relay_domains" instead of "virtual_mailbox_domains" you +have to use "relay_recipient_maps" instead of "virtual_mailbox_maps". + +Dynamic address verification with LMTP +-------------------------------------- + +With Dovecot 2.0 you can also use LMTP and the Postfix setting +"reject_unverified_recipient" for dynamic address verification. It's really +nice because Postfix doesn't need to query an external datasource (MySQL, +LDAP...). Postfix maintain a local database with existing/non existing +addresses (you can configure how long positive/negative results should be +cached). + +To use LMTP and dynamic address verification you must first get Dovecot +working. Then you can configure Postfix to use LMTP and set +"reject_unverified_recipient" in the smtpd_recipient_restrictions. + +On every incoming email Postfix will probe if the recipient address exists. You +will see similar entries in your logfile: + +---%<------------------------------------------------------------------------- +Recipient address rejected: undeliverable address: host +tux.example.com[private/dovecot-lmtp] said: 550 5.1.1 < tzknvtr@example.com > +User doesn't exist: tzknvtr@example.com (in reply to RCPT TO command); from=< +cnrilrgfclra@spammer.org > to=< tzknvtr@example.com > +---%<------------------------------------------------------------------------- + +If the recipient address exists (status=deliverable) Postfix accepts the mail. + +*Info:* you can not use "reject_unverified_recipient" with "pipe" so this +doesn't work with the Dovecot LDA "deliver". + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.Qmail.txt b/doc/wiki/LDA.Qmail.txt new file mode 100644 index 0000000000..d13756dda3 --- /dev/null +++ b/doc/wiki/LDA.Qmail.txt @@ -0,0 +1,31 @@ +Dovecot LDA with Qmail +====================== + +System users +------------ + +The delivery command you need is + +---%<------------------------------------------------------------------------- +|/var/qmail/bin/preline -f /usr/local/libexec/dovecot/dovecot-lda +---%<------------------------------------------------------------------------- + +(You may need to adjust the paths to match your qmail and dovecot +installations.) The 'preline' command will add the 'Return-Path:' and +'Delivered-To:' lines, because 'dovecot-lda' doesn't recognize qmail's +environment variables. + +For site-wide usage, put that in '/var/qmail/control/defaultdelivery' (assuming +you installed qmail according to LWQ [http://www.lifewithqmail.org/lwq.html]). +Or, save it as '.qmail' in selected users' home directories. + +Virtual users +------------- + +Add the '-d' parameter to specify the destination username: + +---%<------------------------------------------------------------------------- +|/var/qmail/bin/preline -f /usr/local/libexec/dovecot/dovecot-lda -d $EXT@$USER +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.Sendmail.txt b/doc/wiki/LDA.Sendmail.txt new file mode 100644 index 0000000000..b14e4f003e --- /dev/null +++ b/doc/wiki/LDA.Sendmail.txt @@ -0,0 +1,105 @@ +Dovecot LDA with Sendmail +========================= + +The following describes how to configure Sendmail to use 'dovecot-lda' where +'root' permission is not granted and Dovecot runs under a single user ID. It +may need some adjustment for more typical setups. Other assumptions are that +Sendmail is configured for virtual hosting and that local-system mail delivery +is not handled by 'dovecot-lda'. + +Allowing that 'sendmail.mc' has 'MAILER(procmail)dnl' included, edit +'sendmail.cf' adding these lines after the 'Mprocmail' definition: + +---%<------------------------------------------------------------------------- +######################*****############## +### DOVECOT Mailer specification ### +##################*****################## +Mdovecot, P=/usr/local/libexec/dovecot/dovecot-lda, F=DFMPhnu9, + S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP/HdrFromSMTP, + T=DNS/RFC822/X-Unix, + A=/usr/local/libexec/dovecot/dovecot-lda -d $u +---%<------------------------------------------------------------------------- + +If you're using 'sendmail.mc' then put the lines above into a new file +'/usr/share/sendmail-cf/mailer/dovecot.m4' and put 'MAILER(dovecot)' into your +'sendmail.mc' + +=================================== + +Another method of doing the above is by editing your 'hostname.mc' with the +following three lines: + +---%<------------------------------------------------------------------------- +FEATURE(`local_procmail', +`/usr/local/libexec/dovecot/dovecot-lda',`/usr/local/libexec/dovecot/dovecot-lda +-d $u') +MODIFY_MAILER_FLAGS(`LOCAL', `-f') +MAILER(procmail) +---%<------------------------------------------------------------------------- + +After editing 'hostname.mc' with the above, be sure to remake your +'hostname.cf' file. This is confirmed to work with: + + * dovecot-1.0.7 + * FreeBSD 6.3-RELEASE-p3 i386 + * sendmail Version 8.14.2 + * Compiled with: DNSMAP LOG MAP_REGEX MATCHGECOS MILTER MIME7TO8 MIME8TO7 + NAMED_BIND NETINET NETINET6 NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF + STARTTLS TCPWRAPPERS USERDB XDEBUG + +=================================== + +If 'sendmail' runs under a different non-'root' UID via + + * 'define(`confRUN_AS_USER', `sendmail')dnl' + +in 'sendmail.mc', then the /env_put(t_strconcat("RESTRICT_/ lines in +'deliver.c' must be commented-out. + +Now add a + +---%<------------------------------------------------------------------------- +virtualdomain.example.com vmail:vmail +---%<------------------------------------------------------------------------- + +line for each virtual domain to 'mailertable.cf' and run 'makemap hash +mailertable.db < mailertable.cf'. The 'dovecot' (or some other random text) +after the colon character is required, else 'sendmail' will fail to pass +command arguments to 'dovecot-lda' correctly. Make sure all the virtual +domains are in the 'virtuserdomains' file. + +=========================================== + +(Fedora 14: dovecot 2.0.8 & sendmail 8.14.4) + +Summing up all previous experience, one may keep all virtual user accounts +under one system account. + +The sendmail's "U=" mailer option with changing the owner of lda (to "keeper" +here for instance): + +---%<------------------------------------------------------------------------- +-rwxr-xr-x. 1 keeper mail 14536 Dec 7 16:43 /usr/libexec/dovecot/dovecot-lda +---%<------------------------------------------------------------------------- + +allows to run virtual users under one system account without applying SUID. + +Sendmail can pass a user account to LDA with or without the domain. Passing a +user name without the domain can be achievedwith S=/R= rewriting rules of the +local mailer. Finally, into '/usr/share/sendmail-cf/mailer/dovecot.m4' goes the +block of lines: + +---%<------------------------------------------------------------------------- +Mdovecot, P=/usr/libexec/dovecot/dovecot-lda, + F=l59DFMPhnu, + S=EnvFromL/HdrFromL, R=EnvToL/HdrToL, + M=51200000, + U=keeper:mail, + T=DNS/RFC822/X-Unix, + A=/usr/libexec/dovecot/dovecot-lda -d $u +---%<------------------------------------------------------------------------- + +dovecot.m4 [http://sites.google.com/site/mclroy/dovecot/dovecot-m4] can be a +bit more complex. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LDA.txt b/doc/wiki/LDA.txt new file mode 100644 index 0000000000..80e2ee6f9e --- /dev/null +++ b/doc/wiki/LDA.txt @@ -0,0 +1,298 @@ +Dovecot LDA +=========== + +The Dovecot LDA is a [MDA.txt], which takes mail from an + and delivers it to a user's mailbox, while keeping Dovecot index +files up to date. Nowadays you should probably use the [LMTP.txt] +instead, because it's somewhat easier to configure (especially related to +permissions) and gives better performance. + +This page describes the common settings required to make LDA work. You should +read it first, and then the MTA specific pages: + + * + * + * + * + * + +Main features of Dovecot LDA +---------------------------- + + * [LDA.Indexing.txt], providing faster + mailbox access later + * [Quota.txt] + * [Pigeonhole.Sieve.txt] + * Mail filtering + * Mail forwarding + * Vacation auto-reply + +Common configuration +-------------------- + +The settings are listed in the example 'conf.d/15-lda.conf' file. The important +settings are: + + * 'postmaster_address' is used as the From: header address in bounce mails + * 'hostname' is used in generated Message-IDs and in Reporting-UA: header in + bounce mails + * 'sendmail_path' is used to send mails. Note that the default is + '/usr/sbin/sendmail', which doesn't necessarily work the same as + '/usr/lib/sendmail'. + * Alternatively you can use 'submission_host' to send mails via the + specified SMTP server. + * 'auth_socket_path' specifies the UNIX socket to auth-userdb where LDA can + lookup userdb information when '-d' parameter is used. See below how to + configure Dovecot to configure the socket. + +Note that the config files must be world readable to enable dovecot-lda process +read them, while running with user privileges. You can put password related +settings to a separate file, which you include with '!include_try' and +dovecot-lda skips them. + +Parameters +---------- + +Parameters accepted by dovecot-lda: + + * '-d ': Destination username. If given, the user information is + looked up from dovecot-auth. Typically used with virtual users, but not + necessarily with system users. + * '-a
': Original envelope recipient address (e.g. user+ext@domain), + typically same as SMTP's RCPT TO: value. If not specified, it's taken from + header specified by 'lda_original_recipient_header' setting (v2.0.3+). If + the header doesn't exist either, defaults to same as username. + * '-r
': Final envelope recipient address. Defaults to -a address, + but may differ if e.g. aliases are used or when dropping the +ext part. + (v2.0.3+) + * '-f
': Envelope sender address. If not specified and message data + begins with a valid mbox-style "From " line, the address is taken from it. + * '-c ': Alternative configuration file path. + * '-m ': Destination mailbox (default is INBOX). If the mailbox + doesn't exist, it will not be created (unless the lda_mailbox_autocreate + setting is set to yes). If message couldn't be saved to the mailbox for any + reason, it's delivered to INBOX instead. + * If Sieve plugin is used, this mailbox is used as the "keep" action's + mailbox. It's also used if there is no Sieve script or if the script + fails for some reason. + * Deliveries to namespace prefix will result in saving the mail to INBOX + instead. For example if you have "Mail/" namespace, this allows you to + specify 'dovecot-lda -m Mail/$mailbox' where mail is stored to + Mail/$mailbox or to INBOX if $mailbox is empty. + * The mailbox name is specified the same as it's visible in IMAP client. + For example if you've a Maildir with '.box.sub/' directory and your + namespace configuration is 'prefix=INBOX/', 'separator=/', the correct + way to deliver mail there is to use '-m INBOX/box/sub' + * '-e': If mail gets rejected, write the rejection reason to stderr and exit + with EX_NOPERM. The default is to send a rejection mail ourself. + * '-k': Don't clear all environment at startup. + * '-p ': Path to the mail to be delivered instead of reading from stdin. + If using maildir the file is hard linked to the destination if possible. + This allows a single mail to be delivered to multiple users using hard + links, but currently it also prevents dovecot-lda from updating cache file + so it shouldn't be used unless really necessary. + * '-o name=value': Override a setting from dovecot.conf. You can give this + parameter multiple times. + +Return values +------------- + +dovecot-lda will exit with one of the following values: + + * 0 (EX_OK): Delivery was successful. + * 64 (EX_USAGE): Invalid parameter given. + * 67 (EX_NOUSER): The destination username was not found. + * 77 (EX_NOPERM): -e parameter was used and mail was rejected. Typically this + happens when user is over quota and 'quota_full_tempfail=no'. + * 75 (EX_TEMPFAIL): A temporary failure. This is returned for almost all + failures. See the log file for details. + +System users +------------ + +You can use LDA with a few selected system users (ie. user is found from +'/etc/passwd' / NSS) by calling dovecot-lda in the user's '~/.forward' file: + +---%<------------------------------------------------------------------------- +| "/usr/local/libexec/dovecot/dovecot-lda" +---%<------------------------------------------------------------------------- + +This should work with any MTA which supports per-user '.forward' files. For +qmail's per-user setup, see . + +This method doesn't require the authentication socket explained below since +it's executed as the user itself. + +Virtual users +------------- + +With a lookup +------------- + +Give the destination username to dovecot-lda with '-d' parameter, for example: + +---%<------------------------------------------------------------------------- +dovecot-lda -f $FROM_ENVELOPE -d $DEST_USERNAME +---%<------------------------------------------------------------------------- + +You'll need to set up a auth-userdb socket for dovecot-lda so it knows where to +find mailboxes for the users: + +---%<------------------------------------------------------------------------- +service auth { + unix_listener auth-userdb { + mode = 0600 + user = vmail # User running dovecot-lda + #group = vmail # Or alternatively mode 0660 + dovecot-lda user in this +group + } +} +---%<------------------------------------------------------------------------- + +The auth-userdb socket can be used to do [UserDatabase.txt] lookups +for given usernames or get a list of all users. Typically the result will +contain the user's UID, GID and home directory, but depending on your +configuration it may return other information as well. So the information is +similar to what can be found from eg.'/etc/passwd' for system users. This means +that it's probably not a problem to use mode=0666 for the socket, but you +should try to restrict it more just to be safe. + +Without a lookup +---------------- + +If you have already looked up the user's home directory and you don't need a +userdb lookup for any other reason either (such as overriding settings for +specific users), you can run dovecot-lda similar to how it's run for system +users: + +---%<------------------------------------------------------------------------- +HOME=/path/to/user/homedir dovecot-lda -f $FROM_ENVELOPE +---%<------------------------------------------------------------------------- + +This way you don't need to have a master listener socket. Note that you should +verify the user's existence prior to running dovecot-lda, otherwise you'll end +up having mail delivered to nonexistent users as well. + +You must have set the proper UID (and GID) before running dovecot-lda. It's not +possible to run dovecot-lda as root without '-d' parameter. + +Multiple UIDs +------------- + +If you're using more than one UID for users, you're going to have problems +running dovecot-lda, as most MTAs won't let you run dovecot-lda as root. Best +solution is to use instead, but if you can't do that, there are two +ways to work around this problem: + + 1. Make dovecot-lda setuid-root. + 2. Use sudo to wrap the invocation of dovecot-lda. + +Making dovecot-lda setuid-root: +------------------------------- + +Beware: *it's insecure to make dovecot-lda setuid-root*, especially if you have +untrusted users in your system.*Setuid-root dovecot-lda can be used to gain +root privileges*. You should take extra steps to make sure that untrusted users +can't run it and potentially gain root privileges. You can do this by making +sure only your MTA has execution access to it. For example: + +---%<------------------------------------------------------------------------- +# chgrp secmail /usr/local/libexec/dovecot/dovecot-lda +# chmod 04750 /usr/local/libexec/dovecot/dovecot-lda +# ls -l /usr/local/libexec/dovecot/dovecot-lda +-rwsr-x--- 1 root secmail 4023932 2010-06-15 16:23 dovecot-lda +---%<------------------------------------------------------------------------- + +Then start dovecot-lda as a user that belongs to secmail group. Note that you +have to recreate these rights after each update of dovecot. + +Using sudo: +----------- + +Alternatively, you can use sudo to wrap the invocation of dovecot-lda. This has +the advantage that updates will not clobber the setuid bit, but note that *it +is just as insecure being able to run dovecot-lda via sudo as setuid-root*. +Make sure you only give your MTA the ability to invoke dovecot-lda via sudo. + +First configure sudo to allow 'dovelda' user to invoke dovecot-lda by adding +the following to your '/etc/sudoers': + +---%<------------------------------------------------------------------------- +Defaults:dovelda !syslog +dovelda ALL=NOPASSWD:/usr/local/libexec/dovecot/dovecot-lda +---%<------------------------------------------------------------------------- + +Then configure your MTA to invoke dovecot-lda as user 'dovelda' and via sudo: + +---%<------------------------------------------------------------------------- +/usr/bin/sudo /usr/local/libexec/dovecot/dovecot-lda +---%<------------------------------------------------------------------------- + +instead of just plain '/usr/local/libexec/dovecot/dovecot-lda'. + +Problems with dovecot-lda +------------------------- + + * If you are using [UserDatabase.Prefetch.txt], keep in mind + that 'dovecot-lda' does not make a password query and thus will not work if + '-d' parameter is used. The page explains how to + fix this. + * See [PasswordDatabase.CheckPassword.txt] for how to make + dovecot-lda work with checkpassword. + +Logging +------- + + * Normally Dovecot logs everything through its log process, which is running + as root. dovecot-lda doesn't, which means that you might need some special + configuration for it to log anything at all. + * If dovecot-lda fails to write to log files it exits with temporary failure. + * If you have trouble finding where Dovecot logs by default, see + . + * Note that Postfix's 'mailbox_size_limit' setting applies to all files that + are written to. So if you have a limit of 50 MB, dovecot-lda can't write to + log files larger than 50 MB and you'll start getting temporary failures. + +If you want dovecot-lda to keep using Dovecot's the default log files: + + * If you're logging to syslog, make sure the syslog socket (usually + '/dev/log') has enough write permissions for dovecot-lda. For example set it + world-read/writable:'chmod a+rw /dev/log'. + * If you're logging to Dovecot's default log files again you'll need to give + enough write permissions to the log files for dovecot-lda. + +You can also specify different log files for dovecot-lda. This way you don't +have to give any extra write permissions to other log files or the syslog +socket. You can do this by overriding the 'log_path' and 'info_log_path' +settings: + +---%<------------------------------------------------------------------------- +protocol lda { + .. + # remember to give proper permissions for these files as well + log_path = /var/log/dovecot-lda-errors.log + info_log_path = /var/log/dovecot-lda.log +} +---%<------------------------------------------------------------------------- + +For using syslog with dovecot-lda, set the paths empty: + +---%<------------------------------------------------------------------------- +protocol lda { + .. + log_path = + info_log_path = + # You can also override the default syslog_facility: + #syslog_facility = mail +} +---%<------------------------------------------------------------------------- + +Plugins +------- + + * Most of the [Plugins.txt] work with dovecot-lda. + * Virtual quota can be enforced using [Quota.txt]. + * Sieve language support can be added with the + [Pigeonhole.Sieve.txt]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LMTP.Exim.txt b/doc/wiki/LMTP.Exim.txt new file mode 100644 index 0000000000..d1150b2d52 --- /dev/null +++ b/doc/wiki/LMTP.Exim.txt @@ -0,0 +1,153 @@ +Contents + + + 1. Using LMTP over UNIX Socket + + 2. Using LMTP over TCP Socket + + 3. Striping domain to avoid user unknown / doesn't exist error + + 4. Verifying recipients using LMTP + + 5. Delivering mails case insensitively + +Exim provides support for LMTP over UNIX sockets using the LMTP transport +[http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_lmtp_transport.html], +your distribution may/not provide this, run exim -bV and check for 'lmtp' in +'Transports:'. Support for LMTP over TCP sockets is provided by the SMTP +transport +[http://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_smtp_transport.html]. + +Using LMTP over UNIX Socket +--------------------------- + +Use this configuration if dovecot runs on the same host as exim. + +Example router: + +---%<------------------------------------------------------------------------- +local_user: + debug_print = "R: local_user for $local_part@$domain" + driver = accept + domains = +local_domains + check_local_user + transport = dovecot_lmtp + cannot_route_message = Unknown user +---%<------------------------------------------------------------------------- + +Example transport: + +---%<------------------------------------------------------------------------- +dovecot_lmtp: + driver = lmtp + socket = /var/run/dovecot/lmtp + #maximum number of deliveries per batch, default 1 + batch_max = 200 + #allow suffixes/prefixes (default unset) + rcpt_include_affixes +---%<------------------------------------------------------------------------- + +Using LMTP over TCP Socket +-------------------------- + +Example router: + +---%<------------------------------------------------------------------------- +local_user: + transport = dovecot_lmtp + domains = +local_domains + driver = manualroute + route_list = "* 192.168.1.0 byname" + #if destination server is the local host enable this + #self = send +---%<------------------------------------------------------------------------- + +Set IP and port as appropriate to your setup. + +Example transport: + +---%<------------------------------------------------------------------------- +dovecot_lmtp: + driver = smtp + #allow suffixes/prefixes (default unset) + rcpt_include_affixes + protocol = lmtp + port = 2525 +---%<------------------------------------------------------------------------- + +Striping domain to avoid user unknown / doesn't exist error +----------------------------------------------------------- + +If you are using a userdb which does not have domain names, you may need to add +a setting to 20-lmtp.conf + +---%<------------------------------------------------------------------------- +protocol lmtp { + ... + # use %n to strip away the domain part + auth_username_format = %n +} +---%<------------------------------------------------------------------------- + +Symptoms: + + * Exim says something like "LMTP error after RCPT ... 550 ... User doesn't + exist someuser@somedomain" + * Dovecot verbose log says something like "auth-worker(9048): + passwd(someuser@somedomain): unknown user" + +Verifying recipients using LMTP +------------------------------- + +You can use callout verification to avoid accepting mail for addresses which do +not exist in Dovecot. Below is a config snippet which could be used in +acl_smtp_rcpt to achieve this: + +---%<------------------------------------------------------------------------- +deny + message = invalid recipient + domains = +local_domains + !verify = recipient/callout=no_cache +---%<------------------------------------------------------------------------- + +For more information on address verification see the related section of the +Exim specification +[http://www.exim.org/exim-html-current/doc/html/spec_html/ch-access_control_lists.html#SECTaddressverification]. + +Delivering mails case insensitively +----------------------------------- + +*Warning: *Just use this setup if all your login names contain only lower case +characters! (On Linux see /etc/adduser.conf under NAME_REGEX variable). + +Exim retains the case of the local part. Dovecot's LMTP /may/ fail looking up +an incorrect cased local part in your userdb. You can solve this problem by +extending the /protocol lmtp/ section: + +---%<------------------------------------------------------------------------- +protocol lmtp { + ... + # use %Ln to strip away the domain part + auth_username_format = %Lu +} +---%<------------------------------------------------------------------------- + +(If you don't mind allowing case insensitive logins for dovecoth +authentication, you may set /auth_username_format/ in the global configuration +accordingly and renounce the above change). + +In case you prefer to configure exim to lower case the local part instead, add +a router just before your local delivery router: + +---%<------------------------------------------------------------------------- +lowercase_local: + debug_print = "R: lower case local_part for local delivery" + driver = redirect + redirect_router = local_user + data = ${lc:${local_part}} +---%<------------------------------------------------------------------------- + +Make sure to reference the name you have chosen for your local delivery router +within /redirect_router/. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LMTP.txt b/doc/wiki/LMTP.txt new file mode 100644 index 0000000000..ead650af23 --- /dev/null +++ b/doc/wiki/LMTP.txt @@ -0,0 +1,131 @@ +LMTP Server +=========== + +LMTP uses the same settings as , as specified in 'conf.d/15-lda.conf' +in example configuration. There is also a bit of extra configuration in +'conf.d/20-lmtp.conf'. The main difference is that the LDA is a short-running +process, started as a binary from command line, while LMTP is a long-running +process started by Dovecot's master process. + +Envelope Addresses +------------------ + +Compared to dovecot-lda parameters, the addresses are taken from: + + * -f / Envelope sender address: This is the MAIL FROM: value from LMTP + session. + * -r / Final envelope recipient address: This is the RCPT TO: value from LMTP + session. + * -a / Original envelope recipient address: This defaults to same as RCPT TO: + value, but may be overridden by reading it from a header specified by + 'lda_original_recipient_header' setting. + * -d / Destination username: This is the same as RCPT TO: value, but with the + "+extension" part removed when 'recipient_delimiter' setting is set. If + usernames differ from recipient email addresses, the userdb must handle the + translation. + +Listeners +--------- + +You can configure LMTP to be listening on TCP or UNIX sockets: + +---%<------------------------------------------------------------------------- +# add lmtp to protocols, otherwise its listeners are ignored +protocols = imap pop3 lmtp + +service lmtp { + inet_listener lmtp { + address = 192.168.0.24 127.0.0.1 ::1 + port = 24 + } + + unix_listener lmtp { + #mode = 0666 + } +} +---%<------------------------------------------------------------------------- + +The UNIX listener on $base_dir/lmtp is enabled by default when protocols +setting contains lmtp. + +Security +-------- + +Unfortunately LMTP process currently needs to run as root, and only temporarily +drop privileges to users. Otherwise it couldn't handle mail deliveries to more +than a single user with different UID. If you're using only a single global +UID/GID, you can improve security by running lmtp processes as that user: + +---%<------------------------------------------------------------------------- +service lmtp { + user = vmail +} +---%<------------------------------------------------------------------------- + +LMTP Proxying +------------- + +It's possible to use Dovecot LMTP server as a proxy to remote LMTP or SMTP +servers. The configuration is similar to +[PasswordDatabase.ExtraFields.Proxy.txt], but you'll need to tell Dovecot LMTP +to issue passdb lookups: + +---%<------------------------------------------------------------------------- +lmtp_proxy = yes +---%<------------------------------------------------------------------------- + +Performance +----------- + +For higher volume sites, it may be desirable to increase the number of active +listener processes. A range of 5 to 20 is probably good for most sites: + +---%<------------------------------------------------------------------------- +service lmtp { + process_min_avail = 5 +} +---%<------------------------------------------------------------------------- + +Logging +------- + +If you want to store LMTP delivery logs to a different file, you can do it +with: + +---%<------------------------------------------------------------------------- +service lmtp { + executable = lmtp -L +} +protocol lmtp { + info_log_path = /var/log/dovecot-lmtp.log +} +---%<------------------------------------------------------------------------- + +For rawlogs, please see + +Plugins +------- + + * Most of the [Plugins.txt] work with LMTP. + * Virtual quota can be enforced using [Quota.txt]. + * 'lmtp_rcpt_check_quota = yes' enables quota checking already at RCPT TO + stage. This check isn't done for proxied connections. + * Sieve language support can be added with the + [Pigeonhole.Sieve.txt]. + +Address extension delivery +-------------------------- + +To make address extension work with LMTP you must check these variables are set + + * lmtp_save_to_detail_mailbox = yes + * recipient_delimiter = + + +Using LMTP with different MTAs +------------------------------ + + * [HowTo.PostfixDovecotLMTP.txt] + * [LMTP.Exim.txt] + * HALON [http://wiki.halon.se/LMTP] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/Logging.txt b/doc/wiki/Logging.txt new file mode 100644 index 0000000000..ef5dc045f1 --- /dev/null +++ b/doc/wiki/Logging.txt @@ -0,0 +1,181 @@ +Contents + + + 1. Dovecot Logging + + 1. Internal Errors + + 2. Changing Log File Paths + + 3. Syslog Example + + 4. Rotating Logs + + 5. Logging verbosity + +Dovecot Logging +=============== + +*Dovecot always logs a detailed error message* if something goes wrong. If it +doesn't, it's considered a bug and will be fixed. However, almost always the +problem is that *you're looking at the wrong log file*; error messages may be +logged to a different file than informational messages. + +You can find the log file locations by running: + +---%<------------------------------------------------------------------------- +doveadm log find +---%<------------------------------------------------------------------------- + +Dovecot log configuration is found in the conf.d/10-logging.conf file in the +dovecot configuration folder (usually */etc/dovecot* but may also be +*/usr/local/etc/dovecot*). + +By default Dovecot logs to syslog using *mail* facility. You can change the +facility from 'syslog_facility' setting. The syslog configuration is often in +'/etc/syslog.conf' or '/etc/rsyslog*' files. You can also configure Dovecot to +write to log files directly, see below. + +When using syslog, Dovecot uses 5 different logging levels: + + * *debug*: Debug-level message. + * *info*: Informational messages. + * *warning*: Warnings that don't cause an actual error, but are useful to know + about. + * *err*: Non-fatal errors. + * *crit*: Fatal errors that cause the process to die. + +Where exactly these messages are logged depends entirely on your syslog +configuration. Often everything is logged to '/var/log/mail.log' or +'/var/log/maillog', and *err* and *crit* are logged to '/var/log/mail.err'. +This is not necessarily true for your configuration though. + +In an ideal configuration the errors would be logged to a separate file than +non-errors. For example you could set 'syslog_facility = local5' and set: + +---%<------------------------------------------------------------------------- +local5.* -/var/log/dovecot.log +local5.warning;local5.error;local5.crit -/var/log/dovecot-errors.log +---%<------------------------------------------------------------------------- + +Here all the Dovecot messages get logged into 'dovecot.log', while all the +important error/warning messages get logged into 'dovecot-errors.log'. + +Internal Errors +--------------- + +If IMAP or POP3 processes encounter some error, they don't show the exact +reason for clients. Instead they show: + +---%<------------------------------------------------------------------------- +Internal error occurred. Refer to server log for more information. [2006-01-07 +22:35:11] +---%<------------------------------------------------------------------------- + +The point is that whenever anything unexpected happens, Dovecot doesn't leak +any extra information about it to clients. They don't need it and they might +try to exploit it in some ways, so the less they know the better. + +The real error message is written to the error log file. The timestamp is meant +for you to help you find it. + +Changing Log File Paths +----------------------- + +If you don't want to use syslog, or if you just can't find the Dovecot's error +logs, you can make Dovecot log elsewhere as well: + +---%<------------------------------------------------------------------------- +log_path = /var/log/dovecot.log +# If you want everything in one file, just don't specify info_log_path and +debug_log_path +info_log_path = /var/log/dovecot-info.log +# Leave empty in order to send debug-level messages to info_log_path +debug_log_path = /var/log/dovecot-debug.log +---%<------------------------------------------------------------------------- + +The warning and error messages will go to file specified by 'log_path', while +informative messages goes to 'info_log_path' and debug messages goes to +'debug_log_path'. If you do this, make sure you're really looking at the +'log_path' file for error messages, since the "Starting up" message is written +to 'info_log_path' file. + +Syslog Example +-------------- + +Dovecot logging asynchronously via 'syslog_facility = local5' with basic rules: + +---%<------------------------------------------------------------------------- +local5.* -/var/log/dovecot.log +local5.info -/var/log/dovecot.info +local5.warn -/var/log/dovecot.warn +local5.err -/var/log/dovecot.err +:msg,contains,"stored mail into mailbox"\ + -/var/log/dovecot.lmtp +---%<------------------------------------------------------------------------- + +Rotating Logs +------------- + +If you change from syslog to an external log file, you can use logrotate +(available on most recent linux distros) to maintain the Dovecot logfile so it +doesn't grow beyond a manageable size. Save the below scriptlet as +'/etc/logrotate.d/dovecot': + +---%<------------------------------------------------------------------------- +/var/log/dovecot*.log { + weekly + rotate 4 + missingok + notifempty + compress + delaycompress + sharedscripts + postrotate + doveadm log reopen + endscript +} +---%<------------------------------------------------------------------------- + +*Note:* doveadm is not working properly with SELinux (e.g. doveadm cannot read +config file when called from logrotate context). SELinux safe postrotate +alternative scriplet: + +---%<------------------------------------------------------------------------- +postrotate + kill -s 0 `cat /var/run/dovecot/master.pid` || kill -s USR1 `cat +/var/run/dovecot/master.pid` +endscript +---%<------------------------------------------------------------------------- + +*Note 2:* When 'syslog_facility = local5' is used for logging (example above), +the line "/var/log/dovecot.log" should be added to the +'/etc/logrotate.d/syslog' file to enable rotation (no +'/etc/logrotate.d/dovecot' in this case!). + +Logging verbosity +----------------- + +There are several settings that control logging verbosity. By default they're +all disabled, but they may be useful for debugging. + + * 'auth_verbose=yes' enables logging all failed authentication attempts. + * 'auth_debug=yes' enables all authentication debug logging (also enables + 'auth_verbose'). Passwords are logged as ''. + * 'auth_debug_passwords=yes' does everything that 'auth_debug=yes' does, but + it also removes password hiding (but only if you are not using PAM, since + PAM errors aren't written to Dovecot's own logs). + * 'mail_debug=yes' enables all kinds of mail related debug logging, such as + showing where Dovecot is looking for mails. + * 'verbose_ssl=yes' enables logging SSL errors and warnings. Even without this + setting if connection is closed because of an SSL error, the error is logged + as the disconnection reason. + * 'auth_verbose_passwords=no|plain|sha1' If authentication fails, this setting + logs the used password. If you don't really need to know what the password + itself was, but are more interested in knowing if the user is simply trying + to use the wrong password every single time or if it's a brute force attack, + you can set this to "sha1" and only the SHA1 of the password is logged. + That's enough to know if the password is same or different between login + attempts. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/LoginProcess.txt b/doc/wiki/LoginProcess.txt new file mode 100644 index 0000000000..0918e1b42f --- /dev/null +++ b/doc/wiki/LoginProcess.txt @@ -0,0 +1,126 @@ +Login processes +=============== + +The main purpose of login processes is to handle the IMAP, POP3, + (v2.3), and [Pigeonhole.ManageSieve.txt] +connections before the user has logged in. The login processes don't need to be +able to do anything else than let the user log in, so they can run in highly +restricted environment. By default they are run as a non-privileged "dovenull" +user chrooted into a non-writable directory containing only authentication UNIX +sockets. + +Login processes also handle proxying the SSL and TLS connections even after the +user has logged in. This way all the SSL code runs in the same restricted +environment, which means that a security hole in the SSL library gives the +attacker access only to the restricted chroot, rather than possibly all the +users' mails. + +The default login settings should be good enough for small sites. There are two +ways to run the login processes: the high-security mode and the +high-performance mode. Both are discussed separately below. + +For explanation on the various settings for services, see + +High-security mode +------------------ + +You can enable high-security mode with: + +---%<------------------------------------------------------------------------- +service imap-login { + service_count = 1 + #process_min_avail = 0 + #process_limit = $default_process_limit + #vsz_limit = 64M +} +service pop3-login { + service_count = 1 +} +---%<------------------------------------------------------------------------- + +This is the default. It works by using a new imap-login or pop3-login process +for each incoming connection. Since the processes run in a highly restricted +chroot, running each connection in a separate process means that in case there +is a security hole in Dovecot's pre-authentication code or in the SSL library, +the attacker can't see other users' connections and can't really do anything +destructive. The only way out of it is to find and exploit a kernel security +hole. + +Since one login process can handle only one connection, the service's +'process_limit' setting limits the number of users that can be logging in at +the same time (defaults to 'default_process_limit=100'). SSL/TLS proxying +processes are also counted here, so if you're using SSL/TLS you'll need to make +sure this count is higher than the maximum number of users that can be logged +in simultaneously. With TLS/SSL connections, the login process will not +terminate, and remains to perform proxying between imap backend process and the +client. + + * If the maximum login process count is reached, the oldest process in + logging-in state (ie. non-proxying) is destroyed. + * To avoid startup latency for new client connections, set 'process_min_avail' + to higher than zero. That many idling processes are always kept around + waiting for new connections. + * 'vsz_limit' should be fine at its default 64MB value. + +High-performance mode +--------------------- + +You can enable high-performance mode with: + +---%<------------------------------------------------------------------------- +service imap-login { + service_count = 0 + #client_limit = $default_client_limit + process_min_avail = 4 # number of CPU cores + vsz_limit = 1G +} +service pop3-login { + service_count = 0 +} +---%<------------------------------------------------------------------------- + +It works by using a number of long running login processes, each handling a +number of connections. This loses much of the security benefits of the login +process design, because in case of a security hole (in Dovecot or SSL library) +the attacker is now able to see other users logging in and steal their +passwords, read their mails, etc. + + * 'process_min_avail' should be set to be at least the number of CPU cores in + the system, so that all of them will be used. + * Otherwise new processes are created only once an existing one's connection + count reaches client_limit + * Default client_limit * process_limit = 1000*100 = 100k connections + * vsz_limit should be increased to avoid out of memory errors, especially if + you're using SSL/TLS. + +Login access check sockets +-------------------------- + +Dovecot login processes can check via UNIX socket if the incoming connection +should be allowed to log in. This is most importantly implemented to enable TCP +wrappers support for Dovecot. + +TCP wrappers support +-------------------- + +You must have built Dovecot with support for TCP wrappers. You can do this by +giving '--with-libwrap' parameter to 'configure'. + +Add to dovecot.conf: + +---%<------------------------------------------------------------------------- +login_access_sockets = tcpwrap + +service tcpwrap { + unix_listener login/tcpwrap { + group = $default_login_user + mode = 0600 + user = $default_login_user + } +} +---%<------------------------------------------------------------------------- + +Remember to configure your rules! The format is described in hosts.allow(5) and +hosts.deny(5). Files used are usually /etc/hosts.allow and /etc/hosts.deny. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MDA.txt b/doc/wiki/MDA.txt new file mode 100644 index 0000000000..a4a0af3b0f --- /dev/null +++ b/doc/wiki/MDA.txt @@ -0,0 +1,17 @@ +MDA (LDA) +========= + +An MDA is a _M_ail _D_elivery _A_gent. + An LDA is a _L_ocal _D_elivery _A_gent. + These two terms are synonyms. + +An MDA is being passed messages from an and delivers it to a real or +virtual mailbox. + +Common choices include: + + * + * procmail [http://www.procmail.org/] (appears to be unmaintained) + * [LDA.txt] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MTA.txt b/doc/wiki/MTA.txt new file mode 100644 index 0000000000..74722e40f5 --- /dev/null +++ b/doc/wiki/MTA.txt @@ -0,0 +1,54 @@ +MTA +=== + +MTA is an acronym for _M_ail _T_ransport _A_gent. It is the software that works +behind the scenes to transport E-Mail messages from one computer to another. +MUAs (such as mutt, thunderbird, sylpheed, evolution, kmail) hand off newly +sent messages to an MTA. MTAs talk to other MTAs, and either deliver mail +locally or hand it off for delivery to an [MTA.txt] if it was +destined to the local system. + +MTA is a generic term and usually refers to one of these popular software +packages: + + * Postfix [http://www.postfix.org/] is Wietse Venema's secure, fast and + flexible mailer. Default on SUSE Linux and NetBSD. + * Exim [http://www.exim.org/] is Philip Hazel's flexible mailer. Default on + Debian GNU/Linux. + * Sendmail [http://www.sendmail.org/] the original BSD mailer. Default on + FreeBSD; a Sun spinoff is used on Solaris. + * Courier [http://www.courier-mta.org/] was inspired by qmail, but intends to + do things right. + * qmail [http://cr.yp.to/qmail.html] is an obsolete and unmaintained server. + Its POP3 part can be taken over by Dovecot. Qmail started off boasting about + speed and security in the mid-1990s, but has lots of unfixed bugs (this + document includes patches where known), + [http://home.pages.de/~mandree/qmail-bugs.html] among them security bugs + that remain unfixed, and the security guarantee (500 USD) denied. If you + really intend to continue using it, read Dave Sill's Life with qmail + [http://www.lifewithqmail.org/] which contains instructions to work around + some of qmail's security issues. + * HALON [https://halon.io/] is a commercial MTA, which supports Dovecot Auth + and LMTP. It's based on FreeBSD with anti-spam/virus, DKIM, DMARC, DANE, + etc. + +Some people also subsume mail fetching utilities under the MTA category, among +them: + + * fetchmail [http://www.fetchmail.info/] a fast mail retriever for the POP2, + POP3/KPOP/SDPS, IMAP2/IMAP4, ODMR and ETRN protocols, SSL and Kerberos + capable. It forwards the retrieved messages to SMTP/ESMTP, LMTP servers or + into an . Developed out of Carl Harris's popclient by Eric S. + Raymond. Fetchmail is now maintained by and Rob F. + Funk. + * getmail [http://pyropus.ca/software/getmail/] a POP3/SDPS and IMAP4 (with + SSL) enabled mail retrieval utility written in Python. Developed by Chales + Cazabon. + +These mail fetching utilities can be used to store mail for later retrieval by +dovecot. + + Contrast this to the [MDA.txt], sometimes +called local delivery agent (LDA). + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.LocalDisk.txt b/doc/wiki/MailLocation.LocalDisk.txt new file mode 100644 index 0000000000..4bd99fe2a5 --- /dev/null +++ b/doc/wiki/MailLocation.LocalDisk.txt @@ -0,0 +1,50 @@ +Mail storage in local disk +========================== + +Filesystems +----------- + + * See for Maildir-specific filesystem + optimizations + * Dovecot doesn't rely on atime updates, so you can mount the filesystem with + noatime + +Index files +----------- + +Keeping index files on a different disk than the mail spool gives you better +performance. The indexes have a lot of write activity so it is recommended to +use RAID-10 instead of RAID-5 for them. + +Fsyncing +-------- + +By default Dovecot calls fsync() and fdatasync() whenever it's useful to +prevent potential data loss. The main reason for this is so that Dovecot won't +lie that the message was saved to the disk, if in fact a power failure a second +later would lose the message. With IMAP clients this is perhaps a less serious +problem, because the lost message was most likely either a mail in Draft +mailbox or a message in "Sent Messages" mailbox. In any case a message that the +user had already seen. However if loses a message, the user never +even knew that the message existed, unless the sender decides to resend it. + +Since power failures and kernel panics are quite rare, many people are tempted +to disable fsyncing because it may increase the performance quite a lot. +Dovecot allows this by setting 'mail_fsync=never'. However you should consider +leaving it at "optimized" for LDA and LMTP. You can do this with: + +---%<------------------------------------------------------------------------- +# Default to no fsyncing +mail_fsync = never + +protocol lda { + # Enable fsyncing for LDA + mail_fsync = optimized +} +protocol lmtp { + # Enable fsyncing for LMTP + mail_fsync = optimized +} +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.Maildir.txt b/doc/wiki/MailLocation.Maildir.txt new file mode 100644 index 0000000000..4552cf83ee --- /dev/null +++ b/doc/wiki/MailLocation.Maildir.txt @@ -0,0 +1,142 @@ +Maildir configuration +===================== + +See for a complete description of how Dovecot has +implemented Maildir support. + +Mail location +------------- + +Maildir exists almost always in '~/Maildir' directory. The mail location is +specified with: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir +---%<------------------------------------------------------------------------- + +Directory layout +---------------- + +By default Dovecot uses Maildir++ directory layout. This means that all +mailboxes are stored in a single directory and prefixed with a dot. For +example: + + * Maildir/.folder/ + * Maildir/.folder.subfolder/ + +If you want maildirs to use hierarchical directories, such as: + + * Maildir/folder/ + * Maildir/folder/subfolder/ + +you'll need to enable fs layout: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir:LAYOUT=fs +---%<------------------------------------------------------------------------- + +Control files +------------- + +Dovecot stores some Maildir metadata into two control files: + + * 'dovecot-uidlist' file contains IMAP UID <-> Maildir filename mapping + * 'dovecot-keywords' file contains Maildir filename flag (a..z = 0..25) <-> + keyword name mapping + +Both of these files are described fully in . The +important thing to remember about them however is that they shouldn't be +treated the same way as index files. Index files can be deleted and rebuilt +without any side effects, but if you delete control files you'll cause messages +to get new UIDs and possibly lose keyword names. + +If the messages get new UIDs, the IMAP clients will invalidate their local +cache and download the messages all over again. If you do this for all the +users, you could cause huge disk I/O bursts to your server. + +Dovecot can't currently handle not being able to write the control files, so it +will cause problems with [Quota.FS.txt]. To avoid problems +with this, you should place control files into a partition where quota isn't +checked. You can specify this by adding ':CONTROL=' to 'mail_location', +for example: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir:CONTROL=/var/no-quota/%u +---%<------------------------------------------------------------------------- + +Index files +----------- + +See [MailLocation.txt] for full explanation of how to +change the index path. For example: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir:INDEX=/var/indexes/%u +---%<------------------------------------------------------------------------- + +Optimizations +------------- + + * 'maildir_very_dirty_syncs=yes': Assume that only Dovecot accesses 'cur/' + directory. If another process (or even Dovecot which doesn't update index + files) does changes to 'cur/' while the mailbox is simultaneously being + modified by Dovecot, Dovecot may not notice those external changes. It's + still safe to deliver new mails to 'new/' using non-Dovecot software (except + with 'mailbox_list_index=yes' changes aren't noticed outside INBOX). + * 'maildir_copy_with_hardlinks=yes' (default): When copying a message, do it + with hard links whenever possible. This makes the performance much better, + and it's unlikely to have any side effects. Only reason to disable this is + if you're using a filesystem where hard links are slow (e.g. HFS+). + * 'maildir_stat_dirs=no' (default): Assume that all the files beginning with a + dot in the maildir are maildirs. You shouldn't have any non-directory files + beginning with a dot in the maildirs, but if you do you may need to set this + to "yes", in which case Dovecot needs to stat() each directory entry, which + degrades the performance. Some filesystems provide the + directory/non-directory status for free without having to stat(). In those + filesystems this setting is ignored. + +Filesystem optimizations +------------------------ + +See +[MailboxFormat.Maildir.txt]. + +Mailbox directory name +---------------------- + +When using 'LAYOUT=fs', there is a potential for naming collisions between +Maildir's 'new/', 'cur/' and 'tmp/' subdirectories, and mail folders of the +same names. For example, consider a mail folder "foo/bar". Under 'LAYOUT=fs', +data for this mail folder will be stored at under Maildir's usual three +directories '~/Maildir/foo/bar/{new,cur,tmp}/'. + +If the user then tries to create a mail folder "foo/bar/new", this would then +imply that data should be stored in Maildir's three directories +'~/Maildir/foo/bar/new/{new,cur,tmp}/'. But this would overlap Maildir's 'new/' +subdirectory of mail folder "foo/bar". + +This may not be a problem in many installations, but if a risk of collisions +with Maildir's three subdirectory names is perceived, then the 'DIRNAME' +parameter can be used. For example, if we specify mail location as: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir:LAYOUT=fs:DIRNAME=mAildir +---%<------------------------------------------------------------------------- + +then this will push Maildir's 'new/', 'cur/' and 'tmp/' subdirectories down +into a subdirectory 'mAildir/', so a mail folder "foo/bar" would be stored at +'~/Maildir/foo/bar/mAildir/{new,cur,tmp}/'. A mail folder "foo/bar/new" would +be stored at '~/Maildir/foo/bar/new/mAildir/{new,cur,tmp}/', which would then +have no overlap with the mail folder "foo/bar". + +'DIRNAME' affects INBOX slightly differently. Without 'DIRNAME', INBOX will be +stored at '~/Maildir/{new,cur,tmp}/', but when 'DIRNAME' is specified, we get +an extra path component 'INBOX/' immediately prior to the 'DIRNAME' value, so +in the example above INBOX would be stored at +'~/Maildir/INBOX/mAildir/{new,cur,tmp}/'. + +The value for 'DIRNAME' should be chosen carefully so as to minimise the +chances of clashing with mail folder names. In the example here, unusual +upper/lower casing has been used. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.SharedDisk.txt b/doc/wiki/MailLocation.SharedDisk.txt new file mode 100644 index 0000000000..528a663f54 --- /dev/null +++ b/doc/wiki/MailLocation.SharedDisk.txt @@ -0,0 +1,75 @@ +Mail storage on shared disks +============================ + +Dovecot supports keeping mails and index files in clustered filesystems, such +as Red Hat GFS [http://www.redhat.com/gfs/], Oracle OCFS2 +[http://oss.oracle.com/projects/ocfs2/] and HP Polyserve Matrix +[http://h18006.www1.hp.com/storage/software/clusteredfs/index.html]. + +Dovecot also supports keeping mails and index files on NFS. Everything +described in this page applies to NFS as well, but see for more +NFS-specific problems and optimizations. + +Memory mapping +-------------- + +By default Dovecot mmap()s the index files. This may not work with all +clustered filesystems, and it most certainly won't work with NFS. Setting +'mmap_disable = yes' disables mmap() and Dovecot does its own internal caching. +If mmap() is supported by your filesystem, it's still not certain that it gives +better performance. Try benchmarking to make sure. + +Locking +------- + +Dovecot supports locking index files with fcntl (default), flock or dotlocks. +Some clustered filesystems may not support fcntl, so you can change it to use +flock instead. Fcntl locks may also cause problems with some NFS +configurations, in which case you can try if switching to dotlocks helps. Note +that dotlocks are the slowest locking method. You can change the locking method +from 'lock_method' setting. Regardless of the 'lock_method' setting, Dovecot +always uses dotlocks for some locks. + +Clock synchronization +--------------------- + +Run ntpd in each computer to make sure their clocks are synchronized. If the +clocks are more than one second apart from each others and multiple computers +access the same mailbox simultaneously, you may get errors. + +Caching +------- + +Your cluster will probably perform better if users are usually redirected to +the same server. This is because the mailbox may already be cached in the +memory and it may also reduce the traffic between the clusterfs nodes. You can +use [Director.txt] service to do this easily automatically. Or at +the very least make sure that your load balancer redirects connections from the +same IP address to the same server. + +FUSE / GlusterFS +---------------- + +FUSE caches dentries and file attributes internally. If you're using multiple +GlusterFS clients to access the same mailboxes, you're going to have problems. +Worst of these problems can be avoided by using NFS cache flushes, which just +happen to work with FUSE as well: + +---%<------------------------------------------------------------------------- +mail_nfs_index = yes +mail_nfs_storage = yes +---%<------------------------------------------------------------------------- + +These probably don't work perfectly. + +Samba / CIFS +------------ + +Dovecot's temporary files may include a colon character ':' in their filename, +which is not a permitted character when using cifs.Dovecot also renames the +temporary files whilst holding a lock in them, which generates the error 'Text +file is busy'. + +Cifs/smbfs is unlikely to work as a remote filesystem. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.dbox.txt b/doc/wiki/MailLocation.dbox.txt new file mode 100644 index 0000000000..6ced4e3c32 --- /dev/null +++ b/doc/wiki/MailLocation.dbox.txt @@ -0,0 +1,88 @@ +dbox configuration +================== + +See for a description of the dbox mailbox format. +NOTE:*You must not lose the dbox index files, they can't be regenerated without +data loss*. + +Mail location +------------- + +dbox can be used in two ways: + + 1. One message per file (*single-dbox*), similar to + [MailboxFormat.Maildir.txt]. + 2. Multiple messages per file (*multi-dbox*), but unlike + [MailboxFormat.mbox.txt] multiple files per mailbox. + +To use *single-dbox*, use the tag 'sdbox' in the +[MailLocation.txt], for example: + +---%<------------------------------------------------------------------------- +# single-dbox +mail_location = sdbox:~/dbox +---%<------------------------------------------------------------------------- + +For backwards compatibility, 'dbox' is an alias to 'sdbox' in the mail +location. + +To use *multi-dbox*, use the tag 'mdbox' in the +[MailLocation.txt], for example: + +---%<------------------------------------------------------------------------- +# multi-dbox +mail_location = mdbox:~/mdbox +---%<------------------------------------------------------------------------- + +Alternate storage +----------------- + +dbox has a feature for transparently moving message data to an [MailboxFormat.dbox.txt] area. + +To specify an alternate storage area, use the 'ALT' parameter in the mail +location. For example, specifying the mail location as: + +---%<------------------------------------------------------------------------- +mail_location = mdbox:/var/vmail/%d/%n:ALT=/altstorage/vmail/%d/%n +---%<------------------------------------------------------------------------- + +will make Dovecot look for message data first under '/var/vmail/%d/%n', and if +it is not found there it will look under '/altstorage/vmail/%d/%n' instead. + +Keep the unmounted '/altstorage' directory permissions such that Dovecot mail +processes can't create directories under it (e.g. root:root 0755). This way if +the alt storage isn't mounted for some reason, Dovecot won't think that all the +messages in alt storage were deleted and lose their flags. + +Mailbox directory name +---------------------- + +When using the default hierarchical layout, there is a potential for naming +collisions between dbox's 'dbox-Mails/' subdirectory and mail folders of the +same name. For example, consider a mail folder "foo/bar". Under the default +hierarchical layout, data about this mail folder would be stored at +'~/mdbox/mailboxes/foo/bar/dbox-Mails/'. + +If the user then tried to create a mail folder "foo/bar/dbox-Mails", this would +then imply that data would be stored at +'~/mdbox/mailboxes/foo/bar/dbox-Mails/dbox-Mails/'. But this would overlap the +'dbox-Mails/' subdirectory of mail folder "foo/bar". + +This may not be a problem in many installations, but if a risk of collisions +with the default name "dbox-Mails" is perceived, then the 'DIRNAME' parameter +can be used. For example, if we specify mail location as: + +---%<------------------------------------------------------------------------- +mail_location = mdbox:~/mdbox:DIRNAME=DbOx-mAiLs +---%<------------------------------------------------------------------------- + +then this will make data for mail folders be stored in a subdirectory +'DbOx-mAiLs/' instead of the default 'dbox-Mails/', so a mail folder "foo/bar" +would have data stored at '~/mdbox/mailboxes/foo/bar/DbOx-mAiLs/'. + +The value for 'DIRNAME' should be chosen carefully so as to minimise the +chances of clashing with mail folder names. In the example here, unusual +upper/lower casing has been used. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.mbox.txt b/doc/wiki/MailLocation.mbox.txt new file mode 100644 index 0000000000..98d6f73d7f --- /dev/null +++ b/doc/wiki/MailLocation.mbox.txt @@ -0,0 +1,194 @@ +mbox configuration +================== + +See for a complete description of how Dovecot has +implemented mbox support. + +Mail location +------------- + +In many systems the user's mails are by default stored in '/var/mail/username' +file. This file is called INBOX in IMAP world. Since IMAP supports multiple +mailboxes, you'll need to have a directory for them as well. Usually '~/mail' +is a good choice for this. For installation such as this, the mail location is +specified with: + +---%<------------------------------------------------------------------------- +# %u is replaced with the username that logs in +mail_location = mbox:~/mail:INBOX=/var/mail/%u +---%<------------------------------------------------------------------------- + +It's in no way a requirement to have the INBOX in '/var/mail/' directory. In +fact this often just brings problems because Dovecot might not be able to write +dotlock files to the directory (see below). You can avoid this completely by +just keeping everything in '~/mail/': + +---%<------------------------------------------------------------------------- +# INBOX exists in ~/mail/inbox +mail_location = mbox:~/mail +---%<------------------------------------------------------------------------- + +Index files +----------- + +See [MailLocation.txt] for full explanation of how to +change the index path. For example: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=/var/mail/%u:INDEX=/var/indexes/%u +---%<------------------------------------------------------------------------- + +Locking +------- + +Make sure that all software accessing the mboxes are using the same locking +methods in the same order. The order is important to prevent deadlocking. From +Dovecot's side you can change these from 'mbox_read_locks' and +'mbox_write_locks' settings. See for more information. + +/var/mail/ dotlocks +------------------- + +Often mbox write locks include dotlock, which means that Dovecot needs to +create a new ".lock" file to the directory where the mbox file exists. If +your INBOXes are in '/var/mail/' directory you may have to give Dovecot write +access to the directory. There are two ways the '/var/mail/' directory's +permissions have traditionally been set up: + + * World-writable with sticky bit set, allowing anyone to create new files but + not overwrite or delete existing files owned by someone else (ie. same as + /tmp). You can do this with 'chmod a+rwxt /var/mail' + * Directory owned by a mail group and the directory set to group-writable + (mode=0770, group=mail) + +You can give Dovecot access to mail group by setting: + +---%<------------------------------------------------------------------------- +mail_privileged_group = mail +---%<------------------------------------------------------------------------- + +NOTE: With the 'mail_privileged_group' setting unfortunately doesn't +work, so you'll have to use the sticky bit, disable dotlocking completely or +use LMTP server instead. + +/var/mail/* permissions +----------------------- + +In some systems the '/var/mail/$USER' files have 0660 mode permissions. This +causes Dovecot to try to preserve the file's group, and if it doesn't have +permissions to do so, it'll fail with an error: + +---%<------------------------------------------------------------------------- +imap(user): Error: chown(/home/user/mail/.imap/INBOX, -1, 12(mail)) failed: +Operation not permitted (egid=1000(user), group based on /var/mail/user) +---%<------------------------------------------------------------------------- + +There is rarely any real need for the files to have 0660 mode, so the best +solution for this problem is to just change the mode to 0600: + +---%<------------------------------------------------------------------------- +chmod 0600 /var/mail/* +---%<------------------------------------------------------------------------- + +Optimizations +------------- + +The settings below are related to mbox performance. See + for more complete description of what they do. + + * 'mbox_lazy_writes=yes' (default): Metadata updates, such as writing X-UID + headers or flag changes, aren't written to mbox file until the mailbox is + closed or CHECK or EXPUNGE IMAP commands are sent by the client. The mbox + rewrites can be costly, so this may avoid a lot of disk writes. + * 'mbox_dirty_syncs=yes' (default): Dovecot assumes that external mbox file + changes only mean that new messages were appended to it. Without this + setting Dovecot re-reads the whole mbox file whenever it changes. There are + various safeguards in place to make this setting safe even when other + changes than appends were done to the mbox. The only downside to this + setting is that external message flag modifications may not be visible + immediately. + * 'mbox_very_dirty_syncs=yes' (not default): When opening mbox file that has + been changed externally, don't re-read it. Otherwise similar to + 'mbox_dirty_syncs=yes'. + * 'mbox_min_index_size=n': If mbox file is smaller than n kilobytes, don't + update its index files. If an index file exists for it, it's still read + however. + +Only /var/mail/ mboxes +---------------------- + +With POP3 it's been traditional that users have their mails only in the +'/var/mail/' directory. IMAP however supports having multiple mailboxes, so +each user has to have a private directory where the mailboxes are stored. +Dovecot also needs a directory for its index files unless you disable them +completely. + +If you *really* want to use Dovecot as a plain POP3 server without index files, +you can work around the problem of not having the per-user directory: + + * Set users' home directory in userdb to some empty non-writable directory, + for example '/var/empty' + * Modify 'mail_location' setting so that the mail root directory is also the + empty directory and append ':INDEX=MEMORY' to it. For example: + 'mail_location = mbox:/var/empty:INBOX=/var/mail/%u:INDEX=MEMORY' + * Note that if you have IMAP users, they'll see the '/var/empty' as the + directory containing other mailboxes than INBOX. If the directory is + writable, all the users will have their mailboxes shared. + +Directory layout +---------------- + +By default Dovecot uses filesystem layout under mbox. This means that mail is +stored in mbox files under hierarchical directories, for example: + + * '~/mail/inbox' - mbox file containing mail for INBOX + * '~/mail/foo' - mbox file containing mail for mailbox "foo" + * '~/mail/bar/baz' - mbox file containing mail for mailbox "bar/baz" + +One upshot of this is that it is not normally possible to have mailboxes which +are subfolders of mailboxes containing messages. + +As an alternative, it is possible to configure Dovecot to store all mailboxes +in a single directory with hierarchical levels separated by a dot. This can be +configured by adding ':LAYOUT=maildir++' to the mail location. There are, +however, some further considerations when doing this; see + for some examples. + +Control files +------------- + +Under mbox format, Dovecot maintains the subscribed mailboxes list in a file +'.subscriptions' which by default is stored in the mail location root. So in +the example configuration this would be at '~/mail/.subscriptions'. + +If you want to put this somewhere else, you can change the directory in which +the '.subscriptions' file is kept by using the 'CONTROL' parameter. So for +example, if we configured the mail location using: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:CONTROL=~/mail-control +---%<------------------------------------------------------------------------- + +then the subscribed mailboxes list would be maintained at +'~/mail-control/.subscriptions'. + +One practical application of the 'CONTROL' parameter is described at +. + +Message file name +----------------- + +By default, Dovecot stores messages for INBOX in an mbox file called "inbox", +and messages for all other mailboxes in an mbox file whose relative path is +equivalent to the name of the mailbox. Under this scheme, it is not possible to +have mailboxes which contain both messages and child mailboxes. + +However, the behaviour (for mailboxes other than INBOX) can be changed using +the 'DIRNAME' parameter. If the 'DIRNAME' parameter is specified with a +particular value, then Dovecot will store messages in a file with a name of +that value, in a directory with a name equivalent to the mailbox name. + +There are, however, some further considerations when doing this; see + for an example. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailLocation.txt b/doc/wiki/MailLocation.txt new file mode 100644 index 0000000000..eb0942bdd0 --- /dev/null +++ b/doc/wiki/MailLocation.txt @@ -0,0 +1,343 @@ +Mail location +============= + + * For mbox-specific settings, see + * For Maildir-specific settings, see + * For dbox-specific settings, see + +There are three different places where the mail location is looked up from: + + 1. 'mail_location' setting in 'dovecot.conf' is used if nothing else overrides + it. + 2. 'mail' [UserDatabase.txt] overrides 'mail_location' setting. + + 3. 'location' setting inside namespaces overrides everything. Usually this + should be used only for public and shared namespaces. + +Autodetection +------------- + +By default the 'mail_location' setting is empty, which means that Dovecot +attempts to locate automatically where your mails are. This is done by looking, +in order, at: + + * '~/mdbox/' + * '~/sdbox/' + * '~/Maildir/' + * '~/mail/.imap/' + * '~/mail/inbox' + * '~/mail/mbox' + * '~/Mail/.imap/' + * '~/Mail/inbox' + * '~/Mail/mbox' + +For autodetection to work, one of the above locations has to be populated; when +autodetection is active, Dovecot will not attempt to create a mail folder. Note +that '.imap' is a directory, and 'inbox' and 'mbox' are files. + +It's usually a good idea to explicitly specify where the mails are, even if the +autodetection happens to work, in particular to benefit from auto-creation of +the folder for new users. + +Mailbox autocreation +-------------------- + +Dovecot in the 1.x era created mailboxes automatically regardless of whether +mail_location was set. In 2.x autocreation only gets triggered if mail_location +is correctly set. You'll see something like this if you enable debug logging: + +---%<------------------------------------------------------------------------- +Debug: Namespace : /home/user/Mail doesn't exist yet, using default permissions +Debug: Namespace : Using permissions from /home/user/Mail: mode=0700 +gid=default +---%<------------------------------------------------------------------------- + +and a 'Mail/.imap' directory will be present once that process has concluded. +This is the easiest way to ensure a freshly created user is correctly set up +for access via Dovecot. + +Format +------ + +The format of the mailbox location specification is as follows: + + * / [MailboxFormat.txt]/ : /path/ [ : /key/ = /value/ … ] + +where: + + * /mailbox-format/ is a tag identifying one of the formats described at + [MailboxFormat.txt]. + * /path/ is the path to a directory where the mail is stored. This must be an + absolute path, not a relative path. Even if relative paths appear to work, + this usage is deprecated and will likely stop working at some point. Do not + use the home directory, for reasons see + [VirtualUsers.Home.txt] + * /key/ = /value/ can appear zero or more times to set various optional + parameters. Possible values for /key/ are: + * 'INDEX' : specifies the location of [MailLocation.txt]. + * 'ITERINDEX' : Perform mailbox listing using the INDEX directories + instead of the mail root directories. Mainly useful when the INDEX + storage is on a faster storage. It takes no value. (v2.2.32+) + * 'INBOX' : specifies the location of the [MailLocation.txt]. + * 'LAYOUT' : specifies the directory layout to use: + * Maildir++: The default used by Maildir format + * fs: The default used by mbox and dbox formats + * index: Uses mailbox GUIDs as the directory names. The mapping between + mailbox names and GUIDs exists in dovecot.list.index* files. + * 'NO-NOSELECT' : Automatically delete any \ mailboxes that + have no children. These mailboxes are sometimes confusing to users. Also + if a \ mailbox is attempted to be created with "CREATE + box/", it's created as selectable mailbox instead. (LAYOUT=Maildir++ + always behaves this same way.) (v2.2.32+) + * 'UTF-8' : Store mailbox names on disk using UTF-8 instead of modified + UTF-7. + * 'BROKENCHAR' : Specifies an escape character that is used for broken + mailbox names. If mailbox name can't be changed reversibly to UTF-8 and + back, encode the problematic parts using in the + user-visible UTF-8 name. The broken_char itself also has to be encoded + the same way. This can be useful with imapc to access mailbox names that + aren't valid mUTF-7 charset from remote servers. (v2.2.32+) + * 'CONTROL' : Specifies the location of control files under the + [MailLocation.mbox.txt] or [MailLocation.Maildir.txt] formats. + * 'VOLATILEDIR' : Specifies the location of volatile files. This includes + lock files and potentially other files that don't need to exist + permanently. This is especially useful to avoid creating lock files to + NFS or other remote filesystems. (v2.2.32+) + * 'SUBSCRIPTIONS' : specifies the file used for storing subscriptions. The + default is "subscriptions". If you're trying to avoid name collisions + with a mailbox named "subscriptions", then also consider setting + 'MAILBOXDIR'. + * 'MAILBOXDIR' : specifies directory name under which all mailbox + directories are stored. With [MailboxFormat.dbox.txt] the + default is "mailboxes/" while with other mailbox formats the default is + empty. Typically this should be changed only for + [Plugins.Lazyexpunge.txt] with mdbox. + * 'DIRNAME' : specifies the directory name used for mailbox directories, or + in the case of mbox specifies the mailbox message file name. With [MailboxFormat.dbox.txt] the default is "dbox-Mails/" while with + other mailbox formats the default is empty. Can be used under either + [MailLocation.mbox.txt], [MailLocation.Maildir.txt] or + [MailLocation.dbox.txt] formats. Note that this directory is used + only for the mail directory and the alt directory, not for index/control + directories (but see below). + * 'FULLDIRNAME' : Same as 'DIRNAME', but use the directory name also for + index and control directory paths. This should be used instead of + 'DIRNAME' for new installations. (v2.2.8+) + * 'ALT' : specifies the [MailLocation.dbox.txt] path + for dbox formats. + * The colons and equals signs are literal and there are no spaces in an actual + mailbox location specification. + +Variables +--------- + +You can use several variables in the 'mail_location' setting. See + for a full list, but the most commonly used ones are: + + * '%u': Full username. + * '%n': User part in user@domain, same as %u if there's no domain. + * '%d': Domain part in user@domain, empty if there's no domain. + +Typical settings +---------------- + +Typically with Maildir it would be set to: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir +---%<------------------------------------------------------------------------- + +with mbox: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=/var/mail/%u +---%<------------------------------------------------------------------------- + +or if you'd like to use the [MailboxFormat.dbox.txt] format: + +---%<------------------------------------------------------------------------- +# single-dbox +mail_location = sdbox:~/dbox +---%<------------------------------------------------------------------------- + +or: + +---%<------------------------------------------------------------------------- +# multi-dbox +mail_location = mdbox:~/mdbox +---%<------------------------------------------------------------------------- + +Use only absolute paths. Even if relative paths would appear to work, they +might just as well break some day. + +Directory hashing +----------------- + +You can use two different kinds of hashes in [Variables.txt]: + + * %H modifiers returns a 32bit hash of the given string as hex. For example + '%2.256H' would return max. 256 different hashes in range 00 .. ff. + * %M returns a MD5 hash of the string as hex. This can be used for two level + hashing by getting substrings of the MD5 hash. For example '%1Mu/%2.1Mu/%u' + returns directories from '0/0/user' to 'f/f/user'. + +Index files +----------- + +Index files are by default stored under the same directory as mails. With +maildir they are stored in the actual maildirs, with mbox they are stored under +'.imap/' directory. You may want to change the index file location if you're +using or if you're setting up +[SharedMailboxes.txt]. + +You can change the index file location by adding ':INDEX=' to +mail_location. For example: + +---%<------------------------------------------------------------------------- +mail_location = maildir:~/Maildir:INDEX=/var/indexes/%u +---%<------------------------------------------------------------------------- + +The index directories are created automatically, but note that it requires that +Dovecot has actually access to create the directories. Either make sure that +the index root directory ('/var/indexes' in the above example) is writable to +the logged in user, or create the user's directory with proper permissions +before the user logs in. + +If you really want to, you can also disable the index files completely by +appending ':INDEX=MEMORY'. + +Private index files (v2.2+) +--------------------------- + +Since v2.2 the recommended way to enable private flags for shared mailboxes is +to create private indexes with :INDEXPVT=. See + for more information. + +INBOX path +---------- + +INBOX path can be specified to exist elsewhere than the rest of the mailboxes, +for example: + +---%<------------------------------------------------------------------------- +mail_location = mbox:~/mail:INBOX=/var/mail/%u +mail_location = maildir:~/Maildir:INBOX=~/Maildir/.INBOX +---%<------------------------------------------------------------------------- + +Note that it's still not possible to mix maildir and mbox formats this way. You +need to use [Namespaces.txt] for that. + +Homeless users +-------------- + +Having a home directory for users is highly recommended. The + [Pigeonhole.Sieve.txt] already requires a home directory to +work, and it probably won't be the last feature to require a home. See + [VirtualUsers.txt] for more reasons why it's a good +idea, and how to give Dovecot a home directory even if you don't have a "real +home directory". + +If you really don't want to set any home directory, you can use something like: + +---%<------------------------------------------------------------------------- +mail_location = maildir:/home/%u/Maildir +---%<------------------------------------------------------------------------- + +Per-user mail locations +----------------------- + +It's possible to override the default 'mail_location' for specific users by +making the [UserDatabase.txt] return 'mail' +[UserDatabase.ExtraFields.txt]. See the [UserDatabase.txt] page +for the specific userdb you're using for more information how to do this. +Below are however a couple of examples. + +Note that %h doesn't work in the userdb queries or templates. ~/ gets expanded +later, so use it instead. + +Note also that since 'location' specified within a [Namespaces.txt] +overrides mail_location setting, in case you specified that parameter, you'll +have to override in in the user database, specifying 'namespace/inbox/location' +extra field instead of 'mail'. + +SQL +--- + +---%<------------------------------------------------------------------------- +user_query = SELECT home, uid, gid, mail FROM users WHERE user = '%u' +---%<------------------------------------------------------------------------- + +LDAP +---- + +---%<------------------------------------------------------------------------- +user_attrs = homeDirectory=home, uidNumber=uid, gidNumber=gid, +mailLocation=mail +---%<------------------------------------------------------------------------- + +Passwd-file +----------- + +---%<------------------------------------------------------------------------- +user:{PLAIN}password:1000:1000::/home/user::userdb_mail=mbox:~/mail:INBOX=/var/mail/%u +---%<------------------------------------------------------------------------- + +Mixing mbox and maildir +----------------------- + +It's possible to use both mboxes and maildirs for the same user by configuring +multiple namespaces. See . + +Having both mboxes and maildirs mixed within the same namespace isn't currently +supported. + +Custom mailbox location detection +--------------------------------- + +Dovecot by default detects the mailboxes in this order: + + 1. maildir: ~/Maildir + 2. mbox: ~/mail, and /var/mail/%u if it exists + 3. mbox: ~/Mail, and /var/mail/%u if it exists + +If you need something else, you can override the 'mail_executable' setting to +run a script, which sets the MAIL environment properly. For example: + +---%<------------------------------------------------------------------------- +#!/bin/sh + +if [ -d $HOME/.maildir ]; then + export MAIL=maildir:$HOME/.maildir +else + export MAIL=mbox:$HOME/mail:INBOX=/var/mail/$USER +fi +export USERDB_KEYS="$USERDB_KEYS mail" + +exec "$@" +---%<------------------------------------------------------------------------- + +Custom namespace location +------------------------- + +If you need to override namespace's location, first give it a name ("inbox" +below): + +---%<------------------------------------------------------------------------- +namespace inbox { + .. +} +---%<------------------------------------------------------------------------- + +Then in the script use: + +---%<------------------------------------------------------------------------- +#!/bin/sh + +# do the lookup here +location=mbox:$HOME/mail + +export USERDB_KEYS="$USERDB_KEYS namespace/inbox/location" +exec env "NAMESPACE/INBOX/LOCATION=$location" "$@" +---%<------------------------------------------------------------------------- + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.Cydir.txt b/doc/wiki/MailboxFormat.Cydir.txt new file mode 100644 index 0000000000..9882bbfb90 --- /dev/null +++ b/doc/wiki/MailboxFormat.Cydir.txt @@ -0,0 +1,20 @@ +Cydir +===== + +This mailbox format is very similar to Cyrus IMAP's internal mail store: + + * Messages are stored in "." named files + * Cyrus's 'cyrus.index' is equivalent to Dovecot's 'dovecot.index'. Dovecot + however also requires 'dovecot.index.log' for its indexing to work. + * Cyrus's 'cyrus.cache' is equivalent to Dovecot's 'dovecot.index.cache'. + +Cydir is a very simple format internally. It relies on Dovecot's +[IndexFiles.txt] completely for its functionality. If the index files are lost, +all the message flags are lost. Currently the code can't even rebuild index +files if they're lost. + +Cydir is mostly meant to be used for benchmarking and stress testing index +handling code. Its code is small and simple, so it can also act as an example +for writing new mail storage backends. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.MH.txt b/doc/wiki/MailboxFormat.MH.txt new file mode 100644 index 0000000000..cbe7190ed8 --- /dev/null +++ b/doc/wiki/MailboxFormat.MH.txt @@ -0,0 +1,52 @@ +MH Mailbox Format +================= + +The MH mailbox format originated with a system developed by the RAND +corporation and the University of California. Each email message is stored in +a single file, with directories indicating folders and subfolders. The index +or order of the messages in the folder determine what each message is named +(which may not correspond to the inode index). The "safe" way to guarantee a +message gets written to a mail folder is to first write the message out to a +randomly chosen temporary file name, then link the file to the number LAST+1, +where LAST is the last sequential message in the folder. If the link fails, +increment the counter and try again. + +MH folders also maintains a meta-file called '~/Mail/.mh_context' that contains +information about the most current folder and message chosen. Each sub-folder +also contains a meta-file called '.mh_sequences' or '.xmhcache', which +maintains keyword association lists for stored queries. New messages are +stored in the "unseen" sequence for a folder. Procmail itself does not bother +making changes to this file, rather simply delivers the message to the folder +and leaves determining new messages as an exercise for the MUA. For example: + +---%<------------------------------------------------------------------------- +unseen: 1-3 8 15 +projectB: 2-8 10 +---%<------------------------------------------------------------------------- + +shows two stored sequences of messages. Command-line utilities can then use +these sequences as shortcuts.'show unseen', for example, is short-hand for +'show 1-3 8 15'. + +Deleted emails are indicated by prepending a "," to the name. One of the +largest problems that IMAP servers have with MH format is the volatility of the +email message name itself. The command-line utility 'sortm' is used to sort +mail folders by date or string matching. To do this, messages are actually +renamed to reflect the new sort order. IMAP servers are required to maintain +an index of the folder contents, so when the names of the file entries cannot +be guaranteed to be stable, IMAP servers have to throw out previous index +caches and re-index. + +When operating with a shell account on a machine that also provides IMAP access +to folders, users are encouraged not to re-sort email locally while accessing +the IMAP server remotely. + +Links + + * NMH [http://www.nongnu.org/nmh/]: New MH Client + * Original RAND MH [http://rand-mh.sourceforge.net] Code + * MH-Book [http://rand-mh.sourceforge.net/book/] + * Mutt Manual [http://www.mutt.org/doc/manual/manual-4.html#ss4.6]: Describing + how it handles MH folders + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.Maildir.txt b/doc/wiki/MailboxFormat.Maildir.txt new file mode 100644 index 0000000000..5629b7dce8 --- /dev/null +++ b/doc/wiki/MailboxFormat.Maildir.txt @@ -0,0 +1,400 @@ +Maildir +======= + +This format debuted with the qmail server in the mid-1990s. Each mailbox folder +is a directory and each message a file. This improves efficiency because +individual emails can be modified, deleted and added without affecting the +mailbox or other emails, and makes it safer to use on networked file systems +such as NFS. + +Dovecot extensions +------------------ + +Since the standard maildir specification [https://cr.yp.to/proto/maildir.html] +doesn't provide everything needed to fully support the IMAP protocol, Dovecot +had to create some of its own non-standard extensions. The extensions still +keep the maildir standards compliant, so MUAs not supporting the extensions can +still safely use it as a normal maildir. + +IMAP UID mapping +---------------- + +IMAP requires each message to have a permanent unique ID number. Dovecot uses +'dovecot-uidlist' file to keep UID <-> filename mapping. The file is basically +in the same format as Courier IMAP's courierimapuiddb file, except for one +difference (see below). + +The file begins with a header: + +---%<------------------------------------------------------------------------- +3 V1275660208 N25022 G3085f01b7f11094c501100008c4a11c1 +---%<------------------------------------------------------------------------- + + * 3 is the file format version number used by Dovecot v1.1+ + * 1275660208 is the IMAP UIDVALIDITY + * 25022 is the UID that will be given to the next added message + * 3085f01b7f11094c501100008c4a11c1 is the 128 bit mailbox global UID in hex + * There may be other fields, and the order of these fields isn't important + +Version 1 file format is compatible with Courier. Version 2 was used by a few +non-release versions. + +After the header comes the list of UID <-> filename mappings: + +---%<------------------------------------------------------------------------- +25006 :1276528487.M364837P9451.kurkku,S=1355,W=1394:2, +25017 W2481 :1276533073.M242911P3632.kurkku:2,F +---%<------------------------------------------------------------------------- + + * 25006, 25017 are message UIDs + * 2481 is the second message's virtual size. First message contains it in the + filename itself, so it's not duplicated. + * There may be more fields before ':' character + * Rest of the line after ':' is the last known filename. This filename doesn't + necessarily exist currently, because filename changes every time message's + flags change. Dovecot doesn't waste disk I/O by rewriting uidlist file every + time flags change, but whenever it is rewritten the latest filenames are + used. This allows Dovecot to try to guess what the message's current + filename is and if successful, avoid having to scan the directory's + contents. + +The dovecot-uidlist file doesn't need to be locked for reading. When writing +dovecot-uidlist.lock file needs to be created. New lines can be appended to the +end of file, but existing data must never be directly modified, it can only be +replaced with rename() call. + +dovecot-uidlist is updated lazily to optimize for disk I/O. If a message is +expunged, it may not be removed from dovecot-uidlist until sometimes later. +This means that if you create a new file using the same file name as what +already exists in dovecot-uidlist, Dovecot thinks you "unexpunged" message by +restoring a message from backup. This causes a warning to be logged and the +file to be renamed. + +Note that messages must not be modified once they've been delivered. IMAP (and +Dovecot) requires that messages are immutable. If you wish to modify them in +any way, create a new message instead and expunge the old one. + +IMAP keywords +------------- + +All the non-standard message flags are called keywords in IMAP. Some clients +use these automatically for marking spam (eg. $Junk, $NonJunk, $Spam, $NonSpam +keywords). Thunderbird uses labels which map to keywords $Label1, $Label2, etc. + +Dovecot stores keywords in the maildir filename's flags field using letters +a..z. This means that only 26 keywords are possible to store in the maildir. If +more are used, they're still stored in Dovecot's index files. The mapping from +single letters to keyword names is stored in dovecot-keywords file. The file is +in format: + +---%<------------------------------------------------------------------------- +0 $Junk +1 $NonJunk +---%<------------------------------------------------------------------------- + +0 means letter 'a' in the maildir filename, 1 means 'b' and so on. The file +doesn't need to be locked for reading, but when writing dovecot-uidlist file +must be locked. The file must not be directly modified, it can only be replaced +with rename() call. + +For example, a file named + +---%<------------------------------------------------------------------------- +1234567890.M20046P2137.mailserver,S=4542,W=4642:2,Sb +---%<------------------------------------------------------------------------- + +would be flagged as '$NonJunk' with the above keywords. + +Maildir filename extensions +--------------------------- + +The standard filename definition is: ":2,". Dovecot has +extended the field to be "[,]". This means +that if Dovecot sees a comma in the field while updating flags in the +filename, it doesn't touch anything after the comma. However other maildir MUAs +may mess them up, so it's still not such a good idea to do that. Basic +are described here [https://cr.yp.to/proto/maildir.html]. The isn't used by Dovecot for anything currently. + +Dovecot supports reading a few fields from the : + + * ',S=': contains the file size. Getting the size from the + filename avoids doing a stat(), which may improve the performance. This is + especially useful with [Quota.Maildir.txt]. + * ',W=': contains the file's RFC822.SIZE, ie. the file size + with linefeeds being CR+LF characters. If the message was stored with CR+LF + linefeeds, and are the same. Setting this may give a small + speedup because now Dovecot doesn't need to calculate the size itself. + +A maildir filename with those fields would look something like: +'1035478339.27041_118.foo.org,S=1000,W=1030:2,S' + +Usage of timestamps +------------------- + +Timestamps of message files: + + * 'mtime' is used as IMAP INTERNALDATE, RFC 3501 sec 2.3.3. + [https://www.faqs.org/rfcs/rfc3501.html], must never change, see sec. + 2.3.1.1. 4) + * 'ctime' used as Dovecot's internal "save/copy date", unless the correct + value is found from 'dovecot.index.cache'. This is used only by external + commands, e.g. "doveadm expunge savedbefore". + * 'atime' not used + +Timestamps of 'cur' and 'new' directories: + + * 'mtime' is used to detect changes of the mailbox and may force regeneration + of [IndexFiles.txt] + * 'atime' and 'ctime' not used + +Filename examples +----------------- + ++---------------------------------------------------------------------------------------------+----------------------------+ +| Filename | Explanation | ++---------------------------------------------------------------------------------------------+----------------------------+ +| *1491941793*.M41850P8566V0000000000000015I0000000004F3030E_0.mx1.example.com,S=10956:2,STln | UNIX timestamp of arrival | ++---------------------------------------------------------------------------------------------+----------------------------+ +| 1491941793.M41850P8566V0000000000000015I0000000004F3030E_0.mx1.example.com,*S=10956*:2,STln | Size of e-mail | ++---------------------------------------------------------------------------------------------+----------------------------+ +| 1491941793.M41850P8566V0000000000000015I0000000004F3030E_0.mx1.example.com,S=10956:2,*STln* | *S*=seen (marked as read) | +| | *T*=trashed | +| | *l*=IMAP tag #12 (0=a, 1=b,| +| | 2=c, etc) as defined in | +| | that folder's | +| | dovecot-keywords. | +| | *n*=IMAP tag #14 (0=a, 1=b,| +| | 2=c, etc) as defined in | +| | that folder's | +| | dovecot-keywords. | ++---------------------------------------------------------------------------------------------+----------------------------+ + +Maildir and filesystems +----------------------- + +General comparisons of Maildir on different filesystems +------------------------------------------------------- + + * https://www.thesmbexchange.com/eng/qmail_fs_benchmark.html + * https://www.htiweb.inf.br/benchmark/fsbench.htm (including some graphs) + +Linux ext2 / ext3 +----------------- + +The main disadvantage is that searching can be slightly slower, and access to +very large mailboxes (thousands of messages) can get slow with filesystems +which don't have directory indexes. + +Old versions of ext2 and ext3 on Linux don't support directory indexing (to +speed up access), but newer versions of ext3 do, although you may have to +manually enable it. You can check if the indexing is already enabled with +tune2fs: + +---%<------------------------------------------------------------------------- +tune2fs -l /dev/hda3 | grep features +---%<------------------------------------------------------------------------- + +If you see dir_index, you're all set. If dir_index is missing, add it using: + +---%<------------------------------------------------------------------------- +umount /dev/hda3 +tune2fs -O dir_index /dev/hda3 +e2fsck -fD /dev/hda3 +mount /dev/hda3 +---%<------------------------------------------------------------------------- + +ReiserFS +-------- + +ReiserFS was built to be fast with lots of small files, so it works well with +maildir. + +XFS +--- + +XFS performance seems to depend on a lot of factors, also on the system and the +file system parameters. + + * There are early reports on the dovecot mailing list which suggest that XFS + seems quite a lot slower than ext3 or + ReiserFS:https://dovecot.org/list/dovecot/2007-January/018994.html + * But then again others recommend XFS for the use with Maildir and dovecot: + https://dovecot.org/list/dovecot/2006-May/013216.html + * This 2007 Linux.conf.au talk about "Choosing and Tuning Linux File Systems" + (Slides as PDF) + [https://mirror.linux.org.au/pub/linux.conf.au/2007/video/talks/348.pdf] + also recommends XFS for Maildir (alternatively ext3 with small blocks and + high inodetofile ratio) + * Someone else wrote here in the wiki: XFS on TSL 3.0.5 works almost twice as + fast as our prior EXT3 installation of which is significant in size. + ReiserFS is also a good option. + * Comparisons which suggest XFS as being best choice: + * https://www.thesmbexchange.com/eng/qmail_fs_benchmark.html + * https://www.htiweb.inf.br/benchmark/fsbench.htm + +Various tips +------------ + + * Mounting XFS with 'logbufs=8' option might increase the speed. + * Create the XFS with options '-b size=1024 -d su=16k,sw=3 -l + logdev=' (Source: + https://www.thesmbexchange.com/eng/qmail_fs_benchmark.html) + * Use 'mkfs.xfs -f -l size=32768b,version=2' and 'mount.xfs -o + noatime,logbufs=8,logbsize=131072' (Source: + https://www.htiweb.inf.br/benchmark/fsbench.htm) + +NFS +--- + +NFS v3 performance can be adversely affected by readdirplus, which causes the +NFS server to stat() every file in a directory. The solution under Linux is to +make sure the NFS filesystem is mounted with the "nordirplus" option. + + * https://dovecot.org/list/dovecot/2012-July/066939.html + +Directory Structure +------------------- + +By default Dovecot uses Maildir++ +[https://www.courier-mta.org/imap/README.maildirquota.html] directory layout +for organizing mailbox directories. This means that all the folders are +directly inside '~/Maildir' directory: + + * '~/Maildir/new', '~/Maildir/cur' and '~/Maildir/tmp' directories contain the + messages for INBOX. The 'tmp' directory is used during delivery, new + messages arrive in 'new' and read shall be moved to 'cur' by the clients. + * '~/Maildir/.folder/' is a mailbox folder + * '~/Maildir/.folder.subfolder/' is a subfolder of a folder (ie. + "folder/subfolder") + +You can also optionally use the "fs" layout by appending ':LAYOUT=fs' to + [MailLocation.txt]. This makes the folder structure look like: + + * '~/Maildir/new', '~/Maildir/cur' and '~/Maildir/tmp' directories contain the + messages for INBOX, just like with Maildir++. + * '~/Maildir/folder/' is a mailbox folder + * '~/Maildir/folder/subfolder/' is a subfolder of a folder + +Filesystem permissions +---------------------- + +See for how permissions are set for newly +created files and directories. + +Since Dovecot v2.0 "Permissions for newly created mail files are no longer +copied from dovecot-shared file", see . + +Issues with the specification +----------------------------- + +Locking +------- + +Although maildir was designed to be lockless, Dovecot locks the maildir while +doing modifications to it or while looking for new messages in it. This is +required because otherwise Dovecot might temporarily see mails incorrectly +deleted, which would cause trouble. Basically the problem is that if one +process modifies the maildir (eg. a rename() to change a message's flag), +another process in the middle of listing files at the same time could skip a +file. The skipping happens because readdir() system call doesn't guarantee that +all the files are returned if the directory is modified between the calls to +it. This problem exists with all the commonly used filesystems. + +Because Dovecot uses its own non-standard locking ('dovecot-uidlist.lock' +dotlock file), other MUAs accessing the maildir don't support it. This means +that if another MUA is updating messages' flags or expunging messages, Dovecot +might temporarily lose some message. After the next sync when it finds it +again, an error message may be written to log and the message will receive a +new UID. + +Delivering mails to new/ directory doesn't have any problems, so there's no +need for LDAs to support any type of locking. + +Mail delivery +------------- + +Qmail's how a message is delivered page +[https://www.qmail.org/man/man5/maildir.html] suggests to deliver the mail like +this: + + 1. Create a unique filename (only "time.pid.host" here, later Maildir spec has + been updated to allow more uniqueness identifiers) + 2. Do 'stat(tmp/)'. If the 'stat()' found a file, wait 2 seconds and + go back to step 1. + 3. Create and write the message to the 'tmp/'. + 4. link() it into new/ directory. Although not mentioned here, the link() + could again fail if the mail existed in new/ dir. In that case you should + probably go back to step 1. + +All this trouble is rather pointless. Only the first step is what really +guarantees that the mails won't get overwritten, the rest just sounds nice. +Even though they might catch a problem once in a while, they give no guaranteed +protection and will just as easily pass duplicate filenames through and +overwrite existing mails. + +Step 2 is pointless because there's a race condition between steps 2 and 3. +PID/host combination by itself should already guarantee that it never finds +such a file. If it does, something's broken and the stat() check won't help +since another process might be doing the same thing at the same time, and you +end up writing to the same file in tmp/, causing the mail to get corrupted. + +In step 4 the link() would fail if an identical file already existed in the +maildir, right? Wrong. The file may already have been moved to cur/ directory, +and since it may contain any number of flags by then you can't check with a +simple stat() anymore if it exists or not. + +Step 2 was pointed out to be useful if clock had moved backwards. However again +this doesn't give any actual safety guarantees, because an identical base +filename could already exist in cur/. Besides if the system was just rebooted, +the file in tmp/ could probably be even overwritten safely (assuming it wasn't +already link()ed to new/). + +So really, all that's important in not getting mails overwritten in your +maildir is the step 1: Always create filenames that are guaranteed to be +unique. Forget about the 2 second waits and such that the Qmail's man page +talks about. + +Maildir and mail header metadata +-------------------------------- + +Unlike when using [MailboxFormat.mbox.txt] as +[MailboxFormat.txt], where mail headers (for example 'Status', 'X-UID', etc.) +are [MailboxFormat.mbox.txt], the mail +headers within maildir files are (usually)*not* used for this purpose by +dovecot; neither when mails are created/moved/etc. via IMAP nor when maildirs +are placed (e.g. copied or moved in the filesystem) in a mail location (and +then "imported" by dovecot).Therefore, it is (usually) *not* necessary, to +strip any such mail headers at the MTA, MDA or LDA (as it is recommended with + [MailboxFormat.mbox.txt]). + +There is one exception, though, namely when 'pop3_reuse_xuidl=yes' (which is +however rather deprecated):In this case 'X-UIDL' is used for the POP3 UIDLs. +Therefore,*in this case, is recommended to strip the 'X-UIDL' mail headers +_case-insensitively_ at the MTA, MDA or LDA*. + +Procmail Problems +----------------- + +Maildir format is somewhat compatible with MH format. This is sometimes a +problem when people configure their procmail to deliver mails to 'Maildir/new'. +This makes procmail create the messages in MH format, which basically means +that the file is called 'msg.inode_number'. While this appears to work first, +after expunging messages from the maildir the inodes are freed and will be +reused later. This means that another file with the same name may come to the +maildir, which makes Dovecot think that an expunged file reappeared into the +mailbox and an error is logged. + +The proper way to configure procmail to deliver to a Maildir is to use +'Maildir/' as the destination. + +References +---------- + + * Official Maildir format page [https://cr.yp.to/proto/maildir.html] + * Qmail's how to deliver to Maildir man page + [https://www.qmail.org/man/man5/maildir.html] + * Maildir++ [https://www.courier-mta.org/imap/README.maildirquota.html] + * Wikipedia [https://en.wikipedia.org/wiki/Maildir] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.dbox.txt b/doc/wiki/MailboxFormat.dbox.txt new file mode 100644 index 0000000000..8df69bf124 --- /dev/null +++ b/doc/wiki/MailboxFormat.dbox.txt @@ -0,0 +1,251 @@ +dbox +==== + +dbox is Dovecot's own high-performance mailbox format. The original version was +introduced in v1.0 alpha4, but since then it has been completely redesigned in +v1.1 series and improved even further in v2.0. + +dbox can be used in two ways: + + 1. *single-dbox* ('sdbox' in [MailLocation.txt]): One message + per file, similar to [MailboxFormat.Maildir.txt]. For backwards + compatibility,'dbox' is an alias to 'sdbox' in + [MailLocation.txt]. + 2. *multi-dbox* ('mdbox' in [MailLocation.txt]): Multiple + messages per file, but unlike [MailboxFormat.mbox.txt] multiple + files per mailbox. + +One of the main reasons for dbox's high performance is that it uses Dovecot's +index files as the only storage for message flags and keywords, so the indexes +don't have to be "synchronized". Dovecot trusts that they're always up-to-date +(unless it sees that something is clearly broken). This also means that *you +must not lose the dbox index files, they can't be regenerated without data +loss*. + +dbox has a feature for transparently moving message data to an alternate +storage area. See [MailboxFormat.dbox.txt] below. + +dbox storage is extensible. Single instance attachment storage was already +implemented as such extension. + +Layout +------ + +By default, the dbox filesystem layout will be as follows. Data which isn't the +actual message content is stored in a layout common to both *single-dbox* and +*multi-dbox*: + + * '/mailboxes/INBOX/dbox-Mails/dovecot.index*' - Index + files for INBOX + * '/mailboxes/foo/dbox-Mails/dovecot.index*' - Index files + for mailbox "foo" + * '/mailboxes/foo/bar/dbox-Mails/dovecot.index*' - Index + files for mailbox "foo/bar" + * '/dovecot.mailbox.log*' - Mailbox changelog + * '/subscriptions' - subscribed mailboxes list + * '/dovecot-uidvalidity*' - IMAP UID validity + +Note that with dbox the Index files actually contain significant data which is +held nowhere else. Index files for both *single-dbox* and *multi-dbox* contain +message flags and keywords. For *multi-dbox*, the index file also contains the +map_uids which link (via the "map index") to the actual message data. This data +cannot be automatically recreated, so it is important that Index files are +treated with the same care as message data files. + +Index files can be stored in a different location by using the INDEX parameter +in the mail location specification. If the INDEX parameter is specified, it +will make Dovecot look for the Index files as follows: + + * '/mailboxes/INBOX/dbox-Mails/dovecot.index*' - Index files + for INBOX + * '/mailboxes/foo/dbox-Mails/dovecot.index*' - Index files for + mailbox "foo" + * '/mailboxes/foo/bar/dbox-Mails/dovecot.index*' - Index files + for mailbox "foo/bar" + +Actual message content is stored differently depending on whether it is +*single-dbox* or *multi-dbox*. + +Under *single-dbox* we have: + + * '/mailboxes/INBOX/dbox-Mails/u.*' - Numbered files + ('u.1','u.2', ...) each containing one message of INBOX + * '/mailboxes/foo/dbox-Mails/u.*' - Files each containing + one message for mailbox "foo" + * '/mailboxes/foo/bar/dbox-Mails/u.*' - Files each + containing one message for for mailbox "foo/bar" + +Under *multi-dbox* we have: + + * '/storage/dovecot.map.index*' - "Map index" containing a + record for each message stored + * '/storage/m.*' - Numbered files ('m.1', 'm.2', ...) each + containing one or multiple messages + +With Dovecot versions 2.0.4 and later, setting the INDEX parameter sets the +location of the "map index" as well as the location of the mailbox indexes. So +this would make the "map index" be stored as follows: + + * '/storage/dovecot.map.index*' - "Map index" containing a + record for each message stored + +Multi-dbox +---------- + +You can enable multi-dbox with: + +---%<------------------------------------------------------------------------- +mail_location = mdbox:~/mdbox +---%<------------------------------------------------------------------------- + +The directory layout (under '~/mdbox/') is: + + * '~/mdbox/storage/' contains the actual mail data for all mailboxes + * '~/mdbox/mailboxes/' contains directories for mailboxes and their index + files + +The storage directory has files: + + * 'dovecot.map.index*' files contain the "map index" + * 'm.*' files contain the mail data + +Each m.* file contains one or more messages. 'mdbox_rotate_size' setting can be +used to configure how large the files can grow. + +The map index contains a record for each message: + + * map_uid: Unique growing 32 bit number for the message. + * refcount: 16 bit reference counter for this message. Each time the message + is copied the refcount is increased. + * file_id: File number containing the message. For example if file_id=5, the + message is in file 'm.5'. + * offset: Offset to message within the file. + * size: Space used by the message in the file, including all metadata. + +Mailbox indexes refer to messages only using map_uids. This allows messages to +be moved to different files by updating only the map index. Copying is done +simply by appending a new record to mailbox index containing the existing +map_uid and increasing its refcount. If refcount grows over 32768, currently +Dovecot gives an error message. It's unlikely anyone really wants to copy the +same message that many times. + +Expunging a message only decreases the message's refcount. The space is later +freed in "purge" step. This is typically done in a nightly cronjob when there's +less disk I/O activity. The purging first finds all files that have refcount=0 +mails. Then it goes through each file and copies the refcount>0 mails to other +mdbox files (to the same files as where newly saved messages would also go), +updates the map index and finally deletes the original file. So there is never +any overwriting or file truncation. + +The purging can be invoked explicitly running +[Tools.Doveadm.Purge.txt]. + +There are several safety features built into dbox to avoid losing messages or +their state if map index or mailbox index gets corrupted: + + * Each message has a 128 bit globally unique identifier (GUID). The GUID is + saved to message metadata in m.* files and also to mailbox indexes. This + allows Dovecot to find messages even if map index gets corrupted. + * Whenever index file is rewritten, the old index is renamed to + 'dovecot.index.backup'. If the main index becomes corrupted, this backup + index is used to restore flags and figure out what messages belong to the + mailbox. + * Initial mailbox where message was saved to is stored in the message metadata + in m.* files. So if all indexes get lost, the messages are put to their + initial mailboxes. This is better than placing everything into a single + mailbox. + +Alternate storage +----------------- + +Unlike Maildir, with dbox the message file names don't change. This makes it +possible to support storing files in multiple directories or mount points. dbox +supports looking up files from "altpath" if they're not found from the primary +path. This means that it's possible to move older mails that are rarely +accessed to cheaper (slower) storage. + +To enable this functionality, use the 'ALT' parameter in the mail location. For +example, specifying the mail location as: + +---%<------------------------------------------------------------------------- +mail_location = mdbox:/var/vmail/%d/%n:ALT=/altstorage/vmail/%d/%n +---%<------------------------------------------------------------------------- + +will make Dovecot look for message data first under '/var/vmail/%d/%n' +("primary storage"), and if it is not found there it will look under +'/altstorage/vmail/%d/%n' ("alternate storage") instead. There's no problem +having the same (identical) file in both storages. + +Keep the unmounted '/altstorage' directory permissions such that Dovecot mail +processes can't create directories under it (e.g. root:root 0755). This way if +the alt storage isn't mounted for some reason, Dovecot won't think that all the +messages in alt storage were deleted and lose their flags. + +When messages are moved from primary storage to alternate storage, only the +actual message data (stored in files 'u.*' under *single-dbox* and 'm.*' under +*multi-dbox*) is moved to alternate storage; everything else remains in the +primary storage. + +Message data can be moved from primary storage to alternate storage using + [Tools.Doveadm.Altmove.txt]. (In theory you could also do +this with some combination of cp/mv, but better not to go there unless you +really need to. The updates must be atomic in any case, so direct cp won't +work.) + +The granularity at which data is moved to alternate storage is individual +messages. This is true even for *multi-dbox* when multiple messages are stored +in a single 'm.*' storage file. If individual messages from an 'm.*' storage +file need to be moved to alternate storage, the message data is written out to +a different 'm.*' storage file (either new or existing) in the alternate +storage area and the "map index" updated accordingly. + +Alternate storage is completely transparent at the IMAP/POP level. Users +accessing mail through IMAP or POP cannot normally tell if any given message is +stored in primary storage or alternate storage. Conceivably users might be able +to measure a performance difference; the point is that there is no IMAP/POP +command which could be used to expose this information. It is entirely possible +to have a mail folder which contains a mix of messages stored in primary +storage and alternate storage. + +dbox and mail header metadata +----------------------------- + +Unlike when using [MailboxFormat.mbox.txt] as +[MailboxFormat.txt], where mail headers (for example 'Status', 'X-UID', etc.) +are [MailboxFormat.mbox.txt], the mail +headers within dbox files are (usually)*not* used for this purpose by dovecot; +neither when mails are created/moved/etc. via IMAP nor when dboxes are placed +(e.g. copied or moved in the filesystem) in a mail location (and then +"imported" by dovecot).Therefore, it is (usually) *not* necessary, to strip any +such mail headers at the MTA, MDA or LDA (as it is recommended with +[MailboxFormat.mbox.txt]). + +There is one exception, though, namely when 'pop3_reuse_xuidl=yes' (which is +however rather deprecated):In this case 'X-UIDL' is used for the POP3 UIDLs. +Therefore,*in this case, is recommended to strip the 'X-UIDL' mail headers +_case-insensitively_ at the MTA, MDA or LDA*. + +Accessing expunged mails with mdbox +----------------------------------- + +"mdbox_deleted" storage can be used to access mdbox's all mails that are +completely deleted (reference count=0). The mdbox_deleted parameters should +otherwise be exactly the same as mdbox's. Then you can use e.g. doveadm fetch +or doveadm import commands to access the mails. For example: + +---%<------------------------------------------------------------------------- +# If you have mail_location=mdbox:~/mdbox:INDEX=/var/index/%u +doveadm import mdbox_deleted:~/mdbox:INDEX=/var/index/%u "" subject oops +---%<------------------------------------------------------------------------- + +This finds a deleted mail with subject "oops" and imports it into INBOX. + +Mail delivery +============= + +Some MTA configurations have the MTA directly dropping mail into Maildirs or +mboxes. Since most MTAs don't understand the dbox format, this option is not +available. Instead, the MTA could be set up to use [LMTP.txt] or + [LDA.txt]. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.imapc.txt b/doc/wiki/MailboxFormat.imapc.txt new file mode 100644 index 0000000000..0fee740f61 --- /dev/null +++ b/doc/wiki/MailboxFormat.imapc.txt @@ -0,0 +1,178 @@ +imapc storage +============= + +The imapc storage accesses a remote IMAP server as if it were a regular Dovecot +mailbox format. Dovecot can treat it as a very dummy storage or optionally a +more capable storage. + +---%<------------------------------------------------------------------------- +# In-memory index files: +mail_location = imapc: + +# Store index files locally: +mail_location = imapc:~/imapc +---%<------------------------------------------------------------------------- + +Connection settings +------------------- + +Do a regular IMAP LOGIN to imap.example.com using 'a LOGIN user@example.com +secret': + +---%<------------------------------------------------------------------------- +imapc_host = imap.example.com +imapc_password = secret +imapc_port = 143 +imapc_user = user@example.com +---%<------------------------------------------------------------------------- + +If you want to use a master user login, set: + +---%<------------------------------------------------------------------------- +imapc_master_user = masteruser +---%<------------------------------------------------------------------------- + +For SSL, use either: + + * ---%<---------------------------------------------------------------------- + imapc_ssl = imaps + imapc_port = 993 + ---%<---------------------------------------------------------------------- + + * ---%<---------------------------------------------------------------------- + imapc_ssl = starttls + imapc_port = 143 + ---%<---------------------------------------------------------------------- + +For testing you may want to disable all SSL certificate checks: + +---%<------------------------------------------------------------------------- +imapc_ssl_verify = no +---%<------------------------------------------------------------------------- + +You can use other SASL mechanisms besides PLAIN by specifying (the first one +advertised by IMAP server is used): + +---%<------------------------------------------------------------------------- +imapc_sasl_mechanisms = external plain login +---%<------------------------------------------------------------------------- + +The SASL client mechanisms are implemented in Dovecot's lib-sasl/ code. It's +possible to add more with plugins. + +Other settings +-------------- + +imapc_cmd_timeout: + How long to wait for an IMAP command to reply before disconnecting and + retrying (default: 5 mins). + +imapc_list_prefix: + Access only mailboxes under this prefix. For example + 'imapc_list_prefix=INBOX/' + +imapc_max_idle_time: + Send something (NOOP/DONE) to server after not sending anything for this + amount of time (default: 29 mins). + +imapc_rawlog_dir: + Log all IMAP traffic input/output to this directory. + +Features and workarounds +------------------------ + +'imapc_features' setting is a space-separated list of features and workarounds +that can be enabled. + +Optimizations +------------- + +rfc822.size: + Allow passing through message sizes using 'FETCH RFC822.SIZE' + +fetch-headers: + Allow fetching specific message headers using 'FETCH BODY.PEEK[HEADER.FIELDS + (..)]' + +fetch-bodystructure: + Allow fetching BODY and BODYSTRUCTURE from remote server. (v2.2.30+) + +search: + Allow using 'SEARCH' command. + +Features +-------- + +gmail-migration: + Enable GMail-specific migration: Use IMAP X-GM-MSGID as POP3 UIDL. Add + $GMailHaveLabels keyword to mails that have X-GM-LABELS except for \Muted (to + be used for migrating only archived emails in "All Mails"). Add + pop3_deleted_flag to mails that don't exist in POP3 server. + +proxyauth: + Use Sun/Oracle IMAP-server specific PROXYAUTH command to do master user + authentication. Normally this would be done using the SASL PLAIN + authentication. + +throttle:INIT:MAX:SHRINK: + When receiving [THROTTLED] response (from GMail), throttling is applied. INIT + = initial throttling msecs (default: 50 ms), afterwards each subsequent + [THROTTLED] doubles the throttling until MAX is reached (default: 16000 ms). + When [THROTTLED] is not received for a while, it's shrank again. The initial + shrinking is done after SHRINK (default: 500 ms). If [THROTTLED] is received + again within this timeout, it's doubled, otherwise both throttling and the + next shrinking timeout is shrank to 3/4. + +modseq: + Access MODSEQ and HIGHESTMODSEQ (v2.2.24+) + +delay-login: + Don't connect to the remote server until some command requires it. By default + the server is connected to immediately on login. (v2.2.29+) + +Quota +----- + +Using the "imapc" quota backend allows asking for the quota from remote IMAP +server (v2.2.30+). By default it uses GETQUOTAROOT INBOX for getting the quota. +There are also two parameters that can be used to control how the quota is +looked up: + + * box=: Use GETQUOTAROOT + * root=: Use GETQUOTA + +For example: + +---%<------------------------------------------------------------------------- +plugin { + quota = imapc:root=User Quota +} +---%<------------------------------------------------------------------------- + +Workarounds +----------- + +zimbra-workarounds: + Fetch full message using 'BODY.PEEK[HEADER] BODY.PEEK[TEXT]' instead of just + 'BODY.PEEK[]' because the header differs between these two when there are + illegal control chars or 8bit chars. This mainly caused problems with dsync, + but as of v2.2.22+ this should no longer be a problem and there's probably no + need to enable this workaround. + +no-examine: + Use SELECT instead of EXAMINE even when we don't want to modify anything in + the mailbox. This is a Courier-workaround where it didn't permanently assign + UIDVALIDITY to an EXAMINEd mailbox, but assigned it for SELECTed mailbox. + +fetch-msn-workarounds: + Try to ignore wrong message sequence numbers in FETCH replies whenever + possible, preferring to use the returned UID number instead. + +fetch-fix-broken-mails: + If a FETCH returns 'NO' (but not 'NO [LIMIT]' or 'NO [SERVERBUG]'), assume + the mail is broken in server and just treat it as if it were an empty email. + NOTE: This is often a dangerous option! It's not safe to assume that NO means + a permanent error rather than a temporary error. This feature should be + enabled only for specific users who have been determined to be broken. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.mailstore.txt b/doc/wiki/MailboxFormat.mailstore.txt new file mode 100644 index 0000000000..3993602703 --- /dev/null +++ b/doc/wiki/MailboxFormat.mailstore.txt @@ -0,0 +1,17 @@ +Mailstore +========= + +This format originated with exim. Each message is written as two unique +filenames ending in .env and .msg. The .env contains the message envelope, and +the .msg file contains the actual message. When an email is delivered, the +email header is created with the suffix .tmp, and the message is created with +.msg. When the .msg file is completed, the .tmp file is renamed to end in .env. +Programs wishing to access an email must wait for both files to be present, or +for the absence of the .tmp file. + +References +---------- + + * http://www.exim.org/exim-html-4.60/doc/html/spec.html/ch26.html#id2639666 + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.mbox.txt b/doc/wiki/MailboxFormat.mbox.txt new file mode 100644 index 0000000000..438f9d89cc --- /dev/null +++ b/doc/wiki/MailboxFormat.mbox.txt @@ -0,0 +1,290 @@ +Mbox Mailbox Format +=================== + +Contents + + + 1. Mbox Mailbox Format + + 1. Locking + + 1. Dotlock + + 2. Deadlocks + + 2. Directory Structure + + 3. Dovecot's Metadata + + 4. Dovecot's Speed Optimizations + + 5. From Escaping + + 6. Mbox Variants + + 7. References + +Usually UNIX systems are configured by default to deliver mails to +'/var/mail/username' or '/var/spool/mail/username' mboxes. In IMAP world these +files are called INBOX mailboxes. IMAP protocol supports multiple mailboxes +however, so there needs to be a place for them as well. Typically they're +stored in '~/mail/' or '~/Mail/' directories. + +The mbox file contains all the messages of a single mailbox. Because of this, +the mbox format is typically thought of as a slow format. However with +Dovecot's indexing this isn't true. Only expunging messages from the beginning +of a large mbox file is slow with Dovecot, most other operations should be +fast. Also because all the mails are in a single file, searching is much faster +than with maildir. + +Modifications to mbox may require moving data around within the file, so +interruptions (eg. power failures) can cause the mbox to break more or less +badly. Although Dovecot tries to minimize the damage by moving the data in a +way that data should never get lost (only duplicated), mboxes still aren't +recommended to be used for important data. + +Locking +------- + +Locking is a mess with mboxes. There are multiple different ways to lock a +mbox, and software often uses incompatible locking. See for +how to check what locking methods some commonly used programs use. + +There are at least four different ways to lock a mbox: + + * *dotlock*: 'mailboxname.lock' file created by almost all software when + writing to mboxes. This grants the writer an exclusive lock over the mbox, + so it's usually not used while reading the mbox so that other processes can + also read it at the same time. So while using a dotlock typically prevents + actual mailbox corruption, it doesn't protect against read errors if mailbox + is modified while a process is reading. + * *flock*: 'flock()' system call is quite commonly used for both read and + write locking. The read lock allows multiple processes to obtain a read lock + for the mbox, so it works well for reading as well. The one downside to it + is that it doesn't work if mailboxes are stored in NFS. + * *fcntl*: Very similar to *flock*, also commonly used by software. In some + systems this 'fcntl()' system call is compatible with 'flock()', but in + other systems it's not, so you shouldn't rely on it.*fcntl* works with NFS + if you're using lockd daemon in both NFS server and client. + * *lockf*: POSIX 'lockf()' locking. Because it allows creating only exclusive + locks, it's somewhat useless so Dovecot doesn't support it. With Linux + 'lockf()' is internally compatible with 'fcntl()' locks, but again you + shouldn't rely on this. + +Dotlock +------- + +Another problem with dotlocks is that if the mailboxes exist in '/var/mail/', +the user may not have write access to the directory, so the dotlock file can't +be created. There are a couple of ways to work around this: + + * Give a mail group write access to the directory and then make sure that all + software requiring access to the directory runs with the group's privileges. + This may mean making the binary itself setgid-mail, or using a separate + dotlock helper program which is setgid-mail. With Dovecot this can be done + by setting 'mail_privileged_group = mail'. + * Set sticky bit to the directory ('chmod +t /var/mail'). This makes it + somewhat safe to use, because users can't delete each others mailboxes, but + they can still create new files (the dotlock files). The downside to this is + that users can create whatever files they wish in there, such as a mbox for + newly created user who hadn't yet received mail. + +Deadlocks +--------- + +If multiple lock methods are used, which is usually the case since dotlocks +aren't typically used for read locking, the order in which the locking is done +is important. Consider if two programs were running at the same time, both use +dotlock and fcntl locking but in different order: + + * Program A: fcntl locks the mbox + * Program B at the same time: dotlocks the mbox + * Program A continues: tries to dotlock the mbox, but since it's already + dotlocked by B, it starts waiting + * Program B continues: tries to fcntl lock the mbox, but since it's already + fcntl locked by A, it starts waiting + +Now both of them are waiting for each others locks. Finally after a couple of +minutes they time out and fail the operation. + +Directory Structure +------------------- + +By default, when listing mailboxes, Dovecot simply assumes that all files it +sees are mboxes and all directories mean that they contain sub-mailboxes. There +are two special cases however which aren't listed: + + * '.subscriptions' file contains IMAP's mailbox subscriptions. + * '.imap/' directory contains Dovecot's index files. + +Because it's not possible to have a file which is also a directory, it's not +normally possible to create a mailbox and child mailboxes under it. + +However if you really want to be able to have mailboxes containing both +messages and child mailboxes under mbox, then Dovecot can be configured to do +this, subject to certain provisos; see . + +Dovecot's Metadata +------------------ + +Dovecot uses C-Client (ie. UW-IMAP, Pine) compatible headers in mbox messages +to store metadata. These headers are: + + * X-IMAPbase: Contains UIDVALIDITY, last used UID and list of used keywords + * X-IMAP: Same as X-IMAPbase but also specifies that the message is a "pseudo + message" + * X-UID: Message's allocated UID + * Status: R (\Seen) and O (non-\Recent) flags + * X-Status: A (\Answered), F (\Flagged), T (\Draft) and D (\Deleted) flags + * X-Keywords: Message's keywords + * Content-Length: Length of the message body in bytes + +Whenever any of these headers exist, Dovecot treats them as its own private +metadata. It does sanity checks for them, so the headers may also be modified +or removed completely. None of these headers are sent to IMAP/POP3 clients when +they read the mail. + +*The MTA, MDA or LDA should strip all these headers _case-insensitively_ before +writing the mail to the mbox.* + +Only the first message contains the X-IMAP or X-IMAPbase header. The difference +is that when all the messages are deleted from mbox file, a "pseudo message" is +written to the mbox which contains X-IMAP header. This is the "DON'T DELETE +THIS MESSAGE -- FOLDER INTERNAL DATA" message which you hate seeing when using +non-C-client and non-Dovecot software. This is however important to prevent +abuse, otherwise the first mail which is received could contain faked +X-IMAPbase header which could cause trouble. + +If message contains X-Keywords header, it contains a space-separated list of +keywords for the mail. Since the same header can come from the mail's sender, +only the keywords are listed in X-IMAP header are used. + +The UID for a new message is calculated from "last used UID" in X-IMAP header + +1. This is done always, so fake X-UID headers don't really matter. This is also +why the pseudo message is important. Otherwise the UIDs could easily grow over +2^31 which some clients start treating as negative numbers, which then cause +all kinds of problems. Also when 2^32 is exceeded, Dovecot will also start +having some problems. + +Content-Length is used as long as another valid mail starts after that many +bytes. Because the byte count must be exact, it's quite unlikely that abusing +it can cause messages to be skipped (or rather appended to the previous +message's body). + +Status and X-Status headers are trusted completely, so it's pretty good idea to +filter them in LDA if possible. + +Dovecot's Speed Optimizations +----------------------------- + +Updating messages' flags and keywords can be a slow operation since you may +have to insert a new header (Status, X-Status, X-Keywords) or at least insert +data in the header's value. Some mbox MUAs do this simply by rewriting all of +the mbox after the inserted data. If the mbox is large, this can be very slow. +Dovecot optimizes this by always leaving some space characters after some of +its internal headers. It can use this space to move only minimal amount of data +necessary to get the necessary data inserted. Also if data is removed, it just +grows these spaces areas. + +'mbox_lazy_writes' setting works by adding and/or updating Dovecot's metadata +headers only after closing the mailbox or when messages are expunged from the +mailbox. C-Client works the same way. The upside of this is that it reduces +writes because multiple flag updates to same message can be grouped, and +sometimes the writes don't have to be done at all if the whole message is +expunged. The downside is that other processes don't notice the changes +immediately (but other Dovecot processes do notice because the changes are in +index files). + +'mbox_dirty_syncs' setting tries to avoid re-reading the mbox every time +something changes. Whenever the mbox changes (ie. timestamp or size), it first +checks if the mailbox's size changed. If it didn't, it most likely meant that +only message flags were changed so it does a full mbox read to find it. If the +mailbox shrunk, it means that mails were expunged and again Dovecot does a full +sync. Usually however the only thing besides Dovecot that modifies the mbox is +the LDA which appends new mails to the mbox. So if the mbox size was grown, +Dovecot first checks if the last known message is still where it was last time. +If it is, Dovecot reads only the newly added messages and goes into a "dirty +mode". As long as Dovecot is in dirty mode, it can't be certain that mails are +where it expects them to be, so whenever accessing some mail, it first verifies +that it really is the correct mail by finding its X-UID header. If the X-UID +header is different, it fallbacks to a full sync to find the mail's correct +position. The dirty mode goes away after a full sync. If 'mbox_lazy_writes' was +enabled and the mail didn't yet have X-UID header, Dovecot uses MD5 sum of a +couple of headers to compare the mails. + +'mbox_very_dirty_syncs' does the same as 'mbox_dirty_syncs', but the dirty +state is kept also when opening the mailbox. Normally opening the mailbox does +a full sync if it had been changed outside Dovecot. + +From Escaping +------------- + +In mboxes a new mail always begins with a "From " line, commonly referred to as +From_-line. To avoid confusion, lines beginning with "From " in message bodies +are usually prefixed with '>' character while the message is being written to +in mbox. + +Dovecot doesn't currently do this escaping however. Instead it prevents this +confusion by adding Content-Length headers so it knows later where the next +message begins. Dovecot doesn't either remove the '>' characters before sending +the data to clients. Both of these will probably be implemented later. + +Mbox Variants +------------- + +There are a few minor variants of this format: + +*mboxo* is the name of original mbox format originated with Unix System V. +Messages are stored in a single file, with each message beginning with a line +containing "From SENDER DATE". If "From " (case-sensitive, with the space) +occurs at the beginning of a line anywhere in the email, it is escaped with a +greater-than sign (to ">From "). Lines already quoted as such, for example +">From " or ">>>From " are *not* quoted again, which leads to irrecoverable +corruption of the message content. + +*mboxrd* was named for Raul Dhesi in June 1995, though several people came up +with the same idea around the same time. An issue with the mboxo format was +that if the text ">From " appeared in the body of an email (such as from a +reply quote), it was not possible to distinguish this from the mailbox format's +quoted ">From ". mboxrd fixes this by always quoting already quoted "From " +lines (e.g. ">From ", ">>From ", ">>>From ", etc.) as well, so readers can just +remove the first ">" character. This format is used by qmail and getmail +(>=4.35.0). + +*mboxcl* format was originated with Unix System V Release 4 mail tools. It adds +a Content-Length field which indicates the number of bytes in the message. This +is used to determine message boundaries. It still quotes "From " as the +original mboxo format does (and *not* as mboxrd does it). + +*mboxcl2* is like mboxcl but does away with the "From " quoting. + +*MMDF* (Multi-channel Memorandum Distribution Facility mailbox format) was +originated with the MMDF daemon. The format surrounds each message with lines +containing four control-A's. This eliminates the need to escape From: lines. + +Dovecot currently uses mboxcl2 format internally, but it's planned to move to +combination of mboxrd and mboxcl. + +*How a message is read stored in mbox extension ?* + + * An email client reader scans throughout mbox file looking for From_ lines. + * Any From_ line marks the beginning of a message. + * Once the reader finds a message, it extracts a (possibly corrupted) envelope + sender and delivery date out of the From_ line. + * It then reads until the next From_ line or scans till the end of file, + whenever From_ comes first. + * It removes the last blank line and deletes the quoting of >From_ lines and + >>From_ lines and so on. + +References +---------- + + * Wikipedia [http://en.wikipedia.org/wiki/Mbox] + * Qmail mbox [http://www.qmail.org/man/man5/mbox.html] + * Mbox family + [http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/mail-mbox-formats.html] + * CommuniGatePro mbox + [http://www.communigate.com/CommuniGatePro/Mailboxes.html#mbox] + * MBOX File Viewer [http://www.freeviewer.org/mbox/] + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.mbx.txt b/doc/wiki/MailboxFormat.mbx.txt new file mode 100644 index 0000000000..c5d76a281e --- /dev/null +++ b/doc/wiki/MailboxFormat.mbx.txt @@ -0,0 +1,24 @@ +mbx +=== + +A high performance mbox-replacement created by Mark Crispin while at University +of Washington for UW-IMAP. Each message in the file is preceded by a record +carrying all the metadata that IMAP-protocol needs. This allows changing the +metadata easily by modifying the fixed-size header, rather than moving data +around in a file like with mbox. + +File locking is handled more intelligently (only appends may need to wait for +locks), making this format a good choice for shared mailboxes. + +Downsides contain: + + * Messages don't get deleted from disk until there's only one client accessing + the mailbox, so you may need to add some forced downtime to get it done. + * Works only in local filesystem. + * Expunging is still as costly and as fragile as with mbox. + * At least on 32 bit systems, mailboxes are limited to 2GB size. (A mailbox + can't be accessed anymore without manual fixing if it reached 2GB (minus 1B) + of size on disk.) This is probably UW-IMAP -specific implementation problem + rather than actual mailbox format issue. + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxFormat.txt b/doc/wiki/MailboxFormat.txt new file mode 100644 index 0000000000..0f70184929 --- /dev/null +++ b/doc/wiki/MailboxFormat.txt @@ -0,0 +1,147 @@ +Mailbox Formats +=============== + +Mailbox formats supported by Dovecot: ++-----------------------------+---------------+-------------------+-------------+ +| *Name* | *Tag* | *Description* | ++-----------------------------+---------------+-------------------+-------------+ +| | 'mbox' | Traditional UNIX | +| [MailboxFormat.mbox.txt] | | mailbox format. | +| | | Users' INBOX | +| | | mboxes are | +| | | commonly stored in| +| | | '/var/spool/mail' | +| | | or '/var/mail' | +| | | directory. Single | +| | | file contains | +| | | multiple messages.| ++-----------------------------+---------------+-------------------+-------------+ +| | 'maildir' | One file contains | +| [MailboxFormat.Maildir.txt] | | one message. A | +| | | reliable choice | +| | | since files are | +| | | never modified and| +| | | all operations are| +| | | atomic. The | +| | | top-level | +| | | 'Maildir' | +| | | directory contains| +| | | the 'Maildir/cur',| +| | | 'Maildir/new' and | +| | | 'Maildir/tmp' | +| | | subdirectories. | ++-----------------------------+---------------+-------------------+-------------+ +| | 'sdbox' | *single-dbox*, one| Dovecot's | +| [MailboxFormat.dbox.txt] | | message per file | own high | +| | | | performance | +| | | | mailbox | +| | | | format. | +| | | | Messages are| +| | | | stored in | +| | | | one or more | +| | | | files, each | +| | | | containing | +| | | | one or more | +| | | | messages. | ++-----------------------------+---------------+-------------------+-------------+ +| 'mdbox' | *multi-dbox*, | +| | multiple | +| | messages per | +| | file | ++-----------------------------+---------------+-------------------+-------------+ +| 'dbox' | deprecated | +| | alias for | +| | 'sdbox' | ++-----------------------------+---------------+-------------------+-------------+ +| | 'cydir' | Dovecot's own | +| [MailboxFormat.Cydir.txt] | | simple and high | +| | | performance | +| | | Cyrus-like mailbox| +| | | format. It should | +| | | be mostly used for| +| | | testing and | +| | | benchmarking only.| ++-----------------------------+---------------+-------------------+-------------+ +| | 'imapc' | Use remote IMAP | +| [MailboxFormat.imapc.txt] | | server as mail | +| | | storage. | ++-----------------------------+---------------+-------------------+-------------+ +| | 'pop3c' | Use remote POP3 | +| [MailboxFormat.pop3c.txt] | | server as mail | +| | | storage. | ++-----------------------------+---------------+-------------------+-------------+ + +The *Tag* column indicates the tag which is used at the beginning of a [MailLocation.txt]. + +Mailbox formats *not* supported by Dovecot: ++-------------------------------+---------------------------------------------+ +| *Name* | *Description* | ++-------------------------------+---------------------------------------------+ +| [MailboxFormat.mbx.txt] | UW-IMAP's old high performance mailbox | +| | format. One file contains all the mailboxes,| +| | so expunges may still be slow. | ++-------------------------------+---------------------------------------------+ +| [MailboxFormat.mix.txt] | UW-IMAP's new (2006) high performance | +| | mailbox format. Similar to multi-dbox. | ++-------------------------------+---------------------------------------------+ +| | A format created by Exim. | +| [MailboxFormat.mailstore.txt] | | ++-------------------------------+---------------------------------------------+ +| [MailboxFormat.MH.txt] | One file contains one message. Sort order of| +| | the folder determines the message ID and | +| | name. Actively used by projects such as | +| | MH-E, NMH, exmh. Experimentally supported | +| | by UW-IMAP | +| | [https://www.washington.edu/imap/]. | ++-------------------------------+---------------------------------------------+ +| [MailboxFormat.MMDF.txt]| Similar to mbox, but instead of From-line | +| | separators it uses four '^A' characters | ++-------------------------------+---------------------------------------------+ +| | One file contains one message, plus there | +| [MailboxFormat.Cyrus.txt] | are a couple of index/cache files. Commonly | +| | referred to as being maildir-like, although | +| | they have only a single thing in common. | ++-------------------------------+---------------------------------------------+ + +Adding support for new formats for Dovecot isn't very difficult, although it +can be time consuming. Dovecot exposes a nice and simple API which needs to be +implemented. Use Cydir format as an example. + +Software Support +---------------- + ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| *Format/Software* | *Dovecot | *UW-IMAP | *Courier-IMAP | *Exim | *Postfix | *PINE | *mutt | *procmail | *maildrop | +| | [https://www.dovecot.org/]* | [https://www.washington.edu/imap/]* | [https://www.courier-mta.org/imap/]* | [https://www.exim.org/]* | [https://www.postfix.org/]* | [https://www.washington.edu/pine/]* | [https://www.mutt.org/]* | [https://www.procmail.org/]* | [https://www.courier-mta.org/maildrop/]* | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| [MailboxFormat.mbox.txt]| Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| [MailboxFormat.mbx.txt] | No | Yes | No | Yes | No | Yes | No | No | No | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| | Yes | No | Yes | Yes | Yes | No | Yes | Yes (3.22) | Yes | +| [MailboxFormat.Maildir.txt] | | | | | | | | | | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| | No | No | No | Yes | No | No | No | No | No | +| [MailboxFormat.mailstore.txt] | | | | | | | | | | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| [MailboxFormat.dbox.txt]| Yes | No | No | No | No | No | No | No | No | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ +| [MailboxFormat.MH.txt] | No | Yes | No | No | No | Yes | Yes | Yes | No | ++-------------------------------+-----------------------------+-------------------------------------+--------------------------------------+--------------------------+-----------------------------+-------------------------------------+--------------------------+------------------------------+------------------------------------------+ + +Conversion Between Mailbox Formats +---------------------------------- + +See . + +References +---------- + + * Mutt mailbox formats: https://rucus.ru.ac.za/docs/mutt/manual58.html + * Article on mailbox formats: + https://www.livejournal.com/users/rfunk/1571.html + * Mbox and maildir comparison: + https://www.linuxmail.info/mbox-maildir-mail-storage-formats/ + +(This file was created from the wiki on 2019-06-19 12:42) diff --git a/doc/wiki/MailboxSettings.txt b/doc/wiki/MailboxSettings.txt new file mode 100644 index 0000000000..3e94581144 --- /dev/null +++ b/doc/wiki/MailboxSettings.txt @@ -0,0 +1,81 @@ +Mailbox settings +================ + +Since Dovecot v2.1 one can assign SPECIAL-USE RFC 6154 +[http://www.faqs.org/rfcs/rfc6154.html] tags and specify, which mailboxes to +create and/or subscribe to automatically. + +The autocreated mailboxes are created lazily to disk only when accessed for the +first time. The autosubscribed mailboxes aren't written to subscriptions file, +unless SUBSCRIBE command is explicitly used for them. + +The mailbox section name specifies the mailbox name. If it has spaces, you can +put it in "quotes". The mailbox settings are: + + * auto: Autocreate/subscribe mailbox? + * no: Neither + * create: Autocreate, but don't autosubscribe + * subscribe: Autocreate and autosubscribe + * special_use: Space-separated list of SPECIAL-USE flags to use for the + mailbox. There are no validity checks, so you could specify anything you + want in here, but it's not a good idea to use other than the standard ones + specified in the RFC. + * NOTE: Due to a bug in Dovecot v2.2.30+ if special-use flags are used, + SPECIAL-USE needs to be added to post-login CAPABILITY response as RFC + 6154 mandates. You can do this with 'imap_capability = +SPECIAL-USE' + * autoexpunge=