2.5/x/2.5.0.rst

:tocdepth: 3

==============================
Cyrus IMAP 2.5.0 Release Notes
==============================

.. IMPORTANT::

    Make sure to read the :ref:`imap-relnotes-2.5.0-upgrading` notes
    (all of them).

*   :ref:`relnotes-2.5.0-new-features`
*   :ref:`relnotes-2.5.0-config-option-changes`
*   :ref:`relnotes-2.5.0-development-autoconf`
*   :ref:`relnotes-2.5.0-development-phabricator`
*   :ref:`relnotes-2.5.0-development-documentation`

Download from GitHub:

    *   https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.0/cyrus-imapd-2.5.0.tar.gz
    *   https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-imapd-2.5.0/cyrus-imapd-2.5.0.tar.gz.sig

.. _imap-relnotes-2.5.0-upgrading:

Upgrading to Cyrus IMAP 2.5.0
=============================

It is strongly recommended to shut down the Cyrus IMAP services before
performing the upgrade, as the newer binaries will end up writing to
``mailboxes.db`` in a way that is not compatible with the older binaries
(that would otherwise still be running).

You can start the server immediately after upgrading, however.

Underscores in ``cmd`` Names in :cyrusman:`cyrus.conf(5)`
---------------------------------------------------------

Underscores (the ``_`` character) are no longer allowed in the
``START``, ``SERVICES`` and ``EVENTS`` sections of
:cyrusman:`cyrus.conf(5)`, as they interfere with configuration options
in :cyrusman:`imapd.conf(5)` being prefixed by service names and an
underscore (``_``) character.

Database Format Upgrade for Mailboxes
-------------------------------------

Upgrading to Cyrus IMAP 2.5.0 recommends the upgrade of database
formats, for which the reconstruct utility has been modified allowing
administrators to run:

.. parsed-literal::

    # :command:`/usr/lib/cyrus-imapd/reconstruct -V max` [options]

This command can be run at the administrator's convenience, and while
running the database format upgrade process is not strictly required, it
is certainly strongly recommended.

Releases prior to 2.5.0, namely the 2.4 series, upgraded ``cyrus.index``
database formats in-line (i.e. as the mailbox in question as being
used). This may have caused an I/O storm in some situations, which 2.5.0
aims to address.

The minor version of the ``cyrus.index`` file in 2.5.0 is **13**. While
mailboxes with an older format will continue to work "forever", newer
features such as annotation quotas will fail, as there is no space in
the database formats prior to version 13 to store the new feature's
information.

Until you upgrade the database format, you may experience slightly
inconsistent search results, due to the ``cyrus.cache`` format in Cyrus
IMAP versions prior to 2.4.0 running words together and not obeying word
boundaries (as well as versions in the 2.4 and 2.5 series).

If you were running Cyrus 2.3.x with ``expunge_mode: delayed``, then the
contents of the ``cyrus.expunge`` file will be removed the first time
the mailbox is opened, and the mailbox will act as if ``expunge_mode``
was set to ``immediate`` -- until it is upgraded.

This is because version 2.3 mailboxes used the ``cyrus.expunge`` file,
and it's not worth the complexity to try to recreate that file.

This upgrade path supports upgrades from Cyrus IMAP version 2.2.12 and
later versions.

.. IMPORTANT::

    The above command can run while the Cyrus IMAP services are in
    operation, and does **not** interfere with the service's operations.

.. NOTE::

    Additional options can be specified to :cyrusman:`reconstruct(8)` to
    more granularly specify which mailboxes' :file:`cyrus.index` is to
    be upgraded.

Quota Fixes and Enhancements
----------------------------

After all mailboxes are upgraded, you should run ``quota -f`` to make
all message and folder counts are correct. The new quota types that
Cyrus IMAP 2.5.0 supports require the counts to need to be accurate.

Cyrus IMAP Murder Topologies
----------------------------

Environments that run a Cyrus IMAP Murder topology will want to upgrade
their backends before they upgrade their frontends.

Greater Memory Footprint
------------------------

.. include:: /assets/cyrus-more-memory-post23.rst

.. _relnotes-2.5.0-new-features:

New Features
============

.. _relnotes-2.5.0-index-namelock-release:

Index Namelock Release
----------------------

Long-running (idling) connections may have previously intervened with
mailbox deletions and general cleanup work, resulting in one connection
blocking another from, for example, deleting and recreating a mailbox:

.. code-block:: bash
    :linenos:

    C1: SELECT "Foo"
    C2: DELETE "Foo"
    C2: CREATE "Foo"

The ``CREATE`` command on line 3 (by client 2, while client 1 still has
the ``Foo`` mailbox selected) would have failed before, but does now
succeed.

Extended Quota Types
--------------------

New ways to restrict resource usage:

*   Number of Folders,
*   Number of Messages,
*   Number of Annotations

CalDAV and CardDAV Support
--------------------------

CalDAV and CardDAV support no longer live out-of-tree in the same GIT
repository, but are now mainstream and included in Cyrus IMAP 2.5.0.

This has been a major effort by Carnegie Mellon University and FastMail,
and continues to be, enriching the experience of Cyrus IMAP users
globally.

If upgrading to Cyrus IMAP 2.5 from one of the 2.4.17-caldav-beta
releases, you MUST run the ``dav_reconstruct`` utility for each of
your CalDAV users.

Support for RFC 5464: IMAP METADATA
-----------------------------------

Cyrus IMAP now fully supports :rfc:`5464`, *The IMAP METADATA
Extension*.

This also means the support for the ANNOTATEMORE draft for IMAP will
ultimately be dropped.

Aside from the trusted folder metadata, this also introduces message
annotations. Users will need to be given the ``n`` right to allow them
to set message annotations.

Event Notifications
-------------------

Various events occuring in Cyrus IMAP, among which mailbox, message and
access events, can now be pushed out through a side-channel and notify
client applications or provide other infrastructure with detailed
information.

Mailbox Distribution Enhancements: Backend and Partition Selection
------------------------------------------------------------------

Thanks to the work of Julien Coloos and colleagues, a new mode is
available for server and partition selection upon mailbox creation.

Prior to Cyrus IMAP 2.5.0, the server and/or partition on which to
create a new mailbox was selected by detecting the largest amount of
absolute free disk space on all servers and partitions. The mailbox
distribution feature allows for more intelligent and flexible routines
to be used in the selection. Please see our Administrator Guide for
more details.

New Database Format for ``mailboxes.db``
----------------------------------------

The database format for mailboxes.db has been upgraded, adding;

*   A new mailbox type for deleted mailboxes.

    In versions of Cyrus IMAP prior to 2.5.0, mailboxes that were
    deleted may have become unavailable for actual cleanup expecting
    another session on the same mailbox to clean up the directories and
    files. See also :ref:`relnotes-2.5.0-index-namelock-release`.

*   A key-value storage format is used, allowing for faster and better
    parsing of :file:`mailboxes.db`, more granular updates to runtime
    environments, and more sustainable future upgrades.

New Database Format ``twoskip``
-------------------------------

A new database format has been added, called ``twoskip`` [#]_.

This new database format is reputedly better, faster, safer and 64-bit
capable, as opposed to our former favorite ``skiplist``.

``twoskip`` can be used for the following databases:

*   ``annotation_db``
*   ``duplicate_db``
*   ``mboxkey_db``
*   ``mboxlist_db``
*   ``ptscache_db``
*   ``quota_db``
*   ``seenstate_db``
*   ``subscription_db``
*   ``statuscache_db``
*   ``tls_sessions_db``
*   ``user_deny_db``

Miscellaneous
-------------

Allowing Undefined Annotations
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Cyrus IMAP 2.5.0 allows administrators to configure that undefined
annotations should be allowed, using a new
``annotation_allow_undefined`` setting in :cyrusman:`imapd.conf(5)`.

Catchall Mailbox for LMTP
^^^^^^^^^^^^^^^^^^^^^^^^^

Thanks to the work by Carsten Hoeger and Ralf Haferkamp, this new
feature enables administrators to configure a target mailbox for mail
delivered through LMTP to targetted mailboxes that do not exist.

For example, a mail that LMTP would deliver to ``user/bovik``, which
for the sake of argument does not exist in this example, setting
``lmtp_catchall_mailbox`` to ``admin`` will instead deliver the mail
to ``user/admin``.

.. NOTE::
    **Mailbox name, not Email Address**

    Note that **lmtp_catchall_mailbox** must be a user mailbox name,
    not an email address. Also note that the **user/** namespace
    indicator as well as the hierarchy separator are to be omitted.

Does this impact lmtp_fuzzy_mailbox_match?
++++++++++++++++++++++++++++++++++++++++++

Environments that have ``lmtp_fuzzy_mailbox_match`` enabled, in order
to have LMTP seek from the targetted, non-existent mailbox sub-folder
(example: ``user/bovik/spam/probably``) all the way to the toplevel
mailbox folder (i.e. ``user/bovik``) until it finds a mailbox
(sub-)folder that does exist (example: ``user/bovik/spam``), are not
impacted by this setting.

Can the lmtp_catchall_mailbox include the path to a sub-folder of a target mailbox?
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

UNCONFIRMED

Can the lmtp_catchall_mailbox be a shared folder?
+++++++++++++++++++++++++++++++++++++++++++++++++

UNCONFIRMED

Callout for SETMETADATA
^^^^^^^^^^^^^^^^^^^^^^^

A callout program can be called when annotations are set, configured
through ``annotation_callout``.

Host & User Login Restrictions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Logins can now be restricted on a per host (source IP address) or per
user basis, using the settings ``maxlogins_per_host`` and
``maxlogins_per_user``.

.. _relnotes-2.5.0-config-option-changes:

Configuration Option Changes and Enhancements
=============================================

.. IMPORTANT::

    While it is not mandatory to update your configuration file with
    these new settings, not doing so may have undesired side-effects,
    including but not limited to deprecation warnings in log messages.

Option Name Changes for ``autocreate``
--------------------------------------

The options related to automatic creation of user mailboxes and
sub-folders (aka. *autocreate*) have been changed to hold a prefix of
``autocreate_``.

The following *autocreate* options are now available:

**autocreate_inbox_folders** (was: ``autocreateinboxfolders``)

    ``autocreate_inbox_folders`` controls which folders to create in
    addition to the INBOX folder.

    Separate the folder names by ``|``.

**autocreate_post** (was: ``createonpost``)

    Controls whether or not to create a folder when a message is first
    posted to it (by LTMP).

**autocreate_quota** (was: ``autocreatequota``)

    When creating a user mailbox, set the quota for that mailbox to the
    value of this configuration option.

**autocreate_quota_messages** (not available)

    When creating a user mailbox, set the message quota (maximum number
    of messages allowed in the folder hierarchy) to the value of this
    configuration option.

**autocreate_sieve_folders** (was: ``autosievefolders``)

    Limit the folders that can be created automatically by a Sieve
    script performing a "fileinto" action, to the folders listed in
    this configuration option.

    Separate the folder names by ``|``.

**autocreate_sieve_script** (unchanged)

    When creating a user mailbox, associate the Sieve script configured
    here.

**autocreate_sieve_script_compile** (was: ``generate_compiled_sieve_script``)

    Whether or not to compile the Sieve script configured by
    ``autocreate_sieve_script``.

**autocreate_sieve_script_compiled** (was: ``autocreate_sieve_compiled_script``)

    When creating a user mailbox, associate the already compiled Sieve
    script configured here.

**autocreate_subscribe_folders** (was: ``autosubscribeinboxfolders``)

    List the folder names to which the user for which a mailbox is
    being created should be subscribed.

    .. NOTE::

        All folders listed here are considered to reside in the
        personal namespace.

    Separate the folder names by ``|``.

**autocreate_subscribe_sharedfolders** (was: ``autosubscribesharedfolders``)

    List the folder names of shared folders to which the user for which
    a mailbox is being automatically created should be subscribed.

    Separate the folder names by ``|``.

**autocreate_subscribe_sharedfolders_all** (was: ``autosubscribe_all_sharedfolders``)

    Rather than subscribe the user for which a mailbox is being
    automatically created to some shared folders, simply subscribe the
    user to all shared folders.

**autocreate_users** (unchanged)

    Limit the users for which mailboxes may be created to the list
    configured here.

Default Change: ``delete_mode``
-------------------------------

The default for the :cyrusman:`imapd.conf(5)` configuration option
``delete_mode`` has changed from ``immediate`` to ``delayed``.

This causes mail folders that are deleted by a client to not
immediately dissappear from the filesystem. Instead, they are renamed
to a deleted namespace that is visible only to administrators.

A separate job ``cyr_expire -D $x`` is to be included in the master
service configuration file :cyrusman:`cyrus.conf(5)`, specifically in
the EVENTS section. ``$x`` is a number of days to keep already deleted
folders.

**Example section of :cyrusman:`cyrus.conf(5)`**

    .. parsed-literal::

        EVENTS {
            deleteprune cmd="cyr_expire -D 69" at=0430
        }

In the aforementioned example, folders are purged from the filesystem
only after 2 times 31 plus 7 days, corresponding with 2 cycles of a
monthly (full, virtual) backup of which one might fail.

Default Change: ``expunge_mode``
--------------------------------

The default for the :cyrusman:`imapd.conf(5)` configuration option
``expunge_mode`` has changed from ``default`` to ``delayed``.

This causes the mail message files associated with messages that are
flagged as \Deleted in a folder that is subsequently expunged, or
individual messages that are expunged, to not be removed from the
filesystem directly.

A separate job ``cyr_expire -X $x`` is to be included in the master
service configuration file :cyrusman:`cyrus.conf(5)`, specifically in
the EVENTS section. ``$x`` is a number of days to keep the message
files on the filesystem.

**Example section of :cyrusman:`cyrus.conf(5)`**

    .. parsed-literal::

        EVENTS {
            expungeprune cmd="cyr_expire -X 69" at=0430
        }

In the aforementioned example, message files are purged from the
filesystem only after 2 times 31 plus 7 days, corresponding with 2
cycles of a monthly (full, virtual) backup of which one might fail.

Option Name Changes for ``ldap_tls_*``
--------------------------------------

Configuration option names for LDAP SSL/TLS configuration in
:cyrusman:`imapd.conf(5)` have been changed:

**ldap_ca_dir** (was: ``ldap_tls_cacert_dir``)

**ldap_ca_file** (was: ``ldap_tls_cacert_file``)

**ldap_client_cert** (was: ``ldap_tls_cert``)

**ldap_verify_peer** (was: ``ldap_tls_check_peer``)

**ldap_ciphers** (was: ``ldap_tls_ciphers``)

**ldap_client_key** (was: ``ldap_tls_key``)

Option Name Changes for ``tls_*``
---------------------------------

Configuration option names for SSL/TLS configuration in
:cyrusman:`imapd.conf(5)` have been changed to better reflect how
they are used, as enhancements would otherwise create great confusion.

**tls_client_ca_dir** (was: ``tls_ca_path``)

**tls_client_ca_file** (was: ``tls_ca_file``)

    The former ``tls_ca_*`` configuration options specified one or more
    SSL Certificate Authority certificates against which SSL
    certificates offered by clients could be verified.

    In a Cyrus IMAP Murder topology however, Cyrus IMAP servers
    themselves become clients of other Cyrus IMAP servers, but may not
    have been issued certificates under the same verification chain.

With the (too) generic names for ``tls_ca_*`` configuration options out
of the way, Cyrus IMAP 2.5.0 adds the following configuration options:

**tls_server_cert** (was: ``tls_cert_file``)

**tls_server_key** (was: ``tls_key_file``)

    Server SSL certificate and key to use for connections from
    clients.

**tls_sessions_db** (was: ``tlscache_db``)

**tls_sessions_db_path** (was: ``tlscache_db_path``)

New Options for ``tls_*``
-------------------------

**tls_client_cert** (<none>)

**tls_client_key** (<none>)

    Client SSL certificate and key to use when cyrus-imapd behaves as
    a client (to other cyrus-imapd server (instances)).

**tls_client_ca_file** (<none>)

**tls_client_ca_dir** (<none>)

    Certificate Authority file or directory used to verify client SSL
    certificates.

**tls_client_certs** (``optional``)

    Disable (``off``), allow (``optional``) or require (``require``)
    clients authenticate with an SSL certificate.

**tls_server_ca_file** (<none>)

**tls_server_ca_dir** (<none>)

    Certificate Authority file or directory used to verify SSL
    certificates offered by other servers.

**tls_compression** (``0``)

    Enable TLS compression. Disabled by default.

**tls_eccurve** (``prime256v1``)

    Select the elliptic curve used for ECDHE. See
    :command:`openssl ecparams -list_curves` for supported values on
    your platform.

**tls_prefer_server_ciphers** (``0``)

    Prefer the cipher order configured on the server-side.

**tls_versions** (``ssl2 ssl3 tls1_0 tls1_1 tls1_2``)

    Disable SSL/TLS protocols not in this list.

.. _relnotes-2.5.0-development-autoconf:

Development: Switch to ``autoconf`` and ``libtool``
===================================================

With the release of Cyrus IMAP 2.5.0, the Cyrus IMAP project has
switched to using autoconf and libtool.

.. _relnotes-2.5.0-development-phabricator:

Development: Switch to Phabricator
==================================

An instance of Phabricator is going to be replacing our old Bugzilla.

We believe this better facilitates our processes, and will make it
easier to contribute code and collaborate.

Please see https://git.cyrus.foundation/.

.. NOTE::

    The Phabricator installation suffered a disk failure. We now use
    GitHub for code and bug management. https://github.com/cyrusimap/cyrus-imapd

.. _relnotes-2.5.0-development-documentation:

Development: Sphinx for Documentation
=====================================

While a work in progress still, you're looking at the new and improved
documentation effort for the Cyrus project as a whole.

This documentation is written in reStructuredText, and rendered by
Sphinx.

The GIT repository for the documentation is at


.. rubric:: Footnotes

.. [#]

    http://opera.brong.fastmail.fm.user.fm/talks/twoskip/twoskip-yapc12.pdf

.. _RFC 5464: http://tools.ietf.org/html/rfc5464>