Improve nametrans user documentation

Fill in more details on nametrans and folder filtering. Also give them a
separate section in our user documentation. Everything will be
immediately online at docs.offlineimap.org.

The main change is to describe the reverse nametrans settings that are
needed since 6.4.0 to support remote folder creation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>

Changelog.draft.rst

@@ -16,5 +16,9 @@ New Features
 Changes
 -------
 
+* internal code changes to prepare for Python3
+
+* Improve user documentation of nametrans/folderfilter
+
 Bug Fixes
 ---------

docs/MANUAL.rst

@@ -329,17 +329,21 @@ Upgrading from plain text cache to SQLITE based cache
 OfflineImap uses a cache to store the last know status of mails (flags etc). Historically that has meant plain text files, but recently we introduced sqlite-based cache, which helps with performance and CPU usage on large folders. Here is how to upgrade existing plain text cache installations to sqlite based one:
 
 1) Sync to make sure things are reasonably similar
- 3) Change the account section to status_backend = sqlite
- 4) A new sync will convert your plain text cache to an sqlite cache (but
- leave the old plain text cache around for easy reverting)
-    This should be quick and not involve any mail up/downloading.
- 5) See if it works :-)
- 6a) If it does not work, go back to the old version or set
-     status_backend=plain
- 6b) Or once you are sure it works, you can delete the
- .offlineimap/Account-foo/LocalStatus folder (the new cache will be in
- the LocalStatus-sqlite folder)
 
+2) Change the account section to status_backend = sqlite
+
+3) A new sync will convert your plain text cache to an sqlite cache
+   (but leave the old plain text cache around for easy reverting) This
+   should be quick and not involve any mail up/downloading.
+
+4) See if it works :-)
+
+5) If it does not work, go back to the old version or set
+   status_backend=plain
+
+6) Or, once you are sure it works, you can delete the
+   .offlineimap/Account-foo/LocalStatus folder (the new cache will be
+   in the LocalStatus-sqlite folder)
 
 Security and SSL
 ================
@@ -399,128 +403,10 @@ accounts will abort any current sleep and will exit after a currently running
 synchronization has finished. This signal can be used to gracefully exit out of
 a running offlineimap "daemon".
 
-Folder filtering and Name translation
+Folder filtering and nametrans
-=====================================
+==============================
 
-OfflineImap provides advanced and potentially complex possibilities for
+OfflineImap offers flexible (and complex) ways of filtering and transforming folder names. Please see the docs/dev-docs-src/folderfilters.rst document about details how to use folder filters and name transformations. The documentation will be autogenerated by a "make dev-doc" in the docs directory. It is also viewable at :ref:`folder_filtering_and_name_translation`.
-filtering and translating folder names. If you don't need this, you can
-safely skip this section.
-
-folderfilter
-------------
-
-If you do not want to synchronize all your filters, you can specify a folderfilter function that determines which folders to include in a sync and which to exclude. Typically, you would set a folderfilter option on the remote repository only, and it would be a lambda or any other python function.
-
-If the filter function returns True, the folder will be synced, if it
-returns False, it. The folderfilter operates on the *UNTRANSLATED* name
-(before any nametrans translation takes place).
-
-Example 1: synchronizing only INBOX and Sent::
-
-   folderfilter = lambda foldername: foldername in ['INBOX', 'Sent']
-
-Example 2: synchronizing everything except Trash::
-
-   folderfilter = lambda foldername: foldername not in ['Trash']
-
-Example 3: Using a regular expression to exclude Trash and all folders
-containing the characters "Del"::
-
-    folderfilter = lambda foldername: not re.search('(^Trash$|Del)', foldername)
-
-If folderfilter is not specified, ALL remote folders will be
-synchronized.
-
-You can span multiple lines by indenting the others.  (Use backslashes
-at the end when required by Python syntax)  For instance::
-
- folderfilter = lambda foldername: foldername in
-        ['INBOX', 'Sent Mail', 'Deleted Items',
-         'Received']
-
-You only need a folderfilter option on the local repository if you want to prevent some folders on the local repository to be created on the remote one.
-
-Even if you filtered out folders, You can specify folderincludes to
-include additional folders.  It should return a Python list.  This might
-be used to include a folder that was excluded by your folderfilter rule,
-to include a folder that your server does not specify with its LIST
-option, or to include a folder that is outside your basic reference. The
-'reference' value will not be prefixed to this folder name, even if you
-have specified one.  For example::
-
-   folderincludes = ['debian.user', 'debian.personal']
-
-nametrans
-----------
-
-Sometimes, folders need to have different names on the remote and the
-local repositories. To achieve this you can specify a folder name
-translator.  This must be a eval-able Python expression that takes a
-foldername arg and returns the new value.  I suggest a lambda.  This
-example below will remove "INBOX." from the leading edge of folders
-(great for Courier IMAP users)::
-
-   nametrans = lambda foldername: re.sub('^INBOX\.', '', foldername)
-
-Using Courier remotely and want to duplicate its mailbox naming
-locally?  Try this::
-
-   nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
-
-
-WARNING: you MUST construct nametrans rules such that it NEVER returns
-the same value for two folders, UNLESS the second values are
-filtered out by folderfilter below. That is, two filters on one side may never point to the same folder on the other side. Failure to follow this rule
-will result in undefined behavior. See also *Sharing a maildir with multiple IMAP servers* in the `PITFALLS & ISSUES`_ section.
-
-Where to put nametrans rules, on the remote and/or local repository?
-++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-
-If you never intend to create new folders on the LOCAL repository that
-need to be synced to the REMOTE repository, it is sufficient to create a
-nametrans rule on the remote Repository section. This will be used to
-determine the names of new folder names on the LOCAL repository, and to
-match existing folders that correspond.
-
-*IF* you create folders on the local repository, that are supposed to be
- automatically created on the remote repository, you will need to create
- a nametrans rule that provides the reverse name translation.
-
-(A nametrans rule provides only a one-way translation of names and in
-order to know which names folders on the LOCAL side would have on the
-REMOTE side, you need to specify the reverse nametrans rule on the local
-repository)
-
-OfflineImap will complain if it needs to create a new folder on the
-remote side and a back-and-forth nametrans-lation does not yield the
-original foldername (as that could potentially lead to infinite folder
-creation cycles).
-
-What folder separators do I need to use in nametrans rules?
-+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-
-**Q:** If I sync from an IMAP server with folder separator '/' to a
-  Maildir using the default folder separator '.' which do I need to use
-  in nametrans rules?::
-
-  nametrans = lambda f: "INBOX/" + f
-or::
-  nametrans = lambda f: "INBOX." + f
-
-**A:** Generally use the folder separator as defined in the repository
-  you write the nametrans rule for. That is, use '/' in the above
-  case. We will pass in the untranslated name of the IMAP folder as
-  parameter (here `f`). The translated name will ultimately have all
-  folder separators be replaced with the destination repositories'
-  folder separator.
-
-So if 'f' was "Sent", the first nametrans yields the translated name
-"INBOX/Sent" to be used on the other side. As that repository uses the
-folder separator '.' rather than '/', the ultimate name to be used will
-be "INBOX.Sent".
-
-(As a final note, the smart will see that both variants of the above
-nametrans rule would have worked identically in this case)
 
 KNOWN BUGS
 ==========
@@ -566,6 +452,7 @@ KNOWN BUGS
    * Use cygwin managed mount (not tested)
       - not available anymore since cygwin 1.7
 
+.. _pitfalls:
 
 PITFALLS & ISSUES
 =================

docs/dev-doc-src/index.rst

@@ -17,6 +17,7 @@ More information on specific topics can be found on the following pages:
 **User documentation**
   * :doc:`installation/uninstall <INSTALL>`
   * :doc:`user manual/Configuration <MANUAL>`
+  * :doc:`Folder filtering & name transformation guide <nametrans>`
   * :doc:`command line options <offlineimap>`
   * :doc:`Frequently Asked Questions <FAQ>`
 
@@ -29,6 +30,7 @@ More information on specific topics can be found on the following pages:
 
    INSTALL
    MANUAL
+   nametrans
    offlineimap
    FAQ
 

docs/dev-doc-src/nametrans.rst

@@ -0,0 +1,182 @@
+.. _folder_filtering_and_name_translation:
+
+Folder filtering and Name translation
+=====================================
+
+OfflineImap provides advanced and potentially complex possibilities for
+filtering and translating folder names. If you don't need any of this, you can
+safely skip this section.
+
+.. warning::
+   Starting with v6.4.0, OfflineImap supports the creation of folders on the remote repostory. This change means that people that only had a nametrans option on the remote repository (everyone) will need to have a nametrans setting on the local repository too that will reverse the name transformation. See section `Reverse nametrans`_ for details.
+
+folderfilter
+------------
+
+If you do not want to synchronize all your filters, you can specify a `folderfilter`_ function that determines which folders to include in a sync and which to exclude. Typically, you would set a folderfilter option on the remote repository only, and it would be a lambda or any other python function.
+
+The only parameter to that function is the folder name. If the filter
+function returns True, the folder will be synced, if it returns False,
+it. will be skipped. The folderfilter operates on the *UNTRANSLATED*
+name (before any `nametrans`_ fudging takes place). Consider the
+examples below to get an idea of what they do.
+
+Example 1: synchronizing only INBOX and Sent::
+
+   folderfilter = lambda folder: folder in ['INBOX', 'Sent']
+
+Example 2: synchronizing everything except Trash::
+
+   folderfilter = lambda folder: folder not in ['Trash']
+
+Example 3: Using a regular expression to exclude Trash and all folders
+containing the characters "Del"::
+
+    folderfilter = lambda folder: not re.search('(^Trash$|Del)', folder)
+
+.. note::
+   If folderfilter is not specified, ALL remote folders will be
+   synchronized.
+
+You can span multiple lines by indenting the others.  (Use backslashes
+at the end when required by Python syntax)  For instance::
+
+ folderfilter = lambda foldername: foldername in
+        ['INBOX', 'Sent Mail', 'Deleted Items',
+         'Received']
+
+Usually it suffices to put a `folderfilter`_ setting in the remote repository section. You might want to put a folderfilter option on the local repository if you want to prevent some folders on the local repository to be created on the remote one. (Even in this case, folder filters on the remote repository will prevent that)
+
+folderincludes
+^^^^^^^^^^^^^^
+
+You can specify `folderincludes`_ to manually include additional folders to be synced, even if they had been filtered out by a folderfilter setting. `folderincludes`_ should return a Python list.
+
+This can be used to 1) add a folder that was excluded by your
+folderfilter rule, 2) to include a folder that your server does not specify
+with its LIST option, or 3) to include a folder that is outside your basic
+`reference`. The `reference` value will not be prefixed to this folder
+name, even if you have specified one.  For example::
+
+   folderincludes = ['debian.user', 'debian.personal']
+
+This will add the "debian.user" and "debian.personal" folders even if you
+have filtered out everything starting with "debian" in your folderfilter
+settings.
+
+
+nametrans
+----------
+
+Sometimes, folders need to have different names on the remote and the
+local repositories. To achieve this you can specify a folder name
+translator.  This must be a eval-able Python expression that takes a
+foldername arg and returns the new value.  We suggest a lambda function,
+but it could be any python function really. If you use nametrans rules, you will need to set them both on the remote and the local repository, see `Reverse nametrans`_ just below for details. The following examples are thought to be put in the remote repository section.
+
+The below will remove "INBOX." from the leading edge of folders (great
+for Courier IMAP users)::
+
+   nametrans = lambda folder: re.sub('^INBOX\.', '', folder)
+
+Using Courier remotely and want to duplicate its mailbox naming
+locally?  Try this::
+
+   nametrans = lambda folder: re.sub('^INBOX\.*', '.', folder)
+
+.. warning::
+    You MUST construct nametrans rules such that it NEVER returns the
+    same value for two folders, UNLESS the second values are filtered
+    out by folderfilter below. That is, two filters on one side may
+    never point to the same folder on the other side. Failure to follow
+    this rule will result in undefined behavior. See also *Sharing a
+    maildir with multiple IMAP servers* in the :ref:`pitfalls` section.
+
+Reverse nametrans
+^^^^^^^^^^^^^^^^^^
+
+Since 6.4.0, OfflineImap supports the creation of folders on the remote repository and that complicates things. Previously, only one nametrans setting on the remote repository was needed and that transformed a remote to a local name. However, nametrans transformations are one-way, and OfflineImap has no way using those rules on the remote repository to back local names to remote names.
+
+Take a remote nametrans rule `lambda f: re.sub('^INBOX/','',f)` which cuts of any existing INBOX prefix. Now, if we parse a list of local folders, finding e.g. a folder "Sent", is it supposed to map to "INBOX/Sent" or to "Sent"? We have no way of knowing. This is why **every nametrans setting on a remote repository requires an equivalent nametrans rule on the local repository that reverses the transformation**.
+
+Take the above examples. If your remote nametrans setting was::
+
+   nametrans = lambda folder: re.sub('^INBOX\.', '', folder)
+
+then you will want to have this in your local repository, prepending "INBOX" to any local folder name::
+
+   nametrans = lambda folder: 'INBOX' + folder
+
+Failure to set the local nametrans rule will lead to weird-looking error messages of -for instance- this type::
+
+  ERROR: Creating folder moo.foo on repository remote
+  Folder 'moo.foo'[remote] could not be created. Server responded: ('NO', ['Unknown namespace.'])
+
+(This indicates that you attempted to create a folder "Sent" when all remote folders needed to be under the prefix of "INBOX.").
+
+OfflineImap will make some sanity checks if it needs to create a new
+folder on the remote side and a back-and-forth nametrans-lation does not
+yield the original foldername (as that could potentially lead to
+infinite folder creation cycles).
+
+You can probably already see now that creating nametrans rules can be a pretty daunting and complex endeavour. Check out the Use cases in the manual. If you have some interesting use cases that we can present as examples here, please let us know.
+
+Debugging folderfilter and nametrans
+------------------------------------
+
+Given the complexity of the functions and regexes involved, it is easy to misconfigure things. One way to test your configuration without danger to corrupt anything or to create unwanted folders is to invoke offlineimap with the `--info` option.
+
+It will output a list of folders and their transformations on the screen (save them to a file with -l info.log), and will help you to tweak your rules as well as to understand your configuration. It also provides good output for bug reporting.
+
+FAQ on nametrans
+----------------
+
+Where to put nametrans rules, on the remote and/or local repository?
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+If you never intend to create new folders on the LOCAL repository that
+need to be synced to the REMOTE repository, it is sufficient to create a
+nametrans rule on the remote Repository section. This will be used to
+determine the names of new folder names on the LOCAL repository, and to
+match existing folders that correspond.
+
+*IF* you create folders on the local repository, that are supposed to be
+ automatically created on the remote repository, you will need to create
+ a nametrans rule that provides the reverse name translation.
+
+(A nametrans rule provides only a one-way translation of names and in
+order to know which names folders on the LOCAL side would have on the
+REMOTE side, you need to specify the reverse nametrans rule on the local
+repository)
+
+OfflineImap will complain if it needs to create a new folder on the
+remote side and a back-and-forth nametrans-lation does not yield the
+original foldername (as that could potentially lead to infinite folder
+creation cycles).
+
+What folder separators do I need to use in nametrans rules?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+**Q:** If I sync from an IMAP server with folder separator '/' to a
+  Maildir using the default folder separator '.' which do I need to use
+  in nametrans rules?::
+
+  nametrans = lambda f: "INBOX/" + f
+or::
+  nametrans = lambda f: "INBOX." + f
+
+**A:** Generally use the folder separator as defined in the repository
+  you write the nametrans rule for. That is, use '/' in the above
+  case. We will pass in the untranslated name of the IMAP folder as
+  parameter (here `f`). The translated name will ultimately have all
+  folder separators be replaced with the destination repositories'
+  folder separator.
+
+So if 'f' was "Sent", the first nametrans yields the translated name
+"INBOX/Sent" to be used on the other side. As that repository uses the
+folder separator '.' rather than '/', the ultimate name to be used will
+be "INBOX.Sent".
+
+(As a final note, the smart will see that both variants of the above
+nametrans rule would have worked identically in this case)
+

offlineimap.conf

@@ -1,11 +1,14 @@
 # Offlineimap sample configuration file
-# This file documents all possible options and can be quite scary.
-# Looking for a quick start?  Take a look at offlineimap.conf.minimal.
 
-# Settings generally support interpolation. This means values can
+# This file documents *all* possible options and can be quite scary.
-# contain python format strings which refer to other values in the same
+# Looking for a quick start?  Take a look at offlineimap.conf.minimal.
-# section, or values in a special DEFAULT section. This allows you for
+# More details can be found in the included user documention, which is
-# example to use common settings for multiple accounts:
+# also available at: http://docs.offlineimap.org/en/latest/
+
+# NOTE: Settings generally support python interpolation. This means
+# values can contain python format strings which refer to other values
+# in the same section, or values in a special DEFAULT section. This
+# allows you for example to use common settings for multiple accounts:
 #
 # [Repository Gmail1]
 # trashfolder: %(gmailtrashfolder)s
@@ -443,6 +446,10 @@ remoteuser = username
 # value.  I suggest a lambda.  This example below will remove "INBOX." from
 # the leading edge of folders (great for Courier IMAP users)
 #
+# See the user documentation for details and use cases. They are also
+# online at:
+# http://docs.offlineimap.org/en/latest/nametrans.html
+#
 # WARNING: you MUST construct this such that it NEVER returns
 # the same value for two folders, UNLESS the second values are
 # filtered out by folderfilter below.  Failure to follow this rule

offlineimap.conf.minimal

@@ -1,5 +1,5 @@
 # Sample minimal config file.  Copy this to ~/.offlineimaprc and edit to
-# suit to get started fast.
+# get started fast.
 
 [general]
 accounts = Test