docs/*: Documentation enhancements

Improve wordings and descriptions all around.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>

docs/FAQ.rst

@@ -49,50 +49,40 @@ increase performance of the sync.
 Don’t set the number too high. If you do that, things might actually slow down
 as your link gets saturated. Also, too many connections can cause mail servers
 to have excessive load. Administrators might take unkindly to this, and the
-server might bog down. There are many variables in the optimal setting;
+server might bog down. There are many variables in the optimal setting; experimentation may help.
-experimentation may help.
 
-An informal benchmark yields these results for my setup::
+See the Performance section in the MANUAL for some tips.
-
-    * 10 minutes with MacOS X Mail.app “manual cache”
-    * 5 minutes with GNUS agent sync
-    * 20 seconds with OfflineIMAP 1.x
-    * 9 seconds with OfflineIMAP 2.x
-    * 3 seconds with OfflineIMAP 3.x “cold start”
-    * 2 seconds with OfflineIMAP 3.x “held connection”
 
 What platforms does OfflineIMAP support?
 ----------------------------------------
 
-It should run on most platforms supported by Python, which are quite a few. I
+It should run on most platforms supported by Python, with one exception: we do not support Windows, but some have made it work there.
-do not support Windows myself, but some have made it work there.  Use on
-Windows
 
-These answers have been reported by OfflineIMAP users. I do not run OfflineIMAP
+The following has been reported by OfflineIMAP users. We do not test
-on Windows myself, so I can’t directly address their accuracy.
+OfflineIMAP on Windows, so we can’t directly address their accuracy.
 
 The basic answer is that it’s possible and doesn’t require hacking OfflineIMAP
 source code. However, it’s not necessarily trivial. The information below is
-based in instructions submitted by Chris Walker.
+based in instructions submitted by Chris Walker::
 
-First, you must run OfflineIMAP in the Cygwin environment. The Windows
+    First, you must run OfflineIMAP in the Cygwin environment. The Windows
-filesystem is not powerful enough to accomodate Maildir by itself.
+    filesystem is not powerful enough to accomodate Maildir by itself.
-
+    
-Next, you’ll need to mount your Maildir directory in a special way. There is
+    Next, you’ll need to mount your Maildir directory in a special
-information for doing that at http://barnson.org/node/295. That site gives this
+    way. There is information for doing that at
-example::
+    http://barnson.org/node/295. That site gives this example::
-
+    
-  mount -f -s -b -o managed "d:/tmp/mail" "/home/of/mail"
+      mount -f -s -b -o managed "d:/tmp/mail" "/home/of/mail"
-
+    
-That URL also has more details on making OfflineIMAP work with Windows.
+    That URL also has more details on making OfflineIMAP work with Windows.
 
 
 Does OfflineIMAP support mbox, mh, or anything else other than Maildir?
 -----------------------------------------------------------------------
 
-Not directly. Maildir was the easiest to implement. I’m not planning to write
+Not directly. Maildir was the easiest to implement. We are not planning
-mbox code for OfflineIMAP, though if someone sent me well-written mbox support
+to write an mbox-backend, though if someone sent me well-written mbox
-and pledged to support it, I’d commit it to the tree.
+support and pledged to support it, it would be committed it to the tree.
 
 However, OfflineIMAP can directly sync accounts on two different IMAP servers
 together. So you could install an IMAP server on your local machine that
@@ -105,22 +95,25 @@ point your mail readers to that IMAP server on localhost.
 What is the UID validity problem for folder?
 --------------------------------------------
 
-IMAP servers use a unique ID (UID) to refer to a specific message.  This number
+IMAP servers use a folders UIDVALIDITY value in combination with a
-is guaranteed to be unique to a particular message forever.  No other message in
+unique ID (UID) to refer to a specific message.  This is guaranteed to
-the same folder will ever get the same UID.  UIDs are an integral part of
+be unique to a particular message forever.  No other message in the same
-`OfflineIMAP`_'s synchronization scheme; they are used to match up messages on
+folder will ever get the same UID as long as UIDVALIDITY remains
-your computer to messages on the server.
+unchanged.  UIDs are an integral part of `OfflineIMAP`_'s
+synchronization scheme; they are used to match up messages on your
+computer to messages on the server.
 
-Sometimes, the UIDs on the server might get reset.  Usually this will happen if
+Sometimes, the UIDs on the server might get reset.  Usually this will
-you delete and then recreate a folder.  When you create a folder, the server
+happen if you delete and then recreate a folder.  When you create a
-will often start the UID back from 1.  But `OfflineIMAP`_ might still have the
+folder, the server will often start the UID back from 1.  But
-UIDs from the previous folder by the same name stored.  `OfflineIMAP`_ will
+`OfflineIMAP`_ might still have the UIDs from the previous folder by the
-detect this condition and skip the folder.  This is GOOD, because it prevents
+same name stored.  `OfflineIMAP`_ will detect this condition because of
-data loss.
+the changed UIDVALIDITY value and skip the folder.  This is GOOD,
+because it prevents data loss.
 
-You can fix it by removing your local folder and cache data.  For instance, if
+In the IMAP<->Maildir case, you can fix it by removing your local folder
-your folders are under `~/Folders` and the folder with the problem is INBOX,
+and cache data.  For instance, if your folders are under `~/Folders` and
-you'd type this::
+the folder with the problem is INBOX, you'd type this::
 
   rm -r ~/Folders/INBOX
   rm -r ~/.offlineimap/Account-AccountName/LocalStatus/INBOX
@@ -146,12 +139,10 @@ This question comes up frequently on the `mailing list`_.  You can find a detail
 discussion of the problem there
 http://lists.complete.org/offlineimap@complete.org/2003/04/msg00012.html.gz.
 
-How do I add or delete a folder?
+How do I automatically delete a folder?
---------------------------------
+---------------------------------------
 
-OfflineIMAP does not currently provide this feature. However, if you create a
+OfflineIMAP does not currently provide this feature. You will have to delete folders manually. See next entry too.
-new folder on the remote server, OfflineIMAP will detect this and create the
-corresponding folder locally automatically.
 
 May I delete local folders?
 ---------------------------
@@ -175,6 +166,7 @@ It will perform a check on startup and abort if another `OfflineIMAP`_ is
 already running.  If you need to schedule synchronizations, you'll probably
 find autorefresh settings more convenient than cron.  Alternatively, you can
 set a separate metadata directory for each instance.
+In the future, we will lock each account individually rather than having one global lock.
 
 Can I copy messages between folders?
 ---------------------------------------
@@ -197,10 +189,7 @@ next run it will upload the message to second server and delete on first, etc.
 Does OfflineIMAP support POP?
 -----------------------------
 
-No. POP is not robust enough to do a completely reliable multi-machine sync
+No.
-like OfflineIMAP can do.
-
-OfflineIMAP will never support POP.
 
 How is OfflineIMAP conformance?
 -------------------------------
@@ -214,17 +203,21 @@ How is OfflineIMAP conformance?
 Can I force OfflineIMAP to sync a folder right now?
 ---------------------------------------------------
 
-Yes, if you use the `Blinkenlights` UI.  That UI shows the active accounts
+Yes, 
+  1) if you use the `Blinkenlights` UI.  That UI shows the active accounts
 as follows::
 
-  4: [active]      *Control: .
+   4: [active]      *Control: .
-  3: [  4:36]      personal:
+   3: [  4:36]      personal:
-  2: [  3:37]          work: .
+   2: [  3:37]          work: .
-  1: [  6:28]           uni:
+   1: [  6:28]           uni:
 
-Simply press the appropriate digit (`3` for `personal`, etc.) to resync that
+   Simply press the appropriate digit (`3` for `personal`, etc.) to
-account immediately.  This will be ignored if a resync is already in progress
+   resync that account immediately.  This will be ignored if a resync is
-for that account.
+   already in progress for that account.
+
+  2) while in sleep mode, you can also send a SIGUSR1. See the `Signals
+    on UNIX`_ section in the MANUAL for details.
 
 Configuration Questions
 =======================
@@ -248,10 +241,12 @@ what folders are present on the IMAP server and synchronize them. You can use
 the folderfilter and nametrans configuration file options to request only
 certain folders and rename them as they come in if you like.
 
+Also you can configure OfflineImap to only synchronize "subscribed" folders.
+
 How do I prevent certain folders from being synced?
 ---------------------------------------------------
 
-Use the folderfilter option.
+Use the folderfilter option. See the MANUAL for details and examples.
 
 What is the mailbox name recorder (mbnames) for?
 ------------------------------------------------
@@ -261,9 +256,11 @@ Some mail readers, such as mutt, are not capable of automatically determining th
 Does OfflineIMAP verify SSL certificates?
 -----------------------------------------
 
-By default, no.  However, as of version 6.3.2, it is possible to enforce verification
+You can verify an imapserver's certificate by specifying the CA
-of SSL certificate on a per-repository basis by setting the `sslcacertfile` option in the
+certificate on a per-repository basis by setting the `sslcacertfile`
-config file.  (See the example offlineimap.conf for details.)
+option in the config file. (See the example offlineimap.conf for
+details.) If you do not specify any CA certificate, you will be presented with the server's certificate fingerprint and add that to the configuration file, to make sure it remains unchanged.
+No verification happens if connecting via STARTTLS.
 
 How do I generate an `sslcacertfile` file?
 ------------------------------------------
@@ -293,23 +290,6 @@ with the IMAP RFCs. Some servers provide imperfect compatibility that may be
 good enough for general clients. OfflineIMAP needs more features, specifically
 support for UIDs, in order to do its job accurately and completely.
 
-Microsoft Exchange
-------------------
-
-Several users have reported problems with Microsoft Exchange servers in
-conjunction with OfflineIMAP. This generally seems to be related to the
-Exchange servers not properly following the IMAP standards.
-
-Mark Biggers has posted some information to the OfflineIMAP `mailing list`_
-about how he made it work.
-
-Other users have indicated that older (5.5) releases of Exchange are so bad
-that they will likely not work at all.
-
-I do not have access to Exchange servers for testing, so any problems with it,
-if they can even be solved at all, will require help from OfflineIMAP users to
-find and fix.
-
 
 Client Notes
 ============
@@ -494,12 +474,13 @@ To send a patch, we recommend using 'git send-email'.
 Where from should my patches be based on?
 -----------------------------------------
 
-Depends. If you're not sure, it should start off of the master branch. master is
+Depends. If you're not sure, it should start off of the master
-the branch where new patches should be based on by default.
+branch. master is the branch where new patches should be based on by
+default.
 
-Obvious materials for next release (e.g. new features) start off of current
+Obvious materials for next release (e.g. new features) start off of
-next.  Also, next is the natural branch to write patches on top of commits not
+current next.  Also, next is the natural branch to write patches on top
-already in master.
+of commits not already in master.
 
 A fix for a very old bug or security issue may start off of maint. This isn't
 needed since such fix are backported by the maintainer, though.

docs/MANUAL.rst

@@ -6,7 +6,7 @@
 Powerful IMAP/Maildir synchronization and reader support
 --------------------------------------------------------
 
-:Author: John Goerzen <jgoerzen@complete.org>
+:Author: John Goerzen <jgoerzen@complete.org> & contributors
 :Date: 2011-01-15
 :Copyright: GPL v2
 :Manual section: 1
@@ -22,10 +22,8 @@ emails between them, so that you can read the same mailbox from multiple
 computers. The REMOTE repository is some IMAP server, while LOCAL can be
 either a local Maildir or another IMAP server.
 
-Missing folders will be automatically created on the LOCAL side, however
+Missing folders will be automatically created on both sides if
-NO folders will currently be created on the REMOTE repository
+needed. No folders will be deleted at the moment.
-automatically (it will sync your emails from local folders if
-corresponding REMOTE folders already exist).
 
 Configuring OfflineImap in basic mode is quite easy, however it provides
 an amazing amount of flexibility for those with special needs.  You can
@@ -159,12 +157,10 @@ Blinkenlights
 Blinkenlights is an interface designed to be sleek, fun to watch, and
 informative of the overall picture of what OfflineIMAP is doing.
 
-
 Blinkenlights contains a row of "LEDs" with command buttons and a log.
 The  log shows more detail about what is happening and is color-coded to match
 the color of the lights.
 
-
 Each light in the Blinkenlights interface represents a thread of execution --
 that is, a particular task that OfflineIMAP is performing right now.  The colors
 indicate what task the particular thread is performing, and are as follows:
@@ -232,7 +228,7 @@ English-speaking world. One version ran in its entirety as follows:
 
 
 TTYUI
----------
+------
 
 TTYUI interface is for people running in terminals.  It prints out basic
 status messages and is generally friendly to use on a console or xterm.
@@ -245,7 +241,7 @@ Basic is designed for situations in which OfflineIMAP will be run
 non-attended and the status of its execution will be logged.  This user
 interface is not capable of reading a password from the keyboard;
 account passwords must be specified using one of the configuration file
-options.
+options. For example, it will not print periodic sleep announcements and tends to be a tad less verbose, in general.
 
 
 Quiet
@@ -367,6 +363,107 @@ accounts will abort any current sleep and will exit after a currently running
 synchronization has finished. This signal can be used to gracefully exit out of
 a running offlineimap "daemon".
 
+Folder filtering and Name translation
+=====================================
+
+OfflineImap provides advanced and potentially complex possibilities for
+filtering and translating folder names. If you don't need this, you can
+safely skip this section.
+
+folderfilter
+------------
+
+If you do not want to synchronize all your filters, you can specify a folderfilter function that determines which folders to include in a sync and which to exclude. Typically, you would set a folderfilter option on the remote repository only, and it would be a lambda or any other python function.
+
+If the filter function returns True, the folder will be synced, if it
+returns False, it. The folderfilter operates on the *UNTRANSLATED* name
+(before any nametrans translation takes place).
+
+Example 1: synchronizing only INBOX and Sent::
+
+   folderfilter = lambda foldername: foldername in ['INBOX', 'Sent']
+
+Example 2: synchronizing everything except Trash::
+
+   folderfilter = lambda foldername: foldername not in ['Trash']
+
+Example 3: Using a regular expression to exclude Trash and all folders
+containing the characters "Del"::
+
+    folderfilter = lambda foldername: not re.search('(^Trash$|Del)', foldername)
+
+If folderfilter is not specified, ALL remote folders will be
+synchronized.
+
+You can span multiple lines by indenting the others.  (Use backslashes
+at the end when required by Python syntax)  For instance::
+
+ folderfilter = lambda foldername: foldername in
+        ['INBOX', 'Sent Mail', 'Deleted Items',
+         'Received']
+
+You only need a folderfilter option on the local repository if you want to prevent some folders on the local repository to be created on the remote one.
+
+Even if you filtered out folders, You can specify folderincludes to
+include additional folders.  It should return a Python list.  This might
+be used to include a folder that was excluded by your folderfilter rule,
+to include a folder that your server does not specify with its LIST
+option, or to include a folder that is outside your basic reference. The
+'reference' value will not be prefixed to this folder name, even if you
+have specified one.  For example::
+
+   folderincludes = ['debian.user', 'debian.personal']
+
+nametrans
+----------
+
+Sometimes, folders need to have different names on the remote and the
+local repositories. To achieve this you can specify a folder name
+translator.  This must be a eval-able Python expression that takes a
+foldername arg and returns the new value.  I suggest a lambda.  This
+example below will remove "INBOX." from the leading edge of folders
+(great for Courier IMAP users)::
+
+   nametrans = lambda foldername: re.sub('^INBOX\.', '', foldername)
+
+Using Courier remotely and want to duplicate its mailbox naming
+locally?  Try this::
+
+   nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)
+
+
+WARNING: you MUST construct nametrans rules such that it NEVER returns
+the same value for two folders, UNLESS the second values are
+filtered out by folderfilter below. That is, two filters on one side may never point to the same folder on the other side. Failure to follow this rule
+will result in undefined behavior. See also *Sharing a maildir with multiple IMAP servers* in the `PITFALLS & ISSUES`_ section.
+
+Where to put nametrans rules, on the remote and/or local repository?
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+If you never intend to create new folders on the LOCAL repository that need to be synced to the REMOTE repository, it is sufficient to create a nametrans rule on the remote Repository section. This will be used to determine the names of new folder names on the LOCAL repository, and to match existing folders that correspond.
+
+*IF* you create folders on the local repository, that are supposed to be automatically created on the remote repository, you will need to create a nametrans rule that provides the reverse name translation.
+
+(A nametrans rule provides only a one-way translation of names and in order to know which names folders on the LOCAL side would have on the REMOTE side, you need to specify the reverse nametrans rule on the local repository)
+
+OfflineImap will complain if it needs to create a new folder on the
+remote side and a back-and-forth nametrans-lation does not yield the
+original foldername (as that could potentially lead to infinite folder
+creation cycles).
+
+What folder separators do I need to use in nametrans rules?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+**Q:** If I sync from an IMAP server with folder separator '/' to a Maildir using the default folder separator '.' which do I need to use in nametrans rules?::
+  nametrans = lambda f: "INBOX/" + f
+or::
+  nametrans = lambda f: "INBOX." + f
+
+**A:** Generally use the folder separator as defined in the repository you write the nametrans rule for. That is, use '/' in the above case. We will pass in the untranslated name of the IMAP folder as parameter (here `f`). The translated name will ultimately have all folder separators be replaced with the destination repositories' folder separator.
+
+So if ̀f` was "Sent", the first nametrans yields the translated name "INBOX/Sent" to be used on the other side. As that repository uses the folder separator '.' rather than '/', the ultimate name to be used will be "INBOX.Sent".
+
+(As a final note, the smart will see that both variants of the above nametrans rule would have worked identically in this case)
 
 KNOWN BUGS
 ==========