Merge branch 'next'

Changelog.rst

@@ -15,6 +15,8 @@ OfflineIMAP v6.5.6.1 (YYYY-MM-DD)
 * Add missing version bump for 6.5.6 (it was released with
   6.5.5 in setup.py and other places).
 
+* Various fixes in documentation.
+
 
 OfflineIMAP v6.5.6 (2014-05-14)
 ===============================

README.md

@@ -39,18 +39,18 @@ detailed information on how to install and configure OfflineImap.
 Quick Start
 ===========
 
-First, install OfflineIMAP. See docs/INSTALL.rst or read
+First, install OfflineIMAP. See `docs/INSTALL.rst` or read
-http://docs.offlineimap.org/en/latest/INSTALL.html.
+<http://docs.offlineimap.org/en/latest/INSTALL.html>
-(hint: `sudo python setup.py install`)
+(hint: `sudo python setup.py install`).
 
 Second, set up your configuration file and run it! The distribution
 includes offlineimap.conf.minimal (Debian users may find this at
-``/usr/share/doc/offlineimap/examples/offlineimap.conf.minimal``) that
+`/usr/share/doc/offlineimap/examples/offlineimap.conf.minimal`) that
 provides you with the bare minimum of setting up OfflineIMAP.  You can
 simply copy this file into your home directory and name it
-``.offlineimaprc``.  A command such as ``cp offlineimap.conf.minimal
+`.offlineimaprc`.  A command such as `cp offlineimap.conf.minimal
-~/.offlineimaprc`` will do it.  Or, if you prefer, you can just copy
+~/.offlineimaprc` will do it.  Or, if you prefer, you can just copy
-this text to ``~/.offlineimaprc``:
+this text to `~/.offlineimaprc`:
 
     [general]
     accounts = Test
@@ -69,17 +69,17 @@ this text to ``~/.offlineimaprc``:
     remoteuser = jgoerzen
 
 
-Now, edit the ``~/.offlineimaprc`` file with your favorite editor.  All you have
+Now, edit the `~/.offlineimaprc` file with your favorite editor.  All you have
-to do is specify a directory for your folders to be in (on the localfolders
+to do is specify a directory for your folders to be in (on the `localfolders`
-line), the host name of your IMAP server (on the remotehost line), and your
+line), the host name of your IMAP server (on the `remotehost` line), and your
-login name on the remote (on the remoteuser line).  That's it!
+login name on the remote (on the `remoteuser` line).  That's it!
 
-If you prefer to be XDG-compatible,
+If you prefer to be compatible with the [XDG Base Directory
-  http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
+spec](http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html),
-then substitute the above ``~/.offlineimaprc'' with
+then substitute the above `~/.offlineimaprc` with
-``$XDG\_CONFIG\_HOME/offlineimap/config'' and don't forget to set
+`$XDG_CONFIG_HOME/offlineimap/config` and don't forget to set
-XDG\_CONFIG\_HOME properly if you want it to be different from
+`XDG_CONFIG_HOME` properly if you want it to be different from
-the default ``$HOME/.config'' for any reason.
+the default `$HOME/.config` for any reason.
 
 To run OfflineIMAP, you just have to say `offlineimap` ― it will fire
 up, ask you for a login password if necessary, synchronize your folders,
@@ -97,12 +97,12 @@ Mailing list & bug reporting
 
 The user discussion, development and all exciting stuff take place in the
 OfflineImap mailing list at
-http://lists.alioth.debian.org/mailman/listinfo/offlineimap-project. You do not
+<http://lists.alioth.debian.org/mailman/listinfo/offlineimap-project>. You do not
 need to subscribe to send emails.
 
 Bugs, issues and contributions should be reported to the mailing list. Bugs can
 also be reported in the issue tracker at
-https://github.com/OfflineIMAP/offlineimap/issues.
+<https://github.com/OfflineIMAP/offlineimap/issues>.
 
 Configuration Examples
 ======================
@@ -117,22 +117,22 @@ Multiple Accounts with Mutt
 This example shows you how to set up OfflineIMAP to synchronize multiple
 accounts with the mutt mail reader.
 
-Start by creating a directory to hold your folders by running ``mkdir ~/Mail``.
+Start by creating a directory to hold your folders by running `mkdir ~/Mail`.
-Then, in your ``~/.offlineimaprc``, specify:
+Then, in your `~/.offlineimaprc`, specify:
 
     accounts = Personal, Work
 
 
-Make sure that you have both an [Account Personal] and an [Account Work]
+Make sure that you have both an `[Account Personal]` and an `[Account Work]`
-section.  The local repository for each account must have different localfolder
+section.  The local repository for each account must have different `localfolder`
-path names.  Also, make sure to enable [mbnames].
+path names.  Also, make sure to enable `[mbnames]`.
 
 In each local repository section, write something like this:
 
     localfolders = ~/Mail/Personal
 
 
-Finally, add these lines to your ``~/.muttrc``:
+Finally, add these lines to your `~/.muttrc`:
 
     source ~/path-to-mbnames-muttrc-mailboxes
     folder-hook Personal set from="youremail@personal.com"
@@ -149,10 +149,10 @@ UW-IMAPD and References
 -----------------------
 
 Some users with a UW-IMAPD server need to use OfflineIMAP's "reference" feature
-to get at their mailboxes, specifying a reference of ``~/Mail`` or ``#mh/``
+to get at their mailboxes, specifying a reference of `~/Mail` or `#mh/`
 depending on the configuration.  The below configuration from (originally from
-docwhat@gerf.org) shows using a reference of Mail, a nametrans that strips the
+docwhat@gerf.org) shows using a reference of Mail, a `nametrans` that strips the
-leading Mail/ off incoming folder names, and a folderfilter that limits the
+leading `Mail/` off incoming folder names, and a `folderfilter` that limits the
 folders synced to just three:
 
     [Account Gerf]
@@ -191,16 +191,16 @@ configuration file options that are Python expressions.  This example is based
 on one supplied by Tommi Virtanen for this feature.
 
 
-In ~/.offlineimaprc, he adds these options:
+In `~/.offlineimaprc`, he adds these options:
 
     [general]
     pythonfile=~/.offlineimap.py
     [Repository foo]
     foldersort=mycmp
 
-Then, the ~/.offlineimap.py file will contain:
+Then, the `~/.offlineimap.py` file will contain:
 
-  prioritized = ['INBOX', 'personal', 'announce', 'list']
+    prioritized = ['INBOX', 'personal', 'announce', 'list']
 
     def mycmp(x, y):
         for prefix in prioritized:
@@ -221,5 +221,5 @@ Then, the ~/.offlineimap.py file will contain:
         print folders
 
 
-This code snippet illustrates how the foldersort option can be customized with a
+This code snippet illustrates how the `foldersort` option can be customized with a
-Python function from the pythonfile to always synchronize certain folders first.
+Python function from the `pythonfile` to always synchronize certain folders first.

offlineimap/folder/Base.py

@@ -386,69 +386,93 @@ class BaseFolder(object):
         for uid in uidlist:
             self.deletemessagelabels(uid, labels)
 
-
+    def addmessageheader(self, content, linebreak, headername, headervalue):
-    """
-    Illustration of all cases for addmessageheader().
-    '+' means the added contents.
-
-Case 1: No '\n\n', leading '\n'
-+X-Flying-Pig-Header: i am here\n
-\n
-This is the body\n
-next line\n
-
-Case 2: '\n\n' at position 0
-+X-Flying-Pig-Header: i am here\n
-\n
-\n
-This is the body\n
-next line\n
-
-Case 3: No '\n\n', no leading '\n'
-+X-Flying-Pig-Header: i am here\n
-+\n
-This is the body\n
-next line\n
-
-Case 4: '\n\n' at non-zero position
-Subject: Something wrong with OI\n
-From: some@person.at+\n
-X-Flying-Pig-Header: i am here\n <-- orig '\n'
-\n
-This is the body\n
-next line\n
-
-    """
-
-    def addmessageheader(self, content, crlf, headername, headervalue):
         """
         Adds new header to the provided message.
 
         Arguments:
         - content: message content, headers and body as a single string
-        - crlf: string that carries line ending
+        - linebreak: string that carries line ending
         - headername: name of the header to add
         - headervalue: value of the header to add
 
+        This has to deal with strange corner cases where the header is
+        missing or empty.  Here are illustrations for all the cases,
+        showing where the header gets inserted and what the end result
+        is.  In each illustration, '+' means the added contents.  Note
+        that these examples assume LF for linebreak, not CRLF, so '\n'
+        denotes a linebreak and '\n\n' corresponds to the transition
+        between header and body.  However if the linebreak parameter
+        is set to '\r\n' then you would have to substitute '\r\n' for
+        '\n' in the below examples.
+
+          * Case 1: No '\n\n', leading '\n'
+
+            +X-Flying-Pig-Header: i am here\n
+            \n
+            This is the body\n
+            next line\n
+
+          * Case 2: '\n\n' at position 0
+
+            +X-Flying-Pig-Header: i am here
+            \n
+            \n
+            This is the body\n
+            next line\n
+
+          * Case 3: No '\n\n', no leading '\n'
+
+            +X-Flying-Pig-Header: i am here\n
+            +\n
+            This is the body\n
+            next line\n
+
+          * Case 4: '\n\n' at non-zero position
+
+            Subject: Something wrong with OI\n
+            From: some@person.at
+            +\nX-Flying-Pig-Header: i am here
+            \n
+            \n
+            This is the body\n
+            next line\n
         """
         self.ui.debug('',
                  'addmessageheader: called to add %s: %s' % (headername,
                                                              headervalue))
-        prefix = crlf
+
-        suffix = ''
+        insertionpoint = content.find(linebreak * 2)
-        insertionpoint = content.find(crlf + crlf)
-        if insertionpoint == 0 or insertionpoint == -1:
-            prefix = ''
-            suffix = crlf
         if insertionpoint == -1:
+            self.ui.debug('', 'addmessageheader: headers were missing')
+        else:
+            self.ui.debug('', 'addmessageheader: headers end at position %d' % insertionpoint)
+            mark = '==>EOH<=='
+            contextstart = max(0,            insertionpoint - 100)
+            contextend   = min(len(content), insertionpoint + 100)
+            self.ui.debug('', 'addmessageheader: header/body transition context (marked by %s): %s' %
+                          (mark, repr(content[contextstart:insertionpoint]) + \
+                          mark + repr(content[insertionpoint:contextend])))
+
+        # Hoping for case #4
+        prefix = linebreak
+        suffix = ''
+        # Case #2
+        if insertionpoint == 0:
+            prefix = ''
+            suffix = ''
+        # Either case #1 or #3
+        elif insertionpoint == -1:
+            prefix = ''
+            suffix = linebreak
             insertionpoint = 0
-            # When body starts immediately, without preceding '\n'
+            # Case #3: when body starts immediately, without preceding '\n'
             # (this shouldn't happen with proper mail messages, but
             # we seen many broken ones), we should add '\n' to make
             # new (and the only header, in this case) to be properly
             # separated from the message body.
-            if content[0:len(crlf)] != crlf:
+            if content[0:len(linebreak)] != linebreak:
-                suffix = suffix + crlf
+                suffix = suffix + linebreak
 
         self.ui.debug('', 'addmessageheader: insertionpoint = %d' % insertionpoint)
         headers = content[0:insertionpoint]
@@ -478,6 +502,8 @@ next line\n
     def getmessageheader(self, content, name):
         """
         Searches for the given header and returns its value.
+        Header name is case-insensitive.
+
         Arguments:
         - contents: message itself
         - name: name of the header to be searched
@@ -491,7 +517,7 @@ next line\n
         headers = content[0:eoh]
         self.ui.debug('', 'getmessageheader: headers = %s' % repr(headers))
 
-        m = re.search('^%s:(.*)$' % name, headers, flags = re.MULTILINE)
+        m = re.search('^%s:(.*)$' % name, headers, flags = re.MULTILINE | re.IGNORECASE)
         if m:
             return m.group(1).strip()
         else:

offlineimap/folder/IMAP.py

@@ -517,6 +517,11 @@ class IMAPFolder(BaseFolder):
         # get the date of the message, so we can pass it to the server.
         date = self.__getmessageinternaldate(content, rtime)
 
+        # Message-ID is handy for debugging messages
+        msg_id = self.getmessageheader(content, "message-id")
+        if not msg_id:
+            msg_id = '[unknown message-id]'
+
         retry_left = 2 # succeeded in APPENDING?
         imapobj = self.imapserver.acquireconnection()
         # NB: in the finally clause for this try we will release
@@ -569,9 +574,9 @@ class IMAPFolder(BaseFolder):
                         # In this case, we should immediately abort the repository sync
                         # and continue with the next account.
                         msg = \
-                            "Saving msg in folder '%s', repository '%s' failed (abort). " \
+                            "Saving msg (%s) in folder '%s', repository '%s' failed (abort). " \
                             "Server responded: %s %s\n" % \
-                            (self, self.getrepository(), typ, dat)
+                            (msg_id, self, self.getrepository(), typ, dat)
                         raise OfflineImapError(msg, OfflineImapError.ERROR.REPO)
                     retry_left = 0 # Mark as success
                 except imapobj.abort as e:
@@ -580,10 +585,10 @@ class IMAPFolder(BaseFolder):
                     self.imapserver.releaseconnection(imapobj, True)
                     imapobj = self.imapserver.acquireconnection()
                     if not retry_left:
-                        raise OfflineImapError("Saving msg in folder '%s', "
+                        raise OfflineImapError("Saving msg (%s) in folder '%s', "
                               "repository '%s' failed (abort). Server responded: %s\n"
                               "Message content was: %s" %
-                              (self, self.getrepository(), str(e), dbg_output),
+                              (msg_id, self, self.getrepository(), str(e), dbg_output),
                                                OfflineImapError.ERROR.MESSAGE)
                     self.ui.error(e, exc_info()[2])
                 except imapobj.error as e: # APPEND failed
@@ -592,9 +597,9 @@ class IMAPFolder(BaseFolder):
                     # drop conn, it might be bad.
                     self.imapserver.releaseconnection(imapobj, True)
                     imapobj = None
-                    raise OfflineImapError("Saving msg folder '%s', repo '%s'"
+                    raise OfflineImapError("Saving msg (%s) folder '%s', repo '%s'"
                         "failed (error). Server responded: %s\nMessage content was: "
-                        "%s" % (self, self.getrepository(), str(e), dbg_output),
+                        "%s" % (msg_id, self, self.getrepository(), str(e), dbg_output),
                                            OfflineImapError.ERROR.MESSAGE)
             # Checkpoint. Let it write out stuff, etc. Eg searches for
             # just uploaded messages won't work if we don't do this.