Make GmailFolder sync GMail labels

When synclabels config flag is set to "yes" for the GMail repo,
offlineimap fetches the message labels along with the messages, and
embeds them into the body under the header X-Keywords (or whatever
'labelsheader' was set to), as a comma separated list.

It also adds an extra pass to savemessageto, that performs label
synchronization on existing messages from GMail to local, the same way
it is done with flags.

We also introduce GmailMaildir repository that adds functionality to
change message labels.  It keeps track of messages modification time,
so one can quickly detect when the labels may have changed.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>

Changelog.rst

@@ -12,6 +12,8 @@ OfflineIMAP v6.5.6 (YYYY-MM-DD)
 * Add knob to apply compression to IMAP connections (Abdó Roig-Maranges)
 * Add knob to filter some headers before uploading message
   to IMAP server (Abdó Roig-Maranges)
+* Allow to sync GMail labels and implement GmailMaildir repository that
+  adds mechanics to change message labels (Abdó Roig-Maranges)
 
 
 OfflineIMAP v6.5.5 (2013-10-07)

docs/MANUAL.rst

@@ -429,6 +429,41 @@ This is an example of a setup where "TheOtherImap" requires all folders to be un
     remoteuser = XXX
     #Do not use nametrans here.
 
+
+Sync from Gmail to a local Maildir with labels
+----------------------------------------------
+
+This is an example of a setup where GMail gets synced with a local Maildir.
+It also keeps track of GMail labels, that get embedded into the messages
+under the header X-Keywords (or whatever labelsheader is set to), and syncs
+them back and forth the same way as flags.
+
+The first time it runs on a large repository may take some time as the labels
+are read / embedded on every message.  Afterwards local label changes are detected
+using modification times (much faster)::
+
+    [Account Gmail-mine]
+    localrepository = Gmaillocal-mine
+    remoterepository = Gmailserver-mine
+    # Need this to be able to sync labels
+    status_backend = sqlite
+    synclabels = yes
+    # This header is where labels go.  Usually you will be fine
+    # with default value, but in case you want it different,
+    # here we go:
+    labelsheader = X-GMail-Keywords
+
+    [Repository Gmailserver-mine]
+    #This is the remote repository
+    type = Gmail
+    remotepass = XXX
+    remoteuser = XXX
+
+    [Repository Gmaillocal-mine]
+    #This is the 'local' repository
+    type = GmailMaildir
+
+
 Selecting only a few folders to sync
 ------------------------------------
 Add this to the remote gmail repository section to only sync mails which are in a certain folder::

offlineimap.conf

@@ -271,6 +271,22 @@ remoterepository = RemoteExample
 #
 #maildir-windows-compatible = no
 
+# Specifies if we want to sync GMail lables with the local repository.
+# Effective only for GMail IMAP repositories.  You should use SQlite
+# backend for this to work (see status_backend).
+#
+#synclabels = no
+
+# Name of the header to use for label storage.
+#
+#labelsheader = X-Keywords
+
+# Set of labels to be ignored.  Comma-separated list.  GMail-specific
+# labels all start with backslash ('\').
+#
+#ignorelabels = \Inbox, \Starred, \Sent, \Draft, \Spam, \Trash, \Important
+
+
 
 # OfflineIMAP can strip off some headers when your messages are propagated
 # back to the IMAP server.  This option carries the comma-separated list
@@ -283,10 +299,11 @@ remoterepository = RemoteExample
 #filterheaders = X-Some-Weird-Header
 
 
+
 [Repository LocalExample]
 
 # Each repository requires a "type" declaration. The types supported for
-# local repositories are Maildir and IMAP.
+# local repositories are Maildir, GmailMaildir and IMAP.
 
 type = Maildir
 
@@ -313,6 +330,24 @@ localfolders = ~/Test
 #restoreatime = no
 
 
+
+[Repository GmailLocalExample]
+
+# This type of repository enables syncing of Gmail. All Maildir
+# configuration settings are also valid here.
+#
+# This is a separate Repository type from Maildir because it involves
+# some extra overhead which sometimes may be significant.  We look for
+# modified tags in local messages by looking only to the files
+# modified since last run.  This is usually rather fast, but the first
+# time OfflineIMAP runs with synclabels enabled, it will have to check
+# the contents of all individual messages for labels and this may take
+# a while.
+
+type = GmailMaildir
+
+
+
 [Repository RemoteExample]
 # And this is the remote repository.  We only support IMAP or Gmail here.
 
@@ -658,3 +693,6 @@ remoteuser = username@gmail.com
 # Enable 1-way synchronization. See above for explanation.
 #
 #readonly = False
+#
+# To enable GMail labels synchronisation, set the option synclabels
+# in the corresponding "Account" section.

offlineimap/folder/Base.py

@@ -273,6 +273,10 @@ class BaseFolder(object):
         """Return the received time for the specified message."""
         raise NotImplementedError
 
+    def getmessagemtime(self, uid):
+        """Returns the message modification time of the specified message."""
+        raise NotImplementedError
+
     def getmessageflags(self, uid):
         """Returns the flags for the specified message."""
         raise NotImplementedError
@@ -324,24 +328,116 @@ class BaseFolder(object):
             self.deletemessageflags(uid, flags)
 
 
+    def getmessagelabels(self, uid):
+        """Returns the labels for the specified message."""
+        raise NotImplementedError
+
+    def savemessagelabels(self, uid, labels, ignorelabels=set(), mtime=0):
+        """Sets the specified message's labels to the given set.
+
+        Note that this function does not check against dryrun settings,
+        so you need to ensure that it is never called in a
+        dryrun mode."""
+        raise NotImplementedError
+
+    def addmessagelabels(self, uid, labels):
+        """Adds the specified labels to the message's labels set.  If a given
+        label is already present, it will not be duplicated.
+
+        Note that this function does not check against dryrun settings,
+        so you need to ensure that it is never called in a
+        dryrun mode.
+
+        :param labels: A set() of labels"""
+        newlabels = self.getmessagelabels(uid) | labels
+        self.savemessagelabels(uid, newlabels)
+
+    def addmessageslabels(self, uidlist, labels):
+        """
+        Note that this function does not check against dryrun settings,
+        so you need to ensure that it is never called in a
+        dryrun mode."""
+        for uid in uidlist:
+            self.addmessagelabels(uid, labels)
+
+    def deletemessagelabels(self, uid, labels):
+        """Removes each label given from the message's label set.  If a given
+        label is already removed, no action will be taken for that label.
+
+        Note that this function does not check against dryrun settings,
+        so you need to ensure that it is never called in a
+        dryrun mode."""
+        newlabels = self.getmessagelabels(uid) - labels
+        self.savemessagelabels(uid, newlabels)
+
+    def deletemessageslabels(self, uidlist, labels):
+        """
+        Note that this function does not check against dryrun settings,
+        so you need to ensure that it is never called in a
+        dryrun mode."""
+        for uid in uidlist:
+            self.deletemessagelabels(uid, labels)
+
+
+    """
+    Illustration of all cases for addmessageheader().
+    '+' means the added contents.
+
+Case 1: No '\n\n', leading '\n'
++X-Flying-Pig-Header: i am here\n
+\n
+This is the body\n
+next line\n
+
+Case 2: '\n\n' at position 0
++X-Flying-Pig-Header: i am here\n
+\n
+\n
+This is the body\n
+next line\n
+
+Case 3: No '\n\n', no leading '\n'
++X-Flying-Pig-Header: i am here\n
++\n
+This is the body\n
+next line\n
+
+Case 4: '\n\n' at non-zero position
+Subject: Something wrong with OI\n
+From: some@person.at+\n
+X-Flying-Pig-Header: i am here\n <-- orig '\n'
+\n
+This is the body\n
+next line\n
+
+    """
+
     def addmessageheader(self, content, headername, headervalue):
         self.ui.debug('',
                  'addmessageheader: called to add %s: %s' % (headername,
                                                              headervalue))
+        prefix = '\n'
+        suffix = ''
         insertionpoint = content.find('\n\n')
-        self.ui.debug('', 'addmessageheader: insertionpoint = %d' % insertionpoint)
-        leader = content[0:insertionpoint]
-        self.ui.debug('', 'addmessageheader: leader = %s' % repr(leader))
         if insertionpoint == 0 or insertionpoint == -1:
-            newline = ''
+            prefix = ''
+            suffix = '\n'
+        if insertionpoint == -1:
             insertionpoint = 0
-        else:
+            # When body starts immediately, without preceding '\n'
-            newline = '\n'
+            # (this shouldn't happen with proper mail messages, but
-        newline += "%s: %s" % (headername, headervalue)
+            # we seen many broken ones), we should add '\n' to make
-        self.ui.debug('', 'addmessageheader: newline = ' + repr(newline))
+            # new (and the only header, in this case) to be properly
-        trailer = content[insertionpoint:]
+            # separated from the message body.
-        self.ui.debug('', 'addmessageheader: trailer = ' + repr(trailer))
+            if content[0] != '\n':
-        return leader + newline + trailer
+                suffix = suffix + '\n'
+
+        self.ui.debug('', 'addmessageheader: insertionpoint = %d' % insertionpoint)
+        headers = content[0:insertionpoint]
+        self.ui.debug('', 'addmessageheader: headers = %s' % repr(headers))
+        new_header = prefix + ("%s: %s" % (headername, headervalue)) + suffix
+        self.ui.debug('', 'addmessageheader: new_header = ' + repr(new_header))
+        return headers + new_header + content[insertionpoint:]
 
 
     def __find_eoh(self, content):
@@ -607,9 +703,10 @@ class BaseFolder(object):
                 continue
 
             selfflags = self.getmessageflags(uid)
-            statusflags = statusfolder.getmessageflags(uid)
+
-            #if we could not get message flags from LocalStatus, assume empty.
+            if statusfolder.uidexists(uid):
-            if statusflags is None:
+                statusflags = statusfolder.getmessageflags(uid)
+            else:
                 statusflags = set()
 
             addflags = selfflags - statusflags

offlineimap/folder/Gmail.py

@@ -16,6 +16,12 @@
 #    along with this program; if not, write to the Free Software
 #    Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
 
+import re
+
+from offlineimap import imaputil
+from offlineimap import imaplibutil
+import offlineimap.accounts
+
 """Folder implementation to support features of the Gmail IMAP server.
 """
 from .IMAP import IMAPFolder
@@ -33,6 +39,7 @@ class GmailFolder(IMAPFolder):
 
     For more information on the Gmail IMAP server:
       http://mail.google.com/support/bin/answer.py?answer=77657&topic=12815
+      https://developers.google.com/google-apps/gmail/imap_extensions
     """
 
     def __init__(self, imapserver, name, repository):
@@ -40,3 +47,315 @@ class GmailFolder(IMAPFolder):
         self.trash_folder = repository.gettrashfolder(name)
         # Gmail will really delete messages upon EXPUNGE in these folders
         self.real_delete_folders =  [ self.trash_folder, repository.getspamfolder() ]
+
+        # The header under which labels are stored
+        self.labelsheader = self.repository.account.getconf('labelsheader', 'X-Keywords')
+
+        # enables / disables label sync
+        self.synclabels = self.repository.account.getconfboolean('synclabels', False)
+
+        # if synclabels is enabled, add a 4th pass to sync labels
+        if self.synclabels:
+            self.imap_query.insert(0, 'X-GM-LABELS')
+            self.syncmessagesto_passes.append(('syncing labels', self.syncmessagesto_labels))
+
+        # Labels to be left alone
+        ignorelabels =  self.repository.account.getconf('ignorelabels', '')
+        self.ignorelabels = set([l for l in re.split(r'\s*,\s*', ignorelabels) if len(l)])
+
+
+    def getmessage(self, uid):
+        """Retrieve message with UID from the IMAP server (incl body).  Also
+           gets Gmail labels and embeds them into the message.
+
+        :returns: the message body or throws and OfflineImapError
+                  (probably severity MESSAGE) if e.g. no message with
+                  this UID could be found.
+        """
+        imapobj = self.imapserver.acquireconnection()
+        try:
+            data = self._fetch_from_imap(imapobj, str(uid), 2)
+        finally:
+            self.imapserver.releaseconnection(imapobj)
+
+        # data looks now e.g.
+        #[('320 (X-GM-LABELS (...) UID 17061 BODY[] {2565}','msgbody....')]
+        # we only asked for one message, and that msg is in data[0].
+        # msbody is in [0][1].
+        body = data[0][1].replace("\r\n", "\n")
+
+        # Embed the labels into the message headers
+        if self.synclabels:
+            m = re.search('X-GM-LABELS\s*\(([^\)]*)\)', data[0][0])
+            if m:
+                labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
+            else:
+                labels = set()
+            labels = labels - self.ignorelabels
+            labels = ', '.join(sorted(labels))
+            body = self.addmessageheader(body, self.labelsheader, labels)
+
+        if len(body)>200:
+            dbg_output = "%s...%s" % (str(body)[:150], str(body)[-50:])
+        else:
+            dbg_output = body
+
+        self.ui.debug('imap', "Returned object from fetching %d: '%s'" %
+                      (uid, dbg_output))
+        return body
+
+    def getmessagelabels(self, uid):
+        if 'labels' in self.messagelist[uid]:
+            return self.messagelist[uid]['labels']
+        else:
+            return set()
+
+    # TODO: merge this code with the parent's cachemessagelist:
+    # TODO: they have too much common logics.
+    def cachemessagelist(self):
+        if not self.synclabels:
+            return super(GmailFolder, self).cachemessagelist()
+
+        self.messagelist = {}
+
+        self.ui.collectingdata(None, self)
+        imapobj = self.imapserver.acquireconnection()
+        try:
+            msgsToFetch = self._msgs_to_fetch(imapobj)
+            if not msgsToFetch:
+                return # No messages to sync
+
+            # Get the flags and UIDs for these. single-quotes prevent
+            # imaplib2 from quoting the sequence.
+            #
+            # NB: msgsToFetch are sequential numbers, not UID's
+            res_type, response = imapobj.fetch("'%s'" % msgsToFetch,
+              '(FLAGS X-GM-LABELS UID)')
+            if res_type != 'OK':
+                raise OfflineImapError("FETCHING UIDs in folder [%s]%s failed. " % \
+                  (self.getrepository(), self) + \
+                  "Server responded '[%s] %s'" % \
+                  (res_type, response), OfflineImapError.ERROR.FOLDER)
+        finally:
+            self.imapserver.releaseconnection(imapobj)
+
+        for messagestr in response:
+            # looks like: '1 (FLAGS (\\Seen Old) X-GM-LABELS (\\Inbox \\Favorites) UID 4807)' or None if no msg
+            # Discard initial message number.
+            if messagestr == None:
+                continue
+            messagestr = messagestr.split(' ', 1)[1]
+            options = imaputil.flags2hash(messagestr)
+            if not 'UID' in options:
+                self.ui.warn('No UID in message with options %s' %\
+                                          str(options),
+                                          minor = 1)
+            else:
+                uid = long(options['UID'])
+                flags = imaputil.flagsimap2maildir(options['FLAGS'])
+                m = re.search('\(([^\)]*)\)', options['X-GM-LABELS'])
+                if m:
+                    labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
+                else:
+                    labels = set()
+                labels = labels - self.ignorelabels
+                rtime = imaplibutil.Internaldate2epoch(messagestr)
+                self.messagelist[uid] = {'uid': uid, 'flags': flags, 'labels': labels, 'time': rtime}
+
+    def savemessage(self, uid, content, flags, rtime):
+        """Save the message on the Server
+
+        This backend always assigns a new uid, so the uid arg is ignored.
+
+        This function will update the self.messagelist dict to contain
+        the new message after sucessfully saving it, including labels.
+
+        See folder/Base for details. Note that savemessage() does not
+        check against dryrun settings, so you need to ensure that
+        savemessage is never called in a dryrun mode.
+
+        :param rtime: A timestamp to be used as the mail date
+        :returns: the UID of the new message as assigned by the server. If the
+                  message is saved, but it's UID can not be found, it will
+                  return 0. If the message can't be written (folder is
+                  read-only for example) it will return -1."""
+
+        if not self.synclabels:
+            return super(GmailFolder, self).savemessage(uid, content, flags, rtime)
+
+        labels = self.getmessageheader(content, self.labelsheader)
+        if labels:
+            labels = set([lb.strip() for lb in labels.split(',') if len(lb.strip()) > 0])
+        else:
+            labels = set()
+
+        ret = super(GmailFolder, self).savemessage(uid, content, flags, rtime)
+        self.savemessagelabels(ret, labels)
+        return ret
+
+    def _messagelabels_aux(self, arg, uidlist, labels):
+        """Common code to savemessagelabels and addmessagelabels"""
+        labels = labels - self.ignorelabels
+        uidlist = [uid for uid in uidlist if uid > 0]
+        if len(uidlist) > 0:
+            imapobj = self.imapserver.acquireconnection()
+            try:
+                labels_str = '(' + ' '.join([imaputil.quote(lb) for lb in labels]) + ')'
+                # Coalesce uid's into ranges
+                uid_str = imaputil.uid_sequence(uidlist)
+                result = self._store_to_imap(imapobj, uid_str, arg, labels_str)
+
+            except imapobj.readonly:
+                self.ui.labelstoreadonly(self, uidlist, data)
+                return None
+
+            finally:
+                self.imapserver.releaseconnection(imapobj)
+
+            if result:
+                retlabels = imaputil.flags2hash(imaputil.imapsplit(result)[1])['X-GM-LABELS']
+                retlabels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(retlabels)])
+                return retlabels
+        return None
+
+    def savemessagelabels(self, uid, labels):
+        """Change a message's labels to `labels`.
+
+        Note that this function does not check against dryrun settings,
+        so you need to ensure that it is never called in a dryrun mode."""
+        if uid in self.messagelist and 'labels' in self.messagelist[uid]:
+            oldlabels = self.messagelist[uid]['labels']
+        else:
+            oldlabels = set()
+        labels = labels - self.ignorelabels
+        newlabels = labels | (oldlabels & self.ignorelabels)
+        if oldlabels != newlabels:
+            result = self._messagelabels_aux('X-GM-LABELS', [uid], newlabels)
+            if result:
+                self.messagelist[uid]['labels'] = newlabels
+            else:
+                self.messagelist[uid]['labels'] = oldlabels
+
+    def addmessageslabels(self, uidlist, labels):
+        """Add `labels` to all messages in uidlist.
+
+        Note that this function does not check against dryrun settings,
+        so you need to ensure that it is never called in a dryrun mode."""
+
+        labels = labels - self.ignorelabels
+        result = self._messagelabels_aux('+X-GM-LABELS', uidlist, labels)
+        if result:
+            for uid in uidlist:
+                self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] | labels
+
+    def deletemessageslabels(self, uidlist, labels):
+        """Delete `labels` from all messages in uidlist.
+
+        Note that this function does not check against dryrun settings,
+        so you need to ensure that it is never called in a dryrun mode."""
+
+        labels = labels - self.ignorelabels
+        result = self._messagelabels_aux('-X-GM-LABELS', uidlist, labels)
+        if result:
+            for uid in uidlist:
+                self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] - labels
+
+    def copymessageto(self, uid, dstfolder, statusfolder, register = 1):
+        """Copies a message from self to dst if needed, updating the status
+
+        Note that this function does not check against dryrun settings,
+        so you need to ensure that it is never called in a
+        dryrun mode.
+
+        :param uid: uid of the message to be copied.
+        :param dstfolder: A BaseFolder-derived instance
+        :param statusfolder: A LocalStatusFolder instance
+        :param register: whether we should register a new thread."
+        :returns: Nothing on success, or raises an Exception."""
+
+        # Check if we are really copying
+        realcopy = uid > 0 and not dstfolder.uidexists(uid)
+
+        # first copy the message
+        super(GmailFolder, self).copymessageto(uid, dstfolder, statusfolder, register)
+
+        # sync labels and mtime now when the message is new (the embedded labels are up to date)
+        # otherwise we may be spending time for nothing, as they will get updated on a later pass.
+        if realcopy and self.synclabels:
+            try:
+                mtime = dstfolder.getmessagemtime(uid)
+                labels = dstfolder.getmessagelabels(uid)
+                statusfolder.savemessagelabels(uid, labels, mtime=mtime)
+
+            # either statusfolder is not sqlite or dstfolder is not GmailMaildir.
+            except NotImplementedError:
+                return
+
+    def syncmessagesto_labels(self, dstfolder, statusfolder):
+        """Pass 4: Label Synchronization (Gmail only)
+
+        Compare label mismatches in self with those in statusfolder. If
+        msg has a valid UID and exists on dstfolder (has not e.g. been
+        deleted there), sync the labels change to both dstfolder and
+        statusfolder.
+
+        This function checks and protects us from action in dryrun mode.
+        """
+        # This applies the labels message by message, as this makes more sense for a
+        # Maildir target. If applied with an other Gmail IMAP target it would not be
+        # the fastest thing in the world though...
+        uidlist = []
+
+        # filter the uids (fast)
+        try:
+            for uid in self.getmessageuidlist():
+                # bail out on CTRL-C or SIGTERM
+                if offlineimap.accounts.Account.abort_NOW_signal.is_set():
+                    break
+
+                # Ignore messages with negative UIDs missed by pass 1 and
+                # don't do anything if the message has been deleted remotely
+                if uid < 0 or not dstfolder.uidexists(uid):
+                    continue
+
+                selflabels = self.getmessagelabels(uid) - self.ignorelabels
+
+                if statusfolder.uidexists(uid):
+                    statuslabels = statusfolder.getmessagelabels(uid) - self.ignorelabels
+                else:
+                    statuslabels = set()
+
+                if selflabels != statuslabels:
+                    uidlist.append(uid)
+
+            # now sync labels (slow)
+            mtimes = {}
+            labels = {}
+            for i, uid in enumerate(uidlist):
+                # bail out on CTRL-C or SIGTERM
+                if offlineimap.accounts.Account.abort_NOW_signal.is_set():
+                    break
+
+                selflabels = self.getmessagelabels(uid) - self.ignorelabels
+
+                if statusfolder.uidexists(uid):
+                    statuslabels = statusfolder.getmessagelabels(uid) - self.ignorelabels
+                else:
+                    statuslabels = set()
+
+                if selflabels != statuslabels:
+                    self.ui.settinglabels(uid, i+1, len(uidlist), sorted(selflabels), dstfolder)
+                    if self.repository.account.dryrun:
+                        continue #don't actually add in a dryrun
+                    dstfolder.savemessagelabels(uid, selflabels, ignorelabels = self.ignorelabels)
+                    mtime = dstfolder.getmessagemtime(uid)
+                    mtimes[uid] = mtime
+                    labels[uid] = selflabels
+
+            # Update statusfolder in a single DB transaction. It is safe, as if something fails,
+            # statusfolder will be updated on the next run.
+            statusfolder.savemessageslabelsbulk(labels)
+            statusfolder.savemessagesmtimebulk(mtimes)
+
+        except NotImplementedError:
+            self.ui.warn("Can't sync labels. You need to configure a local repository of type GmailMaildir")

offlineimap/folder/GmailMaildir.py

@@ -0,0 +1,315 @@
+# Maildir folder support with labels
+# Copyright (C) 2002 - 2011 John Goerzen & contributors
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+
+
+import os
+from .Maildir import MaildirFolder
+from offlineimap import OfflineImapError
+import offlineimap.accounts
+
+class GmailMaildirFolder(MaildirFolder):
+    """Folder implementation to support adding labels to messages in a Maildir.
+    """
+    def __init__(self, root, name, sep, repository):
+        super(GmailMaildirFolder, self).__init__(root, name, sep, repository)
+
+        # The header under which labels are stored
+        self.labelsheader = self.repository.account.getconf('labelsheader', 'X-Keywords')
+
+        # enables / disables label sync
+        self.synclabels = self.repository.account.getconfboolean('synclabels', 0)
+
+        # if synclabels is enabled, add a 4th pass to sync labels
+        if self.synclabels:
+            self.syncmessagesto_passes.append(('syncing labels', self.syncmessagesto_labels))
+
+    def quickchanged(self, statusfolder):
+        """Returns True if the Maildir has changed. Checks uids, flags and mtimes"""
+
+        self.cachemessagelist()
+        # Folder has different uids than statusfolder => TRUE
+        if sorted(self.getmessageuidlist()) != \
+                sorted(statusfolder.getmessageuidlist()):
+            return True
+        # check for flag changes, it's quick on a Maildir
+        for (uid, message) in self.getmessagelist().iteritems():
+            if message['flags'] != statusfolder.getmessageflags(uid):
+                return True
+        # check for newer mtimes. it is also fast
+        for (uid, message) in self.getmessagelist().iteritems():
+            if message['mtime'] > statusfolder.getmessagemtime(uid):
+                return True
+        return False  #Nope, nothing changed
+
+    def cachemessagelist(self):
+        if self.messagelist is None:
+            self.messagelist = self._scanfolder()
+
+        # Get mtimes
+        if self.synclabels:
+            for uid, msg in self.messagelist.items():
+                filepath = os.path.join(self.getfullname(), msg['filename'])
+                msg['mtime'] = long(os.stat(filepath).st_mtime)
+
+    def getmessagelabels(self, uid):
+        # Labels are not cached in cachemessagelist because it is too slow.
+        if not 'labels' in self.messagelist[uid]:
+            filename = self.messagelist[uid]['filename']
+            filepath = os.path.join(self.getfullname(), filename)
+
+            if not os.path.exists(filepath):
+                return set()
+
+            file = open(filepath, 'rt')
+            content = file.read()
+            file.close()
+
+            labels = self.getmessageheader(content, self.labelsheader)
+            if labels:
+                labels = set([lb.strip() for lb in labels.split(',') if len(lb.strip()) > 0])
+            else:
+                labels = set()
+            self.messagelist[uid]['labels'] = labels
+
+        return self.messagelist[uid]['labels']
+
+    def getmessagemtime(self, uid):
+        if not 'mtime' in self.messagelist[uid]:
+            return 0
+        else:
+            return self.messagelist[uid]['mtime']
+
+    def savemessage(self, uid, content, flags, rtime):
+        """Writes a new message, with the specified uid.
+
+        See folder/Base for detail. Note that savemessage() does not
+        check against dryrun settings, so you need to ensure that
+        savemessage is never called in a dryrun mode."""
+
+        if not self.synclabels:
+            return super(GmailMaildirFolder, self).savemessage(uid, content, flags, rtime)
+
+        labels = self.getmessageheader(content, self.labelsheader)
+        if labels:
+            labels = set([lb.strip() for lb in labels.split(',') if len(lb.strip()) > 0])
+        else:
+            labels = set()
+        ret = super(GmailMaildirFolder, self).savemessage(uid, content, flags, rtime)
+
+        # Update the mtime and labels
+        filename = self.messagelist[uid]['filename']
+        filepath = os.path.join(self.getfullname(), filename)
+        self.messagelist[uid]['mtime'] = long(os.stat(filepath).st_mtime)
+        self.messagelist[uid]['labels'] = labels
+        return ret
+
+    def savemessagelabels(self, uid, labels, ignorelabels=set()):
+        """Change a message's labels to `labels`.
+
+        Note that this function does not check against dryrun settings,
+        so you need to ensure that it is never called in a dryrun mode."""
+
+        filename = self.messagelist[uid]['filename']
+        filepath = os.path.join(self.getfullname(), filename)
+
+        file = open(filepath, 'rt')
+        content = file.read()
+        file.close()
+
+        oldlabels = self.getmessageheader(content, self.labelsheader)
+
+        if oldlabels:
+            oldlabels = set([lb.strip() for lb in oldlabels.split(',') if len(lb.strip()) > 0])
+        else:
+            oldlabels = set()
+
+        labels = labels - ignorelabels
+        ignoredlabels = oldlabels & ignorelabels
+        oldlabels = oldlabels - ignorelabels
+
+        # Nothing to change
+        if labels == oldlabels:
+            return
+
+        # Change labels into content
+        labels_str = ', '.join(sorted(labels | ignoredlabels))
+        content = self.addmessageheader(content, self.labelsheader, labels_str)
+        rtime = self.messagelist[uid].get('rtime', None)
+
+        # write file with new labels to a unique file in tmp
+        messagename = self.new_message_filename(uid, set())
+        tmpname = self.save_tmp_file(messagename, content)
+        tmppath = os.path.join(self.getfullname(), tmpname)
+
+        # move to actual location
+        try:
+            os.rename(tmppath, filepath)
+        except OSError as e:
+            raise OfflineImapError("Can't rename file '%s' to '%s': %s" % \
+              (tmppath, filepath, e[1]), OfflineImapError.ERROR.FOLDER)
+
+        if rtime != None:
+            os.utime(filepath, (rtime, rtime))
+
+        # save the new mtime and labels
+        self.messagelist[uid]['mtime'] = long(os.stat(filepath).st_mtime)
+        self.messagelist[uid]['labels'] = labels
+
+    def copymessageto(self, uid, dstfolder, statusfolder, register = 1):
+        """Copies a message from self to dst if needed, updating the status
+
+        Note that this function does not check against dryrun settings,
+        so you need to ensure that it is never called in a
+        dryrun mode.
+
+        :param uid: uid of the message to be copied.
+        :param dstfolder: A BaseFolder-derived instance
+        :param statusfolder: A LocalStatusFolder instance
+        :param register: whether we should register a new thread."
+        :returns: Nothing on success, or raises an Exception."""
+
+        # Check if we are really copying
+        realcopy = uid > 0 and not dstfolder.uidexists(uid)
+
+        # first copy the message
+        super(GmailMaildirFolder, self).copymessageto(uid, dstfolder, statusfolder, register)
+
+        # sync labels and mtime now when the message is new (the embedded labels are up to date,
+        # and have already propagated to the remote server.
+        # for message which already existed on the remote, this is useless, as later the labels may
+        # get updated.
+        if realcopy and self.synclabels:
+            try:
+                labels = dstfolder.getmessagelabels(uid)
+                statusfolder.savemessagelabels(uid, labels, mtime=self.getmessagemtime(uid))
+
+            # either statusfolder is not sqlite or dstfolder is not GmailMaildir.
+            except NotImplementedError:
+                return
+
+    def syncmessagesto_labels(self, dstfolder, statusfolder):
+        """Pass 4: Label Synchronization (Gmail only)
+
+        Compare label mismatches in self with those in statusfolder. If
+        msg has a valid UID and exists on dstfolder (has not e.g. been
+        deleted there), sync the labels change to both dstfolder and
+        statusfolder.
+
+        Also skips messages whose mtime remains the same as statusfolder, as the
+        contents have not changed.
+
+        This function checks and protects us from action in ryrun mode.
+        """
+        # For each label, we store a list of uids to which it should be
+        # added.  Then, we can call addmessageslabels() to apply them in
+        # bulk, rather than one call per message.
+        addlabellist = {}
+        dellabellist = {}
+        uidlist = []
+
+        try:
+            # filter uids (fast)
+            for uid in self.getmessageuidlist():
+                # bail out on CTRL-C or SIGTERM
+                if offlineimap.accounts.Account.abort_NOW_signal.is_set():
+                    break
+
+                # Ignore messages with negative UIDs missed by pass 1 and
+                # don't do anything if the message has been deleted remotely
+                if uid < 0 or not dstfolder.uidexists(uid):
+                    continue
+
+                selfmtime = self.getmessagemtime(uid)
+
+                if statusfolder.uidexists(uid):
+                    statusmtime = statusfolder.getmessagemtime(uid)
+                else:
+                    statusmtime = 0
+
+                if selfmtime > statusmtime:
+                    uidlist.append(uid)
+
+
+            self.ui.collectingdata(uidlist, self)
+            # This can be slow if there is a lot of modified files
+            for uid in uidlist:
+                # bail out on CTRL-C or SIGTERM
+                if offlineimap.accounts.Account.abort_NOW_signal.is_set():
+                    break
+
+                selflabels = self.getmessagelabels(uid)
+
+                if statusfolder.uidexists(uid):
+                    statuslabels = statusfolder.getmessagelabels(uid)
+                else:
+                    statuslabels = set()
+
+                addlabels = selflabels - statuslabels
+                dellabels = statuslabels - selflabels
+
+                for lb in addlabels:
+                    if not lb in addlabellist:
+                        addlabellist[lb] = []
+                    addlabellist[lb].append(uid)
+
+                for lb in dellabels:
+                    if not lb in dellabellist:
+                        dellabellist[lb] = []
+                    dellabellist[lb].append(uid)
+
+            for lb, uids in addlabellist.items():
+                # bail out on CTRL-C or SIGTERM
+                if offlineimap.accounts.Account.abort_NOW_signal.is_set():
+                    break
+
+                self.ui.addinglabels(uids, lb, dstfolder)
+                if self.repository.account.dryrun:
+                    continue #don't actually add in a dryrun
+                dstfolder.addmessageslabels(uids, set([lb]))
+                statusfolder.addmessageslabels(uids, set([lb]))
+
+            for lb, uids in dellabellist.items():
+                # bail out on CTRL-C or SIGTERM
+                if offlineimap.accounts.Account.abort_NOW_signal.is_set():
+                    break
+
+                self.ui.deletinglabels(uids, lb, dstfolder)
+                if self.repository.account.dryrun:
+                    continue #don't actually remove in a dryrun
+                dstfolder.deletemessageslabels(uids, set([lb]))
+                statusfolder.deletemessageslabels(uids, set([lb]))
+
+            # Update mtimes on StatusFolder. It is done last to be safe. If something els fails
+            # and the mtime is not updated, the labels will still be synced next time.
+            mtimes = {}
+            for uid in uidlist:
+                # bail out on CTRL-C or SIGTERM
+                if offlineimap.accounts.Account.abort_NOW_signal.is_set():
+                    break
+
+                if self.repository.account.dryrun:
+                    continue #don't actually update statusfolder
+
+                filename = self.messagelist[uid]['filename']
+                filepath = os.path.join(self.getfullname(), filename)
+                mtimes[uid] = long(os.stat(filepath).st_mtime)
+
+            # finally update statusfolder in a single DB transaction
+            statusfolder.savemessagesmtimebulk(mtimes)
+
+        except NotImplementedError:
+            self.ui.warn("Can't sync labels. You need to configure a remote repository of type Gmail.")

offlineimap/folder/IMAP.py

@@ -142,12 +142,15 @@ class IMAPFolder(BaseFolder):
 
     def _msgs_to_fetch(self, imapobj):
         """
-        Determines UIDS of messages to be fetched
+        Determines sequence numbers of messages to be fetched.
+
+        Message sequence numbers (MSNs) are more easily compacted
+        into ranges which makes transactions slightly faster.
 
         Arguments:
         - imapobj: instance of IMAPlib
 
-        Returns: UID ranges for messages or None if no messages
+        Returns: range(s) for messages or None if no messages
         are to be fetched.
 
         """
@@ -156,7 +159,7 @@ class IMAPFolder(BaseFolder):
             # Empty folder, no need to populate message list
             return None
 
-        # By default examine all UIDs in this folder
+        # By default examine all messages in this folder
         msgsToFetch = '1:*'
 
         maxage = self.config.getdefaultint("Account %s" % self.accountname,
@@ -194,7 +197,7 @@ class IMAPFolder(BaseFolder):
                     self.getrepository(), self, search_cond, res_type, res_data),
                     OfflineImapError.ERROR.FOLDER)
 
-            # Result UIDs are seperated by space, coalesce into ranges
+            # Resulting MSN are separated by space, coalesce into ranges
             msgsToFetch = imaputil.uid_sequence(res_data[0].split())
 
         return msgsToFetch

offlineimap/folder/LocalStatusSQLite.py

@@ -40,7 +40,7 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
     #return connection, cursor
 
     #current version of our db format
-    cur_version = 1
+    cur_version = 2
 
     def __init__(self, name, repository):
         super(LocalStatusSQLiteFolder, self).__init__(name, repository)
@@ -140,21 +140,36 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
                 self.connection.commit()
                 file.close()
                 os.rename(plaintextfilename, plaintextfilename + ".old")
+
+        # Upgrade from database version 1 to version 2
+        # This change adds labels and mtime columns, to be used by Gmail IMAP and Maildir folders.
+        if from_ver <= 1:
+            self.ui._msg('Upgrading LocalStatus cache from version 1 to version 2 for %s:%s' %\
+                           (self.repository, self))
+            self.connection.executescript("""ALTER TABLE status ADD mtime INTEGER DEFAULT 0;
+                                             ALTER TABLE status ADD labels VARCHAR(256) DEFAULT '';
+                                             UPDATE metadata SET value='2' WHERE key='db_version';
+                                          """)
+            self.connection.commit()
+
         # Future version upgrades come here...
-        # if from_ver <= 1: ... #upgrade from 1 to 2
         # if from_ver <= 2: ... #upgrade from 2 to 3
+        # if from_ver <= 3: ... #upgrade from 3 to 4
+
 
     def __create_db(self):
-        """Create a new db file"""
+        """
+        Create a new db file.
+
+        self.connection must point to the opened and valid SQlite
+        database connection.
+        """
         self.ui._msg('Creating new Local Status db for %s:%s' \
                          % (self.repository, self))
-        if hasattr(self, 'connection'):
-            self.connection.close() #close old connections first
-        self.connection = sqlite.connect(self.filename, check_same_thread = False)
         self.connection.executescript("""
         CREATE TABLE metadata (key VARCHAR(50) PRIMARY KEY, value VARCHAR(128));
         INSERT INTO metadata VALUES('db_version', '1');
-        CREATE TABLE status (id INTEGER PRIMARY KEY, flags VARCHAR(50));
+        CREATE TABLE status (id INTEGER PRIMARY KEY, flags VARCHAR(50), mtime INTEGER, labels VARCHAR(256));
         """)
         self.connection.commit()
 
@@ -173,10 +188,11 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
     # Interface from BaseFolder
     def cachemessagelist(self):
         self.messagelist = {}
-        cursor = self.connection.execute('SELECT id,flags from status')
+        cursor = self.connection.execute('SELECT id,flags,mtime,labels from status')
         for row in cursor:
-                flags = set(row[1])
+            flags = set(row[1])
-                self.messagelist[row[0]] = {'uid': row[0], 'flags': flags}
+            labels = set([lb.strip() for lb in row[3].split(',') if len(lb.strip()) > 0])
+            self.messagelist[row[0]] = {'uid': row[0], 'flags': flags, 'mtime': row[2], 'labels': labels}
 
     # Interface from LocalStatusFolder
     def save(self):
@@ -220,12 +236,15 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
     #        assert False,"getmessageflags() called on non-existing message"
 
     # Interface from BaseFolder
-    def savemessage(self, uid, content, flags, rtime):
+    def savemessage(self, uid, content, flags, rtime, mtime=0, labels=set()):
-        """Writes a new message, with the specified uid.
+        """
+        Writes a new message, with the specified uid.
 
         See folder/Base for detail. Note that savemessage() does not
         check against dryrun settings, so you need to ensure that
-        savemessage is never called in a dryrun mode."""
+        savemessage is never called in a dryrun mode.
+        
+        """
         if uid < 0:
             # We cannot assign a uid.
             return uid
@@ -234,10 +253,11 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
             self.savemessageflags(uid, flags)
             return uid
 
-        self.messagelist[uid] = {'uid': uid, 'flags': flags, 'time': rtime}
+        self.messagelist[uid] = {'uid': uid, 'flags': flags, 'time': rtime, 'mtime': mtime, 'labels': labels}
         flags = ''.join(sorted(flags))
-        self.__sql_write('INSERT INTO status (id,flags) VALUES (?,?)',
+        labels = ', '.join(sorted(labels))
-                         (uid,flags))
+        self.__sql_write('INSERT INTO status (id,flags,mtime,labels) VALUES (?,?,?,?)',
+                         (uid,flags,mtime,labels))
         return uid
 
     # Interface from BaseFolder
@@ -246,6 +266,69 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
         flags = ''.join(sorted(flags))
         self.__sql_write('UPDATE status SET flags=? WHERE id=?',(flags,uid))
 
+
+    def getmessageflags(self, uid):
+        return self.messagelist[uid]['flags']
+
+
+    def savemessagelabels(self, uid, labels, mtime=None):
+        self.messagelist[uid]['labels'] = labels
+        if mtime: self.messagelist[uid]['mtime'] = mtime
+
+        labels = ', '.join(sorted(labels))
+        if mtime:
+            self.__sql_write('UPDATE status SET labels=?, mtime=? WHERE id=?',(labels,mtime,uid))
+        else:
+            self.__sql_write('UPDATE status SET labels=? WHERE id=?',(labels,uid))
+
+
+    def savemessageslabelsbulk(self, labels):
+        """
+        Saves labels from a dictionary in a single database operation.
+        
+        """
+        data = [(', '.join(sorted(l)), uid) for uid, l in labels.items()]
+        self.__sql_write('UPDATE status SET labels=? WHERE id=?', data, executemany=True)
+        for uid, l in labels.items():
+            self.messagelist[uid]['labels'] = l
+
+
+    def addmessageslabels(self, uids, labels):
+        data = []
+        for uid in uids:
+            newlabels = self.messagelist[uid]['labels'] | labels
+            data.append((', '.join(sorted(newlabels)), uid))
+        self.__sql_write('UPDATE status SET labels=? WHERE id=?', data, executemany=True)
+        for uid in uids:
+            self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] | labels
+
+
+    def deletemessageslabels(self, uids, labels):
+        data = []
+        for uid in uids:
+            newlabels = self.messagelist[uid]['labels'] - labels
+            data.append((', '.join(sorted(newlabels)), uid))
+        self.__sql_write('UPDATE status SET labels=? WHERE id=?', data, executemany=True)
+        for uid in uids:
+            self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] - labels
+
+
+    def getmessagelabels(self, uid):
+        return self.messagelist[uid]['labels']
+
+
+    def savemessagesmtimebulk(self, mtimes):
+        """Saves mtimes from the mtimes dictionary in a single database operation."""
+        data = [(mt, uid) for uid, mt in mtimes.items()]
+        self.__sql_write('UPDATE status SET mtime=? WHERE id=?', data, executemany=True)
+        for uid, mt in mtimes.items():
+            self.messagelist[uid]['mtime'] = mt
+
+
+    def getmessagemtime(self, uid):
+        return self.messagelist[uid]['mtime']
+
+
     # Interface from BaseFolder
     def deletemessage(self, uid):
         if not uid in self.messagelist:

offlineimap/folder/Maildir.py

@@ -19,6 +19,7 @@ import socket
 import time
 import re
 import os
+import tempfile
 from .Base import BaseFolder
 from threading import Lock
 
@@ -234,7 +235,7 @@ class MaildirFolder(BaseFolder):
         filepath = os.path.join(self.getfullname(), filename)
         return os.path.getmtime(filepath)
 
-    def __new_message_filename(self, uid, flags=set()):
+    def new_message_filename(self, uid, flags=set()):
         """Creates a new unique Maildir filename
 
         :param uid: The UID`None`, or a set of maildir flags
@@ -245,6 +246,52 @@ class MaildirFolder(BaseFolder):
             (timeval, timeseq, os.getpid(), socket.gethostname(),
              uid, self._foldermd5, self.infosep, ''.join(sorted(flags)))
 
+
+    def save_to_tmp_file(self, filename, content):
+        """
+        Saves given content to the named temporary file in the
+        'tmp' subdirectory of $CWD.
+
+        Arguments:
+        - filename: name of the temporary file;
+        - content: data to be saved.
+
+        Returns: relative path to the temporary file
+        that was created.
+
+        """
+
+        tmpname = os.path.join('tmp', filename)
+        # open file and write it out
+        tries = 7
+        while tries:
+            tries = tries - 1
+            try:
+                fd = os.open(os.path.join(self.getfullname(), tmpname),
+                             os.O_EXCL|os.O_CREAT|os.O_WRONLY, 0o666)
+                break
+            except OSError as e:
+                if e.errno == e.EEXIST:
+                    if tries:
+                        time.slep(0.23)
+                        continue
+                    severity = OfflineImapError.ERROR.MESSAGE
+                    raise OfflineImapError("Unique filename %s already exists." % \
+                      filename, severity)
+                else:
+                    raise
+
+        fd = os.fdopen(fd, 'wt')
+        fd.write(content)
+        # Make sure the data hits the disk
+        fd.flush()
+        if self.dofsync:
+            os.fsync(fd)
+        fd.close()
+
+        return tmpname
+
+
     # Interface from BaseFolder
     def savemessage(self, uid, content, flags, rtime):
         """Writes a new message, with the specified uid.
@@ -267,33 +314,12 @@ class MaildirFolder(BaseFolder):
         # Otherwise, save the message in tmp/ and then call savemessageflags()
         # to give it a permanent home.
         tmpdir = os.path.join(self.getfullname(), 'tmp')
-        messagename = self.__new_message_filename(uid, flags)
+        messagename = self.new_message_filename(uid, flags)
-        # open file and write it out
+        tmpname = self.save_to_tmp_file(messagename, content)
-        try:
-            fd = os.open(os.path.join(tmpdir, messagename),
-                           os.O_EXCL|os.O_CREAT|os.O_WRONLY, 0o666)
-        except OSError as e:
-            if e.errno == 17:
-                #FILE EXISTS ALREADY
-                severity = OfflineImapError.ERROR.MESSAGE
-                raise OfflineImapError("Unique filename %s already existing." %\
-                                           messagename, severity)
-            else:
-                raise
-
-        file = os.fdopen(fd, 'wt')
-        file.write(content)
-        # Make sure the data hits the disk
-        file.flush()
-        if self.dofsync:
-            os.fsync(fd)
-        file.close()
-
         if rtime != None:
-            os.utime(os.path.join(tmpdir, messagename), (rtime, rtime))
+            os.utime(os.path.join(self.getfullname(), tmpname), (rtime, rtime))
 
-        self.messagelist[uid] = {'flags': flags,
+        self.messagelist[uid] = {'flags': flags, 'filename': tmpname}
-                                 'filename': os.path.join('tmp', messagename)}
         # savemessageflags moves msg to 'cur' or 'new' as appropriate
         self.savemessageflags(uid, flags)
         self.ui.debug('maildir', 'savemessage: returning uid %d' % uid)
@@ -357,7 +383,7 @@ class MaildirFolder(BaseFolder):
         dir_prefix, filename = os.path.split(oldfilename)
         flags = self.getmessageflags(uid)
         newfilename = os.path.join(dir_prefix,
-          self.__new_message_filename(new_uid, flags))
+          self.new_message_filename(new_uid, flags))
         os.rename(os.path.join(self.getfullname(), oldfilename),
                   os.path.join(self.getfullname(), newfilename))
         self.messagelist[new_uid] = self.messagelist[uid]

offlineimap/imaputil.py

@@ -39,6 +39,16 @@ def dequote(string):
         string = string.replace('\\\\', '\\')
     return string
 
+def quote(string):
+    """Takes an unquoted string and quotes it.
+
+    It only adds double quotes. This function does NOT consider
+    parenthised lists to be quoted.
+    """
+    string = string.replace('"', '\\"')
+    string = string.replace('\\', '\\\\')
+    return '"%s"' % string
+
 def flagsplit(string):
     """Converts a string of IMAP flags to a list
 

offlineimap/repository/Gmail.py

@@ -36,6 +36,13 @@ class GmailRepository(IMAPRepository):
                                 'ssl', 'yes')
         IMAPRepository.__init__(self, reposname, account)
 
+        if self.account.getconfboolean('synclabels', 0) and \
+              self.account.getconf('status_backend', 'plain') != 'sqlite':
+            raise OfflineImapError("The Gmail repository needs the sqlite backend to sync labels.\n"
+                                   "To enable it add 'status_backend = sqlite' in the account section",
+                                   OfflineImapError.ERROR.REPO)
+
+
     def gethost(self):
         """Return the server name to connect to.
 
@@ -71,4 +78,3 @@ class GmailRepository(IMAPRepository):
     def getspamfolder(self):
         #: Gmail also deletes messages upon EXPUNGE in the Spam folder
         return  self.getconf('spamfolder','[Gmail]/Spam')
-

offlineimap/repository/GmailMaildir.py

@@ -0,0 +1,37 @@
+# Maildir repository support
+# Copyright (C) 2002 John Goerzen
+# <jgoerzen@complete.org>
+#
+#    This program is free software; you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation; either version 2 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program; if not, write to the Free Software
+#    Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+
+from offlineimap.repository.Maildir import MaildirRepository
+from offlineimap.folder.GmailMaildir import GmailMaildirFolder
+from offlineimap.error import OfflineImapError
+
+class GmailMaildirRepository(MaildirRepository):
+    def __init__(self, reposname, account):
+        """Initialize a MaildirRepository object.  Takes a path name
+        to the directory holding all the Maildir directories."""
+        super(GmailMaildirRepository, self).__init__(reposname, account)
+        if self.account.getconfboolean('synclabels', 0) and \
+              self.account.getconf('status_backend', 'plain') != 'sqlite':
+            raise OfflineImapError("The GmailMaildir repository needs the sqlite backend to sync labels.\n"
+                                   "To enable it add 'status_backend = sqlite' in the account section",
+                                   OfflineImapError.ERROR.REPO)
+
+
+
+    def getfoldertype(self):
+        return GmailMaildirFolder

offlineimap/repository/Maildir.py

@@ -177,10 +177,9 @@ class MaildirRepository(BaseRepository):
                 self.debug("  This is maildir folder '%s'." % foldername)
                 if self.getconfboolean('restoreatime', False):
                     self._append_folder_atimes(foldername)
-                retval.append(folder.Maildir.MaildirFolder(self.root,
+                fd = self.getfoldertype()(self.root, foldername,
-                                                           foldername,
+                                          self.getsep(), self)
-                                                           self.getsep(),
+                retval.append(fd)
-                                                           self))
 
             if self.getsep() == '/' and dirname != '':
                 # Recursively check sub-directories for folders too.
@@ -194,6 +193,9 @@ class MaildirRepository(BaseRepository):
             self.folders = self._getfolders_scandir(self.root)
         return self.folders
 
+    def getfoldertype(self):
+        return folder.Maildir.MaildirFolder
+
     def forgetfolders(self):
         """Forgets the cached list of folders, if any.  Useful to run
         after a sync run."""

offlineimap/repository/__init__.py

@@ -23,6 +23,7 @@ except ImportError: #python2
 from offlineimap.repository.IMAP import IMAPRepository, MappedIMAPRepository
 from offlineimap.repository.Gmail import GmailRepository
 from offlineimap.repository.Maildir import MaildirRepository
+from offlineimap.repository.GmailMaildir import GmailMaildirRepository
 from offlineimap.repository.LocalStatus import LocalStatusRepository
 from offlineimap.error import OfflineImapError
 
@@ -46,7 +47,8 @@ class Repository(object):
         elif reqtype == 'local':
             name = account.getconf('localrepository')
             typemap = {'IMAP': MappedIMAPRepository,
-                       'Maildir': MaildirRepository}
+                       'Maildir': MaildirRepository,
+                       'GmailMaildir': GmailMaildirRepository}
 
         elif reqtype == 'status':
             # create and return a LocalStatusRepository
@@ -84,4 +86,3 @@ class Repository(object):
         :param regtype: 'remote', 'local', or 'status'
         """
         pass
-

offlineimap/ui/UIBase.py

@@ -254,6 +254,15 @@ class UIBase(object):
                   "for that message." % (
                 str(uidlist), self.getnicename(destfolder), destfolder))
 
+    def labelstoreadonly(self, destfolder, uidlist, labels):
+        if self.config.has_option('general', 'ignore-readonly') and \
+                self.config.getboolean('general', 'ignore-readonly'):
+            return
+        self.warn("Attempted to modify labels for messages %s in folder %s[%s], "
+                  "but that folder is read-only.  No labels have been modified "
+                  "for that message." % (
+                str(uidlist), self.getnicename(destfolder), destfolder))
+
     def deletereadonly(self, destfolder, uidlist):
         if self.config.has_option('general', 'ignore-readonly') and \
                 self.config.getboolean('general', 'ignore-readonly'):
@@ -361,6 +370,25 @@ class UIBase(object):
         self.logger.info("Deleting flag %s from %d messages on %s" % (
                 ", ".join(flags), len(uidlist), dest))
 
+    def addinglabels(self, uidlist, label, dest):
+        self.logger.info("Adding label %s to %d messages on %s" % (
+                label, len(uidlist), dest))
+
+    def deletinglabels(self, uidlist, label, dest):
+        self.logger.info("Deleting label %s from %d messages on %s" % (
+                label, len(uidlist), dest))
+
+    def settinglabels(self, uid, num, num_to_set, labels, dest):
+        self.logger.info("Setting labels to message %d on %s (%d of %d): %s" % (
+                uid, dest, num, num_to_set, ", ".join(labels)))
+
+    def collectingdata(self, uidlist, source):
+      if uidlist:
+        self.logger.info("Collecting data from %d messages on %s" % (
+                len(uidlist), source))
+      else:
+        self.logger.info("Collecting data from messages on %s" % source)
+
     def serverdiagnostics(self, repository, type):
         """Connect to repository and output useful information for debugging"""
         conn = None