Make GmailFolder sync GMail labels

When synclabels config flag is set to "yes" for the GMail repo,
offlineimap fetches the message labels along with the messages, and
embeds them into the body under the header X-Keywords (or whatever
'labelsheader' was set to), as a comma separated list.

It also adds an extra pass to savemessageto, that performs label
synchronization on existing messages from GMail to local, the same way
it is done with flags.

We also introduce GmailMaildir repository that adds functionality to
change message labels.  It keeps track of messages modification time,
so one can quickly detect when the labels may have changed.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
This commit is contained in:
Abdó Roig-Maranges 2012-10-16 20:20:35 +02:00 committed by Eygene Ryabinkin
parent 9319ae212b
commit 0e4afa9132
15 changed files with 1071 additions and 69 deletions

View File

@ -12,6 +12,8 @@ OfflineIMAP v6.5.6 (YYYY-MM-DD)
* Add knob to apply compression to IMAP connections (Abdó Roig-Maranges)
* Add knob to filter some headers before uploading message
to IMAP server (Abdó Roig-Maranges)
* Allow to sync GMail labels and implement GmailMaildir repository that
adds mechanics to change message labels (Abdó Roig-Maranges)
OfflineIMAP v6.5.5 (2013-10-07)

View File

@ -429,6 +429,41 @@ This is an example of a setup where "TheOtherImap" requires all folders to be un
remoteuser = XXX
#Do not use nametrans here.
Sync from Gmail to a local Maildir with labels
----------------------------------------------
This is an example of a setup where GMail gets synced with a local Maildir.
It also keeps track of GMail labels, that get embedded into the messages
under the header X-Keywords (or whatever labelsheader is set to), and syncs
them back and forth the same way as flags.
The first time it runs on a large repository may take some time as the labels
are read / embedded on every message. Afterwards local label changes are detected
using modification times (much faster)::
[Account Gmail-mine]
localrepository = Gmaillocal-mine
remoterepository = Gmailserver-mine
# Need this to be able to sync labels
status_backend = sqlite
synclabels = yes
# This header is where labels go. Usually you will be fine
# with default value, but in case you want it different,
# here we go:
labelsheader = X-GMail-Keywords
[Repository Gmailserver-mine]
#This is the remote repository
type = Gmail
remotepass = XXX
remoteuser = XXX
[Repository Gmaillocal-mine]
#This is the 'local' repository
type = GmailMaildir
Selecting only a few folders to sync
------------------------------------
Add this to the remote gmail repository section to only sync mails which are in a certain folder::

View File

@ -271,6 +271,22 @@ remoterepository = RemoteExample
#
#maildir-windows-compatible = no
# Specifies if we want to sync GMail lables with the local repository.
# Effective only for GMail IMAP repositories. You should use SQlite
# backend for this to work (see status_backend).
#
#synclabels = no
# Name of the header to use for label storage.
#
#labelsheader = X-Keywords
# Set of labels to be ignored. Comma-separated list. GMail-specific
# labels all start with backslash ('\').
#
#ignorelabels = \Inbox, \Starred, \Sent, \Draft, \Spam, \Trash, \Important
# OfflineIMAP can strip off some headers when your messages are propagated
# back to the IMAP server. This option carries the comma-separated list
@ -283,10 +299,11 @@ remoterepository = RemoteExample
#filterheaders = X-Some-Weird-Header
[Repository LocalExample]
# Each repository requires a "type" declaration. The types supported for
# local repositories are Maildir and IMAP.
# local repositories are Maildir, GmailMaildir and IMAP.
type = Maildir
@ -313,6 +330,24 @@ localfolders = ~/Test
#restoreatime = no
[Repository GmailLocalExample]
# This type of repository enables syncing of Gmail. All Maildir
# configuration settings are also valid here.
#
# This is a separate Repository type from Maildir because it involves
# some extra overhead which sometimes may be significant. We look for
# modified tags in local messages by looking only to the files
# modified since last run. This is usually rather fast, but the first
# time OfflineIMAP runs with synclabels enabled, it will have to check
# the contents of all individual messages for labels and this may take
# a while.
type = GmailMaildir
[Repository RemoteExample]
# And this is the remote repository. We only support IMAP or Gmail here.
@ -658,3 +693,6 @@ remoteuser = username@gmail.com
# Enable 1-way synchronization. See above for explanation.
#
#readonly = False
#
# To enable GMail labels synchronisation, set the option synclabels
# in the corresponding "Account" section.

View File

@ -273,6 +273,10 @@ class BaseFolder(object):
"""Return the received time for the specified message."""
raise NotImplementedError
def getmessagemtime(self, uid):
"""Returns the message modification time of the specified message."""
raise NotImplementedError
def getmessageflags(self, uid):
"""Returns the flags for the specified message."""
raise NotImplementedError
@ -324,24 +328,116 @@ class BaseFolder(object):
self.deletemessageflags(uid, flags)
def getmessagelabels(self, uid):
"""Returns the labels for the specified message."""
raise NotImplementedError
def savemessagelabels(self, uid, labels, ignorelabels=set(), mtime=0):
"""Sets the specified message's labels to the given set.
Note that this function does not check against dryrun settings,
so you need to ensure that it is never called in a
dryrun mode."""
raise NotImplementedError
def addmessagelabels(self, uid, labels):
"""Adds the specified labels to the message's labels set. If a given
label is already present, it will not be duplicated.
Note that this function does not check against dryrun settings,
so you need to ensure that it is never called in a
dryrun mode.
:param labels: A set() of labels"""
newlabels = self.getmessagelabels(uid) | labels
self.savemessagelabels(uid, newlabels)
def addmessageslabels(self, uidlist, labels):
"""
Note that this function does not check against dryrun settings,
so you need to ensure that it is never called in a
dryrun mode."""
for uid in uidlist:
self.addmessagelabels(uid, labels)
def deletemessagelabels(self, uid, labels):
"""Removes each label given from the message's label set. If a given
label is already removed, no action will be taken for that label.
Note that this function does not check against dryrun settings,
so you need to ensure that it is never called in a
dryrun mode."""
newlabels = self.getmessagelabels(uid) - labels
self.savemessagelabels(uid, newlabels)
def deletemessageslabels(self, uidlist, labels):
"""
Note that this function does not check against dryrun settings,
so you need to ensure that it is never called in a
dryrun mode."""
for uid in uidlist:
self.deletemessagelabels(uid, labels)
"""
Illustration of all cases for addmessageheader().
'+' means the added contents.
Case 1: No '\n\n', leading '\n'
+X-Flying-Pig-Header: i am here\n
\n
This is the body\n
next line\n
Case 2: '\n\n' at position 0
+X-Flying-Pig-Header: i am here\n
\n
\n
This is the body\n
next line\n
Case 3: No '\n\n', no leading '\n'
+X-Flying-Pig-Header: i am here\n
+\n
This is the body\n
next line\n
Case 4: '\n\n' at non-zero position
Subject: Something wrong with OI\n
From: some@person.at+\n
X-Flying-Pig-Header: i am here\n <-- orig '\n'
\n
This is the body\n
next line\n
"""
def addmessageheader(self, content, headername, headervalue):
self.ui.debug('',
'addmessageheader: called to add %s: %s' % (headername,
headervalue))
prefix = '\n'
suffix = ''
insertionpoint = content.find('\n\n')
self.ui.debug('', 'addmessageheader: insertionpoint = %d' % insertionpoint)
leader = content[0:insertionpoint]
self.ui.debug('', 'addmessageheader: leader = %s' % repr(leader))
if insertionpoint == 0 or insertionpoint == -1:
newline = ''
prefix = ''
suffix = '\n'
if insertionpoint == -1:
insertionpoint = 0
else:
newline = '\n'
newline += "%s: %s" % (headername, headervalue)
self.ui.debug('', 'addmessageheader: newline = ' + repr(newline))
trailer = content[insertionpoint:]
self.ui.debug('', 'addmessageheader: trailer = ' + repr(trailer))
return leader + newline + trailer
# When body starts immediately, without preceding '\n'
# (this shouldn't happen with proper mail messages, but
# we seen many broken ones), we should add '\n' to make
# new (and the only header, in this case) to be properly
# separated from the message body.
if content[0] != '\n':
suffix = suffix + '\n'
self.ui.debug('', 'addmessageheader: insertionpoint = %d' % insertionpoint)
headers = content[0:insertionpoint]
self.ui.debug('', 'addmessageheader: headers = %s' % repr(headers))
new_header = prefix + ("%s: %s" % (headername, headervalue)) + suffix
self.ui.debug('', 'addmessageheader: new_header = ' + repr(new_header))
return headers + new_header + content[insertionpoint:]
def __find_eoh(self, content):
@ -607,9 +703,10 @@ class BaseFolder(object):
continue
selfflags = self.getmessageflags(uid)
statusflags = statusfolder.getmessageflags(uid)
#if we could not get message flags from LocalStatus, assume empty.
if statusflags is None:
if statusfolder.uidexists(uid):
statusflags = statusfolder.getmessageflags(uid)
else:
statusflags = set()
addflags = selfflags - statusflags

View File

@ -16,6 +16,12 @@
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
import re
from offlineimap import imaputil
from offlineimap import imaplibutil
import offlineimap.accounts
"""Folder implementation to support features of the Gmail IMAP server.
"""
from .IMAP import IMAPFolder
@ -33,6 +39,7 @@ class GmailFolder(IMAPFolder):
For more information on the Gmail IMAP server:
http://mail.google.com/support/bin/answer.py?answer=77657&topic=12815
https://developers.google.com/google-apps/gmail/imap_extensions
"""
def __init__(self, imapserver, name, repository):
@ -40,3 +47,315 @@ class GmailFolder(IMAPFolder):
self.trash_folder = repository.gettrashfolder(name)
# Gmail will really delete messages upon EXPUNGE in these folders
self.real_delete_folders = [ self.trash_folder, repository.getspamfolder() ]
# The header under which labels are stored
self.labelsheader = self.repository.account.getconf('labelsheader', 'X-Keywords')
# enables / disables label sync
self.synclabels = self.repository.account.getconfboolean('synclabels', False)
# if synclabels is enabled, add a 4th pass to sync labels
if self.synclabels:
self.imap_query.insert(0, 'X-GM-LABELS')
self.syncmessagesto_passes.append(('syncing labels', self.syncmessagesto_labels))
# Labels to be left alone
ignorelabels = self.repository.account.getconf('ignorelabels', '')
self.ignorelabels = set([l for l in re.split(r'\s*,\s*', ignorelabels) if len(l)])
def getmessage(self, uid):
"""Retrieve message with UID from the IMAP server (incl body). Also
gets Gmail labels and embeds them into the message.
:returns: the message body or throws and OfflineImapError
(probably severity MESSAGE) if e.g. no message with
this UID could be found.
"""
imapobj = self.imapserver.acquireconnection()
try:
data = self._fetch_from_imap(imapobj, str(uid), 2)
finally:
self.imapserver.releaseconnection(imapobj)
# data looks now e.g.
#[('320 (X-GM-LABELS (...) UID 17061 BODY[] {2565}','msgbody....')]
# we only asked for one message, and that msg is in data[0].
# msbody is in [0][1].
body = data[0][1].replace("\r\n", "\n")
# Embed the labels into the message headers
if self.synclabels:
m = re.search('X-GM-LABELS\s*\(([^\)]*)\)', data[0][0])
if m:
labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
else:
labels = set()
labels = labels - self.ignorelabels
labels = ', '.join(sorted(labels))
body = self.addmessageheader(body, self.labelsheader, labels)
if len(body)>200:
dbg_output = "%s...%s" % (str(body)[:150], str(body)[-50:])
else:
dbg_output = body
self.ui.debug('imap', "Returned object from fetching %d: '%s'" %
(uid, dbg_output))
return body
def getmessagelabels(self, uid):
if 'labels' in self.messagelist[uid]:
return self.messagelist[uid]['labels']
else:
return set()
# TODO: merge this code with the parent's cachemessagelist:
# TODO: they have too much common logics.
def cachemessagelist(self):
if not self.synclabels:
return super(GmailFolder, self).cachemessagelist()
self.messagelist = {}
self.ui.collectingdata(None, self)
imapobj = self.imapserver.acquireconnection()
try:
msgsToFetch = self._msgs_to_fetch(imapobj)
if not msgsToFetch:
return # No messages to sync
# Get the flags and UIDs for these. single-quotes prevent
# imaplib2 from quoting the sequence.
#
# NB: msgsToFetch are sequential numbers, not UID's
res_type, response = imapobj.fetch("'%s'" % msgsToFetch,
'(FLAGS X-GM-LABELS UID)')
if res_type != 'OK':
raise OfflineImapError("FETCHING UIDs in folder [%s]%s failed. " % \
(self.getrepository(), self) + \
"Server responded '[%s] %s'" % \
(res_type, response), OfflineImapError.ERROR.FOLDER)
finally:
self.imapserver.releaseconnection(imapobj)
for messagestr in response:
# looks like: '1 (FLAGS (\\Seen Old) X-GM-LABELS (\\Inbox \\Favorites) UID 4807)' or None if no msg
# Discard initial message number.
if messagestr == None:
continue
messagestr = messagestr.split(' ', 1)[1]
options = imaputil.flags2hash(messagestr)
if not 'UID' in options:
self.ui.warn('No UID in message with options %s' %\
str(options),
minor = 1)
else:
uid = long(options['UID'])
flags = imaputil.flagsimap2maildir(options['FLAGS'])
m = re.search('\(([^\)]*)\)', options['X-GM-LABELS'])
if m:
labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
else:
labels = set()
labels = labels - self.ignorelabels
rtime = imaplibutil.Internaldate2epoch(messagestr)
self.messagelist[uid] = {'uid': uid, 'flags': flags, 'labels': labels, 'time': rtime}
def savemessage(self, uid, content, flags, rtime):
"""Save the message on the Server
This backend always assigns a new uid, so the uid arg is ignored.
This function will update the self.messagelist dict to contain
the new message after sucessfully saving it, including labels.
See folder/Base for details. Note that savemessage() does not
check against dryrun settings, so you need to ensure that
savemessage is never called in a dryrun mode.
:param rtime: A timestamp to be used as the mail date
:returns: the UID of the new message as assigned by the server. If the
message is saved, but it's UID can not be found, it will
return 0. If the message can't be written (folder is
read-only for example) it will return -1."""
if not self.synclabels:
return super(GmailFolder, self).savemessage(uid, content, flags, rtime)
labels = self.getmessageheader(content, self.labelsheader)
if labels:
labels = set([lb.strip() for lb in labels.split(',') if len(lb.strip()) > 0])
else:
labels = set()
ret = super(GmailFolder, self).savemessage(uid, content, flags, rtime)
self.savemessagelabels(ret, labels)
return ret
def _messagelabels_aux(self, arg, uidlist, labels):
"""Common code to savemessagelabels and addmessagelabels"""
labels = labels - self.ignorelabels
uidlist = [uid for uid in uidlist if uid > 0]
if len(uidlist) > 0:
imapobj = self.imapserver.acquireconnection()
try:
labels_str = '(' + ' '.join([imaputil.quote(lb) for lb in labels]) + ')'
# Coalesce uid's into ranges
uid_str = imaputil.uid_sequence(uidlist)
result = self._store_to_imap(imapobj, uid_str, arg, labels_str)
except imapobj.readonly:
self.ui.labelstoreadonly(self, uidlist, data)
return None
finally:
self.imapserver.releaseconnection(imapobj)
if result:
retlabels = imaputil.flags2hash(imaputil.imapsplit(result)[1])['X-GM-LABELS']
retlabels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(retlabels)])
return retlabels
return None
def savemessagelabels(self, uid, labels):
"""Change a message's labels to `labels`.
Note that this function does not check against dryrun settings,
so you need to ensure that it is never called in a dryrun mode."""
if uid in self.messagelist and 'labels' in self.messagelist[uid]:
oldlabels = self.messagelist[uid]['labels']
else:
oldlabels = set()
labels = labels - self.ignorelabels
newlabels = labels | (oldlabels & self.ignorelabels)
if oldlabels != newlabels:
result = self._messagelabels_aux('X-GM-LABELS', [uid], newlabels)
if result:
self.messagelist[uid]['labels'] = newlabels
else:
self.messagelist[uid]['labels'] = oldlabels
def addmessageslabels(self, uidlist, labels):
"""Add `labels` to all messages in uidlist.
Note that this function does not check against dryrun settings,
so you need to ensure that it is never called in a dryrun mode."""
labels = labels - self.ignorelabels
result = self._messagelabels_aux('+X-GM-LABELS', uidlist, labels)
if result:
for uid in uidlist:
self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] | labels
def deletemessageslabels(self, uidlist, labels):
"""Delete `labels` from all messages in uidlist.
Note that this function does not check against dryrun settings,
so you need to ensure that it is never called in a dryrun mode."""
labels = labels - self.ignorelabels
result = self._messagelabels_aux('-X-GM-LABELS', uidlist, labels)
if result:
for uid in uidlist:
self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] - labels
def copymessageto(self, uid, dstfolder, statusfolder, register = 1):
"""Copies a message from self to dst if needed, updating the status
Note that this function does not check against dryrun settings,
so you need to ensure that it is never called in a
dryrun mode.
:param uid: uid of the message to be copied.
:param dstfolder: A BaseFolder-derived instance
:param statusfolder: A LocalStatusFolder instance
:param register: whether we should register a new thread."
:returns: Nothing on success, or raises an Exception."""
# Check if we are really copying
realcopy = uid > 0 and not dstfolder.uidexists(uid)
# first copy the message
super(GmailFolder, self).copymessageto(uid, dstfolder, statusfolder, register)
# sync labels and mtime now when the message is new (the embedded labels are up to date)
# otherwise we may be spending time for nothing, as they will get updated on a later pass.
if realcopy and self.synclabels:
try:
mtime = dstfolder.getmessagemtime(uid)
labels = dstfolder.getmessagelabels(uid)
statusfolder.savemessagelabels(uid, labels, mtime=mtime)
# either statusfolder is not sqlite or dstfolder is not GmailMaildir.
except NotImplementedError:
return
def syncmessagesto_labels(self, dstfolder, statusfolder):
"""Pass 4: Label Synchronization (Gmail only)
Compare label mismatches in self with those in statusfolder. If
msg has a valid UID and exists on dstfolder (has not e.g. been
deleted there), sync the labels change to both dstfolder and
statusfolder.
This function checks and protects us from action in dryrun mode.
"""
# This applies the labels message by message, as this makes more sense for a
# Maildir target. If applied with an other Gmail IMAP target it would not be
# the fastest thing in the world though...
uidlist = []
# filter the uids (fast)
try:
for uid in self.getmessageuidlist():
# bail out on CTRL-C or SIGTERM
if offlineimap.accounts.Account.abort_NOW_signal.is_set():
break
# Ignore messages with negative UIDs missed by pass 1 and
# don't do anything if the message has been deleted remotely
if uid < 0 or not dstfolder.uidexists(uid):
continue
selflabels = self.getmessagelabels(uid) - self.ignorelabels
if statusfolder.uidexists(uid):
statuslabels = statusfolder.getmessagelabels(uid) - self.ignorelabels
else:
statuslabels = set()
if selflabels != statuslabels:
uidlist.append(uid)
# now sync labels (slow)
mtimes = {}
labels = {}
for i, uid in enumerate(uidlist):
# bail out on CTRL-C or SIGTERM
if offlineimap.accounts.Account.abort_NOW_signal.is_set():
break
selflabels = self.getmessagelabels(uid) - self.ignorelabels
if statusfolder.uidexists(uid):
statuslabels = statusfolder.getmessagelabels(uid) - self.ignorelabels
else:
statuslabels = set()
if selflabels != statuslabels:
self.ui.settinglabels(uid, i+1, len(uidlist), sorted(selflabels), dstfolder)
if self.repository.account.dryrun:
continue #don't actually add in a dryrun
dstfolder.savemessagelabels(uid, selflabels, ignorelabels = self.ignorelabels)
mtime = dstfolder.getmessagemtime(uid)
mtimes[uid] = mtime
labels[uid] = selflabels
# Update statusfolder in a single DB transaction. It is safe, as if something fails,
# statusfolder will be updated on the next run.
statusfolder.savemessageslabelsbulk(labels)
statusfolder.savemessagesmtimebulk(mtimes)
except NotImplementedError:
self.ui.warn("Can't sync labels. You need to configure a local repository of type GmailMaildir")

View File

@ -0,0 +1,315 @@
# Maildir folder support with labels
# Copyright (C) 2002 - 2011 John Goerzen & contributors
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
import os
from .Maildir import MaildirFolder
from offlineimap import OfflineImapError
import offlineimap.accounts
class GmailMaildirFolder(MaildirFolder):
"""Folder implementation to support adding labels to messages in a Maildir.
"""
def __init__(self, root, name, sep, repository):
super(GmailMaildirFolder, self).__init__(root, name, sep, repository)
# The header under which labels are stored
self.labelsheader = self.repository.account.getconf('labelsheader', 'X-Keywords')
# enables / disables label sync
self.synclabels = self.repository.account.getconfboolean('synclabels', 0)
# if synclabels is enabled, add a 4th pass to sync labels
if self.synclabels:
self.syncmessagesto_passes.append(('syncing labels', self.syncmessagesto_labels))
def quickchanged(self, statusfolder):
"""Returns True if the Maildir has changed. Checks uids, flags and mtimes"""
self.cachemessagelist()
# Folder has different uids than statusfolder => TRUE
if sorted(self.getmessageuidlist()) != \
sorted(statusfolder.getmessageuidlist()):
return True
# check for flag changes, it's quick on a Maildir
for (uid, message) in self.getmessagelist().iteritems():
if message['flags'] != statusfolder.getmessageflags(uid):
return True
# check for newer mtimes. it is also fast
for (uid, message) in self.getmessagelist().iteritems():
if message['mtime'] > statusfolder.getmessagemtime(uid):
return True
return False #Nope, nothing changed
def cachemessagelist(self):
if self.messagelist is None:
self.messagelist = self._scanfolder()
# Get mtimes
if self.synclabels:
for uid, msg in self.messagelist.items():
filepath = os.path.join(self.getfullname(), msg['filename'])
msg['mtime'] = long(os.stat(filepath).st_mtime)
def getmessagelabels(self, uid):
# Labels are not cached in cachemessagelist because it is too slow.
if not 'labels' in self.messagelist[uid]:
filename = self.messagelist[uid]['filename']
filepath = os.path.join(self.getfullname(), filename)
if not os.path.exists(filepath):
return set()
file = open(filepath, 'rt')
content = file.read()
file.close()
labels = self.getmessageheader(content, self.labelsheader)
if labels:
labels = set([lb.strip() for lb in labels.split(',') if len(lb.strip()) > 0])
else:
labels = set()
self.messagelist[uid]['labels'] = labels
return self.messagelist[uid]['labels']
def getmessagemtime(self, uid):
if not 'mtime' in self.messagelist[uid]:
return 0
else:
return self.messagelist[uid]['mtime']
def savemessage(self, uid, content, flags, rtime):
"""Writes a new message, with the specified uid.
See folder/Base for detail. Note that savemessage() does not
check against dryrun settings, so you need to ensure that
savemessage is never called in a dryrun mode."""
if not self.synclabels:
return super(GmailMaildirFolder, self).savemessage(uid, content, flags, rtime)
labels = self.getmessageheader(content, self.labelsheader)
if labels:
labels = set([lb.strip() for lb in labels.split(',') if len(lb.strip()) > 0])
else:
labels = set()
ret = super(GmailMaildirFolder, self).savemessage(uid, content, flags, rtime)
# Update the mtime and labels
filename = self.messagelist[uid]['filename']
filepath = os.path.join(self.getfullname(), filename)
self.messagelist[uid]['mtime'] = long(os.stat(filepath).st_mtime)
self.messagelist[uid]['labels'] = labels
return ret
def savemessagelabels(self, uid, labels, ignorelabels=set()):
"""Change a message's labels to `labels`.
Note that this function does not check against dryrun settings,
so you need to ensure that it is never called in a dryrun mode."""
filename = self.messagelist[uid]['filename']
filepath = os.path.join(self.getfullname(), filename)
file = open(filepath, 'rt')
content = file.read()
file.close()
oldlabels = self.getmessageheader(content, self.labelsheader)
if oldlabels:
oldlabels = set([lb.strip() for lb in oldlabels.split(',') if len(lb.strip()) > 0])
else:
oldlabels = set()
labels = labels - ignorelabels
ignoredlabels = oldlabels & ignorelabels
oldlabels = oldlabels - ignorelabels
# Nothing to change
if labels == oldlabels:
return
# Change labels into content
labels_str = ', '.join(sorted(labels | ignoredlabels))
content = self.addmessageheader(content, self.labelsheader, labels_str)
rtime = self.messagelist[uid].get('rtime', None)
# write file with new labels to a unique file in tmp
messagename = self.new_message_filename(uid, set())
tmpname = self.save_tmp_file(messagename, content)
tmppath = os.path.join(self.getfullname(), tmpname)
# move to actual location
try:
os.rename(tmppath, filepath)
except OSError as e:
raise OfflineImapError("Can't rename file '%s' to '%s': %s" % \
(tmppath, filepath, e[1]), OfflineImapError.ERROR.FOLDER)
if rtime != None:
os.utime(filepath, (rtime, rtime))
# save the new mtime and labels
self.messagelist[uid]['mtime'] = long(os.stat(filepath).st_mtime)
self.messagelist[uid]['labels'] = labels
def copymessageto(self, uid, dstfolder, statusfolder, register = 1):
"""Copies a message from self to dst if needed, updating the status
Note that this function does not check against dryrun settings,
so you need to ensure that it is never called in a
dryrun mode.
:param uid: uid of the message to be copied.
:param dstfolder: A BaseFolder-derived instance
:param statusfolder: A LocalStatusFolder instance
:param register: whether we should register a new thread."
:returns: Nothing on success, or raises an Exception."""
# Check if we are really copying
realcopy = uid > 0 and not dstfolder.uidexists(uid)
# first copy the message
super(GmailMaildirFolder, self).copymessageto(uid, dstfolder, statusfolder, register)
# sync labels and mtime now when the message is new (the embedded labels are up to date,
# and have already propagated to the remote server.
# for message which already existed on the remote, this is useless, as later the labels may
# get updated.
if realcopy and self.synclabels:
try:
labels = dstfolder.getmessagelabels(uid)
statusfolder.savemessagelabels(uid, labels, mtime=self.getmessagemtime(uid))
# either statusfolder is not sqlite or dstfolder is not GmailMaildir.
except NotImplementedError:
return
def syncmessagesto_labels(self, dstfolder, statusfolder):
"""Pass 4: Label Synchronization (Gmail only)
Compare label mismatches in self with those in statusfolder. If
msg has a valid UID and exists on dstfolder (has not e.g. been
deleted there), sync the labels change to both dstfolder and
statusfolder.
Also skips messages whose mtime remains the same as statusfolder, as the
contents have not changed.
This function checks and protects us from action in ryrun mode.
"""
# For each label, we store a list of uids to which it should be
# added. Then, we can call addmessageslabels() to apply them in
# bulk, rather than one call per message.
addlabellist = {}
dellabellist = {}
uidlist = []
try:
# filter uids (fast)
for uid in self.getmessageuidlist():
# bail out on CTRL-C or SIGTERM
if offlineimap.accounts.Account.abort_NOW_signal.is_set():
break
# Ignore messages with negative UIDs missed by pass 1 and
# don't do anything if the message has been deleted remotely
if uid < 0 or not dstfolder.uidexists(uid):
continue
selfmtime = self.getmessagemtime(uid)
if statusfolder.uidexists(uid):
statusmtime = statusfolder.getmessagemtime(uid)
else:
statusmtime = 0
if selfmtime > statusmtime:
uidlist.append(uid)
self.ui.collectingdata(uidlist, self)
# This can be slow if there is a lot of modified files
for uid in uidlist:
# bail out on CTRL-C or SIGTERM
if offlineimap.accounts.Account.abort_NOW_signal.is_set():
break
selflabels = self.getmessagelabels(uid)
if statusfolder.uidexists(uid):
statuslabels = statusfolder.getmessagelabels(uid)
else:
statuslabels = set()
addlabels = selflabels - statuslabels
dellabels = statuslabels - selflabels
for lb in addlabels:
if not lb in addlabellist:
addlabellist[lb] = []
addlabellist[lb].append(uid)
for lb in dellabels:
if not lb in dellabellist:
dellabellist[lb] = []
dellabellist[lb].append(uid)
for lb, uids in addlabellist.items():
# bail out on CTRL-C or SIGTERM
if offlineimap.accounts.Account.abort_NOW_signal.is_set():
break
self.ui.addinglabels(uids, lb, dstfolder)
if self.repository.account.dryrun:
continue #don't actually add in a dryrun
dstfolder.addmessageslabels(uids, set([lb]))
statusfolder.addmessageslabels(uids, set([lb]))
for lb, uids in dellabellist.items():
# bail out on CTRL-C or SIGTERM
if offlineimap.accounts.Account.abort_NOW_signal.is_set():
break
self.ui.deletinglabels(uids, lb, dstfolder)
if self.repository.account.dryrun:
continue #don't actually remove in a dryrun
dstfolder.deletemessageslabels(uids, set([lb]))
statusfolder.deletemessageslabels(uids, set([lb]))
# Update mtimes on StatusFolder. It is done last to be safe. If something els fails
# and the mtime is not updated, the labels will still be synced next time.
mtimes = {}
for uid in uidlist:
# bail out on CTRL-C or SIGTERM
if offlineimap.accounts.Account.abort_NOW_signal.is_set():
break
if self.repository.account.dryrun:
continue #don't actually update statusfolder
filename = self.messagelist[uid]['filename']
filepath = os.path.join(self.getfullname(), filename)
mtimes[uid] = long(os.stat(filepath).st_mtime)
# finally update statusfolder in a single DB transaction
statusfolder.savemessagesmtimebulk(mtimes)
except NotImplementedError:
self.ui.warn("Can't sync labels. You need to configure a remote repository of type Gmail.")

View File

@ -142,12 +142,15 @@ class IMAPFolder(BaseFolder):
def _msgs_to_fetch(self, imapobj):
"""
Determines UIDS of messages to be fetched
Determines sequence numbers of messages to be fetched.
Message sequence numbers (MSNs) are more easily compacted
into ranges which makes transactions slightly faster.
Arguments:
- imapobj: instance of IMAPlib
Returns: UID ranges for messages or None if no messages
Returns: range(s) for messages or None if no messages
are to be fetched.
"""
@ -156,7 +159,7 @@ class IMAPFolder(BaseFolder):
# Empty folder, no need to populate message list
return None
# By default examine all UIDs in this folder
# By default examine all messages in this folder
msgsToFetch = '1:*'
maxage = self.config.getdefaultint("Account %s" % self.accountname,
@ -194,7 +197,7 @@ class IMAPFolder(BaseFolder):
self.getrepository(), self, search_cond, res_type, res_data),
OfflineImapError.ERROR.FOLDER)
# Result UIDs are seperated by space, coalesce into ranges
# Resulting MSN are separated by space, coalesce into ranges
msgsToFetch = imaputil.uid_sequence(res_data[0].split())
return msgsToFetch

View File

@ -40,7 +40,7 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
#return connection, cursor
#current version of our db format
cur_version = 1
cur_version = 2
def __init__(self, name, repository):
super(LocalStatusSQLiteFolder, self).__init__(name, repository)
@ -140,21 +140,36 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
self.connection.commit()
file.close()
os.rename(plaintextfilename, plaintextfilename + ".old")
# Upgrade from database version 1 to version 2
# This change adds labels and mtime columns, to be used by Gmail IMAP and Maildir folders.
if from_ver <= 1:
self.ui._msg('Upgrading LocalStatus cache from version 1 to version 2 for %s:%s' %\
(self.repository, self))
self.connection.executescript("""ALTER TABLE status ADD mtime INTEGER DEFAULT 0;
ALTER TABLE status ADD labels VARCHAR(256) DEFAULT '';
UPDATE metadata SET value='2' WHERE key='db_version';
""")
self.connection.commit()
# Future version upgrades come here...
# if from_ver <= 1: ... #upgrade from 1 to 2
# if from_ver <= 2: ... #upgrade from 2 to 3
# if from_ver <= 3: ... #upgrade from 3 to 4
def __create_db(self):
"""Create a new db file"""
"""
Create a new db file.
self.connection must point to the opened and valid SQlite
database connection.
"""
self.ui._msg('Creating new Local Status db for %s:%s' \
% (self.repository, self))
if hasattr(self, 'connection'):
self.connection.close() #close old connections first
self.connection = sqlite.connect(self.filename, check_same_thread = False)
self.connection.executescript("""
CREATE TABLE metadata (key VARCHAR(50) PRIMARY KEY, value VARCHAR(128));
INSERT INTO metadata VALUES('db_version', '1');
CREATE TABLE status (id INTEGER PRIMARY KEY, flags VARCHAR(50));
CREATE TABLE status (id INTEGER PRIMARY KEY, flags VARCHAR(50), mtime INTEGER, labels VARCHAR(256));
""")
self.connection.commit()
@ -173,10 +188,11 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
# Interface from BaseFolder
def cachemessagelist(self):
self.messagelist = {}
cursor = self.connection.execute('SELECT id,flags from status')
cursor = self.connection.execute('SELECT id,flags,mtime,labels from status')
for row in cursor:
flags = set(row[1])
self.messagelist[row[0]] = {'uid': row[0], 'flags': flags}
flags = set(row[1])
labels = set([lb.strip() for lb in row[3].split(',') if len(lb.strip()) > 0])
self.messagelist[row[0]] = {'uid': row[0], 'flags': flags, 'mtime': row[2], 'labels': labels}
# Interface from LocalStatusFolder
def save(self):
@ -220,12 +236,15 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
# assert False,"getmessageflags() called on non-existing message"
# Interface from BaseFolder
def savemessage(self, uid, content, flags, rtime):
"""Writes a new message, with the specified uid.
def savemessage(self, uid, content, flags, rtime, mtime=0, labels=set()):
"""
Writes a new message, with the specified uid.
See folder/Base for detail. Note that savemessage() does not
check against dryrun settings, so you need to ensure that
savemessage is never called in a dryrun mode."""
savemessage is never called in a dryrun mode.
"""
if uid < 0:
# We cannot assign a uid.
return uid
@ -234,10 +253,11 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
self.savemessageflags(uid, flags)
return uid
self.messagelist[uid] = {'uid': uid, 'flags': flags, 'time': rtime}
self.messagelist[uid] = {'uid': uid, 'flags': flags, 'time': rtime, 'mtime': mtime, 'labels': labels}
flags = ''.join(sorted(flags))
self.__sql_write('INSERT INTO status (id,flags) VALUES (?,?)',
(uid,flags))
labels = ', '.join(sorted(labels))
self.__sql_write('INSERT INTO status (id,flags,mtime,labels) VALUES (?,?,?,?)',
(uid,flags,mtime,labels))
return uid
# Interface from BaseFolder
@ -246,6 +266,69 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
flags = ''.join(sorted(flags))
self.__sql_write('UPDATE status SET flags=? WHERE id=?',(flags,uid))
def getmessageflags(self, uid):
return self.messagelist[uid]['flags']
def savemessagelabels(self, uid, labels, mtime=None):
self.messagelist[uid]['labels'] = labels
if mtime: self.messagelist[uid]['mtime'] = mtime
labels = ', '.join(sorted(labels))
if mtime:
self.__sql_write('UPDATE status SET labels=?, mtime=? WHERE id=?',(labels,mtime,uid))
else:
self.__sql_write('UPDATE status SET labels=? WHERE id=?',(labels,uid))
def savemessageslabelsbulk(self, labels):
"""
Saves labels from a dictionary in a single database operation.
"""
data = [(', '.join(sorted(l)), uid) for uid, l in labels.items()]
self.__sql_write('UPDATE status SET labels=? WHERE id=?', data, executemany=True)
for uid, l in labels.items():
self.messagelist[uid]['labels'] = l
def addmessageslabels(self, uids, labels):
data = []
for uid in uids:
newlabels = self.messagelist[uid]['labels'] | labels
data.append((', '.join(sorted(newlabels)), uid))
self.__sql_write('UPDATE status SET labels=? WHERE id=?', data, executemany=True)
for uid in uids:
self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] | labels
def deletemessageslabels(self, uids, labels):
data = []
for uid in uids:
newlabels = self.messagelist[uid]['labels'] - labels
data.append((', '.join(sorted(newlabels)), uid))
self.__sql_write('UPDATE status SET labels=? WHERE id=?', data, executemany=True)
for uid in uids:
self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] - labels
def getmessagelabels(self, uid):
return self.messagelist[uid]['labels']
def savemessagesmtimebulk(self, mtimes):
"""Saves mtimes from the mtimes dictionary in a single database operation."""
data = [(mt, uid) for uid, mt in mtimes.items()]
self.__sql_write('UPDATE status SET mtime=? WHERE id=?', data, executemany=True)
for uid, mt in mtimes.items():
self.messagelist[uid]['mtime'] = mt
def getmessagemtime(self, uid):
return self.messagelist[uid]['mtime']
# Interface from BaseFolder
def deletemessage(self, uid):
if not uid in self.messagelist:

View File

@ -19,6 +19,7 @@ import socket
import time
import re
import os
import tempfile
from .Base import BaseFolder
from threading import Lock
@ -234,7 +235,7 @@ class MaildirFolder(BaseFolder):
filepath = os.path.join(self.getfullname(), filename)
return os.path.getmtime(filepath)
def __new_message_filename(self, uid, flags=set()):
def new_message_filename(self, uid, flags=set()):
"""Creates a new unique Maildir filename
:param uid: The UID`None`, or a set of maildir flags
@ -245,6 +246,52 @@ class MaildirFolder(BaseFolder):
(timeval, timeseq, os.getpid(), socket.gethostname(),
uid, self._foldermd5, self.infosep, ''.join(sorted(flags)))
def save_to_tmp_file(self, filename, content):
"""
Saves given content to the named temporary file in the
'tmp' subdirectory of $CWD.
Arguments:
- filename: name of the temporary file;
- content: data to be saved.
Returns: relative path to the temporary file
that was created.
"""
tmpname = os.path.join('tmp', filename)
# open file and write it out
tries = 7
while tries:
tries = tries - 1
try:
fd = os.open(os.path.join(self.getfullname(), tmpname),
os.O_EXCL|os.O_CREAT|os.O_WRONLY, 0o666)
break
except OSError as e:
if e.errno == e.EEXIST:
if tries:
time.slep(0.23)
continue
severity = OfflineImapError.ERROR.MESSAGE
raise OfflineImapError("Unique filename %s already exists." % \
filename, severity)
else:
raise
fd = os.fdopen(fd, 'wt')
fd.write(content)
# Make sure the data hits the disk
fd.flush()
if self.dofsync:
os.fsync(fd)
fd.close()
return tmpname
# Interface from BaseFolder
def savemessage(self, uid, content, flags, rtime):
"""Writes a new message, with the specified uid.
@ -267,33 +314,12 @@ class MaildirFolder(BaseFolder):
# Otherwise, save the message in tmp/ and then call savemessageflags()
# to give it a permanent home.
tmpdir = os.path.join(self.getfullname(), 'tmp')
messagename = self.__new_message_filename(uid, flags)
# open file and write it out
try:
fd = os.open(os.path.join(tmpdir, messagename),
os.O_EXCL|os.O_CREAT|os.O_WRONLY, 0o666)
except OSError as e:
if e.errno == 17:
#FILE EXISTS ALREADY
severity = OfflineImapError.ERROR.MESSAGE
raise OfflineImapError("Unique filename %s already existing." %\
messagename, severity)
else:
raise
file = os.fdopen(fd, 'wt')
file.write(content)
# Make sure the data hits the disk
file.flush()
if self.dofsync:
os.fsync(fd)
file.close()
messagename = self.new_message_filename(uid, flags)
tmpname = self.save_to_tmp_file(messagename, content)
if rtime != None:
os.utime(os.path.join(tmpdir, messagename), (rtime, rtime))
os.utime(os.path.join(self.getfullname(), tmpname), (rtime, rtime))
self.messagelist[uid] = {'flags': flags,
'filename': os.path.join('tmp', messagename)}
self.messagelist[uid] = {'flags': flags, 'filename': tmpname}
# savemessageflags moves msg to 'cur' or 'new' as appropriate
self.savemessageflags(uid, flags)
self.ui.debug('maildir', 'savemessage: returning uid %d' % uid)
@ -357,7 +383,7 @@ class MaildirFolder(BaseFolder):
dir_prefix, filename = os.path.split(oldfilename)
flags = self.getmessageflags(uid)
newfilename = os.path.join(dir_prefix,
self.__new_message_filename(new_uid, flags))
self.new_message_filename(new_uid, flags))
os.rename(os.path.join(self.getfullname(), oldfilename),
os.path.join(self.getfullname(), newfilename))
self.messagelist[new_uid] = self.messagelist[uid]

View File

@ -39,6 +39,16 @@ def dequote(string):
string = string.replace('\\\\', '\\')
return string
def quote(string):
"""Takes an unquoted string and quotes it.
It only adds double quotes. This function does NOT consider
parenthised lists to be quoted.
"""
string = string.replace('"', '\\"')
string = string.replace('\\', '\\\\')
return '"%s"' % string
def flagsplit(string):
"""Converts a string of IMAP flags to a list

View File

@ -36,6 +36,13 @@ class GmailRepository(IMAPRepository):
'ssl', 'yes')
IMAPRepository.__init__(self, reposname, account)
if self.account.getconfboolean('synclabels', 0) and \
self.account.getconf('status_backend', 'plain') != 'sqlite':
raise OfflineImapError("The Gmail repository needs the sqlite backend to sync labels.\n"
"To enable it add 'status_backend = sqlite' in the account section",
OfflineImapError.ERROR.REPO)
def gethost(self):
"""Return the server name to connect to.
@ -71,4 +78,3 @@ class GmailRepository(IMAPRepository):
def getspamfolder(self):
#: Gmail also deletes messages upon EXPUNGE in the Spam folder
return self.getconf('spamfolder','[Gmail]/Spam')

View File

@ -0,0 +1,37 @@
# Maildir repository support
# Copyright (C) 2002 John Goerzen
# <jgoerzen@complete.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
from offlineimap.repository.Maildir import MaildirRepository
from offlineimap.folder.GmailMaildir import GmailMaildirFolder
from offlineimap.error import OfflineImapError
class GmailMaildirRepository(MaildirRepository):
def __init__(self, reposname, account):
"""Initialize a MaildirRepository object. Takes a path name
to the directory holding all the Maildir directories."""
super(GmailMaildirRepository, self).__init__(reposname, account)
if self.account.getconfboolean('synclabels', 0) and \
self.account.getconf('status_backend', 'plain') != 'sqlite':
raise OfflineImapError("The GmailMaildir repository needs the sqlite backend to sync labels.\n"
"To enable it add 'status_backend = sqlite' in the account section",
OfflineImapError.ERROR.REPO)
def getfoldertype(self):
return GmailMaildirFolder

View File

@ -177,10 +177,9 @@ class MaildirRepository(BaseRepository):
self.debug(" This is maildir folder '%s'." % foldername)
if self.getconfboolean('restoreatime', False):
self._append_folder_atimes(foldername)
retval.append(folder.Maildir.MaildirFolder(self.root,
foldername,
self.getsep(),
self))
fd = self.getfoldertype()(self.root, foldername,
self.getsep(), self)
retval.append(fd)
if self.getsep() == '/' and dirname != '':
# Recursively check sub-directories for folders too.
@ -194,6 +193,9 @@ class MaildirRepository(BaseRepository):
self.folders = self._getfolders_scandir(self.root)
return self.folders
def getfoldertype(self):
return folder.Maildir.MaildirFolder
def forgetfolders(self):
"""Forgets the cached list of folders, if any. Useful to run
after a sync run."""

View File

@ -23,6 +23,7 @@ except ImportError: #python2
from offlineimap.repository.IMAP import IMAPRepository, MappedIMAPRepository
from offlineimap.repository.Gmail import GmailRepository
from offlineimap.repository.Maildir import MaildirRepository
from offlineimap.repository.GmailMaildir import GmailMaildirRepository
from offlineimap.repository.LocalStatus import LocalStatusRepository
from offlineimap.error import OfflineImapError
@ -46,7 +47,8 @@ class Repository(object):
elif reqtype == 'local':
name = account.getconf('localrepository')
typemap = {'IMAP': MappedIMAPRepository,
'Maildir': MaildirRepository}
'Maildir': MaildirRepository,
'GmailMaildir': GmailMaildirRepository}
elif reqtype == 'status':
# create and return a LocalStatusRepository
@ -84,4 +86,3 @@ class Repository(object):
:param regtype: 'remote', 'local', or 'status'
"""
pass

View File

@ -254,6 +254,15 @@ class UIBase(object):
"for that message." % (
str(uidlist), self.getnicename(destfolder), destfolder))
def labelstoreadonly(self, destfolder, uidlist, labels):
if self.config.has_option('general', 'ignore-readonly') and \
self.config.getboolean('general', 'ignore-readonly'):
return
self.warn("Attempted to modify labels for messages %s in folder %s[%s], "
"but that folder is read-only. No labels have been modified "
"for that message." % (
str(uidlist), self.getnicename(destfolder), destfolder))
def deletereadonly(self, destfolder, uidlist):
if self.config.has_option('general', 'ignore-readonly') and \
self.config.getboolean('general', 'ignore-readonly'):
@ -361,6 +370,25 @@ class UIBase(object):
self.logger.info("Deleting flag %s from %d messages on %s" % (
", ".join(flags), len(uidlist), dest))
def addinglabels(self, uidlist, label, dest):
self.logger.info("Adding label %s to %d messages on %s" % (
label, len(uidlist), dest))
def deletinglabels(self, uidlist, label, dest):
self.logger.info("Deleting label %s from %d messages on %s" % (
label, len(uidlist), dest))
def settinglabels(self, uid, num, num_to_set, labels, dest):
self.logger.info("Setting labels to message %d on %s (%d of %d): %s" % (
uid, dest, num, num_to_set, ", ".join(labels)))
def collectingdata(self, uidlist, source):
if uidlist:
self.logger.info("Collecting data from %d messages on %s" % (
len(uidlist), source))
else:
self.logger.info("Collecting data from messages on %s" % source)
def serverdiagnostics(self, repository, type):
"""Connect to repository and output useful information for debugging"""
conn = None