maxage: fix timezone issues, remove IMAP-IMAP support, add startdate option

1. When using maxage, local and remote messagelists are supposed to only
contain messages from at most maxage days ago. But local and remote used
different timezones to calculate what "maxage days ago" means, resulting
in removals on one side. Now, we ask the local folder for maxage days'
worth of mail, find the lowest UID, and then ask the remote folder for
all UID's starting with that lowest one.

2. maxage was fundamentally wrong in the IMAP-IMAP case: it assumed that
remote messages have UIDs in the same order as their local counterparts,
which could be false, e.g. when messages are copied in quick succession.
So, remove support for maxage in the IMAP-IMAP case.

3. Add startdate option for IMAP-IMAP syncs: use messages from the given
repository starting at startdate, and all messages from the other
repository. In the first sync, the other repository must be empty.

4. Allow maxage to be specified either as number of days to sync (as
previously) or as a fixed date.

Signed-off-by: Janna Martl <janna.martl109@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This commit is contained in:
Janna Martl 2015-04-07 01:14:11 -07:00 committed by Nicolas Sebrecht
parent 71693b7d8c
commit 8096f6cd5b
9 changed files with 361 additions and 128 deletions

View File

@ -135,7 +135,8 @@ Ignore any autorefresh setting in the configuration file.
Run only quick synchronizations. Run only quick synchronizations.
+ +
Ignore any flag updates on IMAP servers. If a flag on the remote IMAP changes, Ignore any flag updates on IMAP servers. If a flag on the remote IMAP changes,
and we have the message locally, it will be left untouched in a quick run. and we have the message locally, it will be left untouched in a quick run. This
option is ignored if maxage is set.
-u <UI>:: -u <UI>::
@ -400,8 +401,19 @@ If you then point your local mutt, or whatever MUA you use to `~/mail/`
as root, it should still recognize all folders. as root, it should still recognize all folders.
Authors * Edge cases with maxage causing too many messages to be synced.
------- +
All messages from at most maxage days ago (+/- a few hours, depending on
timezones) are synced, but there are cases in which older messages can also be
synced. This happens when a message's UID is significantly higher than those of
other messages with similar dates, e.g. when messages are added to the local
folder behind offlineimap's back, causing them to get assigned a new UID, or
when offlineimap first syncs a pre-existing Maildir. In the latter case, it
could appear as if a noticeable and random subset of old messages are synced.
Main authors
------------
John Goerzen, Sebastian Spaetz, Eygene Ryabinkin, Nicolas Sebrecht. John Goerzen, Sebastian Spaetz, Eygene Ryabinkin, Nicolas Sebrecht.

View File

@ -260,6 +260,8 @@ remoterepository = RemoteExample
# This option stands in the [Account Test] section. # This option stands in the [Account Test] section.
# #
# OfflineImap can replace a number of full updates by quick synchronizations. # OfflineImap can replace a number of full updates by quick synchronizations.
# This option is ignored if maxage or startdate are used.
#
# It only synchronizes a folder if # It only synchronizes a folder if
# #
# 1) a Maildir folder has changed # 1) a Maildir folder has changed
@ -327,21 +329,26 @@ remoterepository = RemoteExample
# This option stands in the [Account Test] section. # This option stands in the [Account Test] section.
# #
# When you are starting to sync an already existing account you can tell # maxage enables you to sync only recent messages. There are two ways to specify
# OfflineIMAP to sync messages from only the last x days. When you do this, # what "recent" means: if maxage is given as an integer, then only messages from
# messages older than x days will be completely ignored. This can be useful for # the last maxage days will be synced. If maxage is given as a date, then only
# importing existing accounts when you do not want to download large amounts of # messages later than that date will be synced.
# archive email.
# #
# Messages older than maxage days will not be synced, their flags will not be # Messages older than the cutoff will not be synced, their flags will not be
# changed, they will not be deleted, etc. For OfflineIMAP it will be like these # changed, they will not be deleted, etc. For OfflineIMAP it will be like these
# messages do not exist. This will perform an IMAP search in the case of IMAP # messages do not exist. This will perform an IMAP search in the case of IMAP or
# or Gmail and therefore requires that the server support server side searching. # Gmail and therefore requires that the server support server side searching.
# This will calculate the earliest day that would be included in the search and #
# include all messages from that day until today. The maxage option expects an # Known edge cases are described in offlineimap(1).
# integer (for the number of days). #
# maxage is allowed only when the local folder is of type Maildir. It can't be
# used with startdate.
#
# The maxage option expects an integer (for the number of days) or a date of the
# form yyyy-mm-dd.
# #
#maxage = 3 #maxage = 3
#maxage = 2015-04-01
# This option stands in the [Account Test] section. # This option stands in the [Account Test] section.
@ -446,6 +453,21 @@ localfolders = ~/Test
#sep = "." #sep = "."
# This option stands in the [Repository LocalExample] section.
#
# startdate syncs mails starting from a given date. It applies the date
# restriction to LocalExample only. The remote repository MUST be empty
# at the first sync where this option is used.
#
# Unlike maxage, this is supported for IMAP-IMAP sync.
#
# startdate can't be used with maxage.
#
# The startdate option expects a date in the format yyyy-mm-dd.
#
#startdate = 2015-04-01
# This option stands in the [Repository LocalExample] section. # This option stands in the [Repository LocalExample] section.
# #
# Some users may not want the atime (last access time) of folders to be # Some users may not want the atime (last access time) of folders to be

View File

@ -17,10 +17,11 @@
from subprocess import Popen, PIPE from subprocess import Popen, PIPE
from threading import Event from threading import Event
import os import os
import time
from sys import exc_info from sys import exc_info
import traceback import traceback
from offlineimap import mbnames, CustomConfig, OfflineImapError from offlineimap import mbnames, CustomConfig, OfflineImapError, imaplibutil
from offlineimap import globals from offlineimap import globals
from offlineimap.repository import Repository from offlineimap.repository import Repository
from offlineimap.ui import getglobalui from offlineimap.ui import getglobalui
@ -402,6 +403,96 @@ def syncfolder(account, remotefolder, quick):
Filtered folders on the remote side will not invoke this function.""" Filtered folders on the remote side will not invoke this function."""
def check_uid_validity(localfolder, remotefolder, statusfolder):
# If either the local or the status folder has messages and
# there is a UID validity problem, warn and abort. If there are
# no messages, UW IMAPd loses UIDVALIDITY. But we don't really
# need it if both local folders are empty. So, in that case,
# just save it off.
if localfolder.getmessagecount() > 0 or statusfolder.getmessagecount() > 0:
if not localfolder.check_uidvalidity():
ui.validityproblem(localfolder)
localfolder.repository.restore_atime()
return
if not remotefolder.check_uidvalidity():
ui.validityproblem(remotefolder)
localrepos.restore_atime()
return
else:
# Both folders empty, just save new UIDVALIDITY
localfolder.save_uidvalidity()
remotefolder.save_uidvalidity()
def save_min_uid(folder, min_uid):
uidfile = folder.get_min_uid_file()
fd = open(uidfile, 'wt')
fd.write(str(min_uid) + "\n")
fd.close()
def cachemessagelists_upto_date(localfolder, remotefolder, date):
""" Returns messages with uid > min(uids of messages newer than date)."""
localfolder.cachemessagelist(min_date=date)
check_uid_validity(localfolder, remotefolder, statusfolder)
# local messagelist had date restriction applied already. Restrict
# sync to messages with UIDs >= min_uid from this list.
#
# local messagelist might contain new messages (with uid's < 0).
positive_uids = filter(
lambda uid: uid > 0, localfolder.getmessageuidlist())
if len(positive_uids) > 0:
remotefolder.cachemessagelist(min_uid=min(positive_uids))
else:
# No messages with UID > 0 in range in localfolder.
# date restriction was applied with respect to local dates but
# remote folder timezone might be different from local, so be
# safe and make sure the range isn't bigger than in local.
remotefolder.cachemessagelist(
min_date=time.gmtime(time.mktime(date) + 24*60*60))
def cachemessagelists_startdate(new, partial, date):
""" Retrieve messagelists when startdate has been set for
the folder 'partial'.
Idea: suppose you want to clone the messages after date in one
account (partial) to a new one (new). If new is empty, then copy
messages in partial newer than date to new, and keep track of the
min uid. On subsequent syncs, sync all the messages in new against
those after that min uid in partial. This is a partial replacement
for maxage in the IMAP-IMAP sync case, where maxage doesn't work:
the UIDs of the messages in localfolder might not be in the same
order as those of corresponding messages in remotefolder, so if L in
local corresponds to R in remote, the ranges [L, ...] and [R, ...]
might not correspond. But, if we're cloning a folder into a new one,
[min_uid, ...] does correspond to [1, ...].
This is just for IMAP-IMAP. For Maildir-IMAP, use maxage instead.
"""
new.cachemessagelist()
min_uid = partial.retrieve_min_uid()
if min_uid == None: # min_uid file didn't exist
if len(new.getmessageuidlist()) > 0:
raise OfflineImapError("To use startdate on Repository %s, "
"Repository %s must be empty"%
(partial.repository.name, new.repository.name),
OfflineImapError.ERROR.MESSAGE)
else:
partial.cachemessagelist(min_date=date)
# messagelist.keys() instead of getuidmessagelist() because in
# the UID mapped case we want the actual local UIDs, not their
# remote counterparts
positive_uids = filter(
lambda uid: uid > 0, partial.messagelist.keys())
if len(positive_uids) > 0:
min_uid = min(positive_uids)
else:
min_uid = 1
save_min_uid(partial, min_uid)
else:
partial.cachemessagelist(min_uid=min_uid)
remoterepos = account.remoterepos remoterepos = account.remoterepos
localrepos = account.localrepos localrepos = account.localrepos
statusrepos = account.statusrepos statusrepos = account.statusrepos
@ -429,43 +520,46 @@ def syncfolder(account, remotefolder, quick):
statusfolder.cachemessagelist() statusfolder.cachemessagelist()
if quick:
if (not localfolder.quickchanged(statusfolder) and
not remotefolder.quickchanged(statusfolder)):
ui.skippingfolder(remotefolder)
localrepos.restore_atime()
return
# Load local folder. # Load local folder.
ui.syncingfolder(remoterepos, remotefolder, localrepos, localfolder) ui.syncingfolder(remoterepos, remotefolder, localrepos, localfolder)
ui.loadmessagelist(localrepos, localfolder)
localfolder.cachemessagelist()
ui.messagelistloaded(localrepos, localfolder, localfolder.getmessagecount())
# If either the local or the status folder has messages and # Retrieve messagelists, taking into account age-restriction
# there is a UID validity problem, warn and abort. If there are # options
# no messages, UW IMAPd loses UIDVALIDITY. But we don't really maxage = localfolder.getmaxage()
# need it if both local folders are empty. So, in that case, localstart = localfolder.getstartdate()
# just save it off. remotestart = remotefolder.getstartdate()
if localfolder.getmessagecount() or statusfolder.getmessagecount(): if (maxage != None) + (localstart != None) + (remotestart != None) > 1:
if not localfolder.check_uidvalidity(): raise OfflineImapError("You can set at most one of the "
ui.validityproblem(localfolder) "following: maxage, startdate (for the local folder), "
localrepos.restore_atime() "startdate (for the remote folder)",
return OfflineImapError.ERROR.REPO), None, exc_info()[2]
if not remotefolder.check_uidvalidity(): if (maxage != None or localstart or remotestart) and quick:
ui.validityproblem(remotefolder) # IMAP quickchanged isn't compatible with options that
localrepos.restore_atime() # involve restricting the messagelist, since the "quick"
return # check can only retrieve a full list of UIDs in the folder.
ui.warn("Quick syncs (-q) not supported in conjunction "
"with maxage or startdate; ignoring -q.")
if maxage != None:
cachemessagelists_upto_date(localfolder, remotefolder, maxage)
elif localstart != None:
cachemessagelists_startdate(remotefolder, localfolder,
localstart)
check_uid_validity(localfolder, remotefolder, statusfolder)
elif remotestart != None:
cachemessagelists_startdate(localfolder, remotefolder,
remotestart)
check_uid_validity(localfolder, remotefolder, statusfolder)
else: else:
# Both folders empty, just save new UIDVALIDITY localfolder.cachemessagelist()
localfolder.save_uidvalidity() if quick:
remotefolder.save_uidvalidity() if (not localfolder.quickchanged(statusfolder) and
not remotefolder.quickchanged(statusfolder)):
# Load remote folder. ui.skippingfolder(remotefolder)
ui.loadmessagelist(remoterepos, remotefolder) localrepos.restore_atime()
remotefolder.cachemessagelist() return
ui.messagelistloaded(remoterepos, remotefolder, check_uid_validity(localfolder, remotefolder, statusfolder)
remotefolder.getmessagecount()) remotefolder.cachemessagelist()
# Synchronize remote changes. # Synchronize remote changes.
if not localrepos.getconfboolean('readonly', False): if not localrepos.getconfboolean('readonly', False):

View File

@ -17,6 +17,7 @@
import os.path import os.path
import re import re
import time
from sys import exc_info from sys import exc_info
from offlineimap import threadutil from offlineimap import threadutil
@ -298,6 +299,76 @@ class BaseFolder(object):
raise NotImplementedError raise NotImplementedError
def getmaxage(self):
""" maxage is allowed to be either an integer or a date of the
form YYYY-mm-dd. This returns a time_struct. """
maxagestr = self.config.getdefault("Account %s"%
self.accountname, "maxage", None)
if maxagestr == None:
return None
# is it a number?
try:
maxage = int(maxagestr)
if maxage < 1:
raise OfflineImapError("invalid maxage value %d"% maxage,
OfflineImapError.ERROR.MESSAGE)
return time.gmtime(time.time() - 60*60*24*maxage)
except ValueError:
pass # maybe it was a date
# is it a date string?
try:
date = time.strptime(maxagestr, "%Y-%m-%d")
if date[0] < 1900:
raise OfflineImapError("maxage led to year %d. "
"Abort syncing."% date[0],
OfflineImapError.ERROR.MESSAGE)
return date
except ValueError:
raise OfflineImapError("invalid maxage value %s"% maxagestr,
OfflineImapError.ERROR.MESSAGE)
def getmaxsize(self):
return self.config.getdefaultint("Account %s"%
self.accountname, "maxsize", None)
def getstartdate(self):
""" Retrieve the value of the configuration option startdate """
datestr = self.config.getdefault("Repository " + self.repository.name,
'startdate', None)
try:
if not datestr:
return None
date = time.strptime(datestr, "%Y-%m-%d")
if date[0] < 1900:
raise OfflineImapError("startdate led to year %d. "
"Abort syncing."% date[0],
OfflineImapError.ERROR.MESSAGE)
return date
except ValueError:
raise OfflineImapError("invalid startdate value %s",
OfflineImapError.ERROR.MESSAGE)
def get_min_uid_file(self):
startuiddir = os.path.join(self.config.getmetadatadir(),
'Repository-' + self.repository.name, 'StartUID')
if not os.path.exists(startuiddir):
os.mkdir(startuiddir, 0o700)
return os.path.join(startuiddir, self.getfolderbasename())
def retrieve_min_uid(self):
uidfile = self.get_min_uid_file()
if not os.path.exists(uidfile):
return None
try:
fd = open(uidfile, 'rt')
min_uid = long(fd.readline().strip())
fd.close()
return min_uid
except:
raise IOError("Can't read %s"% uidfile)
def savemessage(self, uid, content, flags, rtime): def savemessage(self, uid, content, flags, rtime):
"""Writes a new message, with the specified uid. """Writes a new message, with the specified uid.

View File

@ -121,16 +121,18 @@ class GmailFolder(IMAPFolder):
# TODO: merge this code with the parent's cachemessagelist: # TODO: merge this code with the parent's cachemessagelist:
# TODO: they have too much common logics. # TODO: they have too much common logics.
def cachemessagelist(self): def cachemessagelist(self, min_date=None, min_uid=None):
if not self.synclabels: if not self.synclabels:
return super(GmailFolder, self).cachemessagelist() return super(GmailFolder, self).cachemessagelist(
min_date=min_date, min_uid=min_uid)
self.messagelist = {} self.messagelist = {}
self.ui.collectingdata(None, self) self.ui.collectingdata(None, self)
imapobj = self.imapserver.acquireconnection() imapobj = self.imapserver.acquireconnection()
try: try:
msgsToFetch = self._msgs_to_fetch(imapobj) msgsToFetch = self._msgs_to_fetch(
imapobj, min_date=min_date, min_uid=min_uid)
if not msgsToFetch: if not msgsToFetch:
return # No messages to sync return # No messages to sync

View File

@ -64,9 +64,9 @@ class GmailMaildirFolder(MaildirFolder):
'filename': '/no-dir/no-such-file/', 'mtime': 0} 'filename': '/no-dir/no-such-file/', 'mtime': 0}
def cachemessagelist(self): def cachemessagelist(self, min_date=None, min_uid=None):
if self.ismessagelistempty(): if self.ismessagelistempty():
self.messagelist = self._scanfolder() self.messagelist = self._scanfolder(min_date=min_date, min_uid=min_uid)
# Get mtimes # Get mtimes
if self.synclabels: if self.synclabels:

View File

@ -18,6 +18,7 @@
import random import random
import binascii import binascii
import re import re
import os
import time import time
from sys import exc_info from sys import exc_info
@ -79,6 +80,12 @@ class IMAPFolder(BaseFolder):
def waitforthread(self): def waitforthread(self):
self.imapserver.connectionwait() self.imapserver.connectionwait()
def getmaxage(self):
if self.config.getdefault("Account %s"%
self.accountname, "maxage", None):
raise OfflineImapError("maxage is not supported on IMAP-IMAP sync",
OfflineImapError.ERROR.REPO), None, exc_info()[2]
# Interface from BaseFolder # Interface from BaseFolder
def getcopyinstancelimit(self): def getcopyinstancelimit(self):
return 'MSGCOPY_' + self.repository.getname() return 'MSGCOPY_' + self.repository.getname()
@ -143,8 +150,7 @@ class IMAPFolder(BaseFolder):
return True return True
return False return False
def _msgs_to_fetch(self, imapobj, min_date=None, min_uid=None):
def _msgs_to_fetch(self, imapobj):
"""Determines sequence numbers of messages to be fetched. """Determines sequence numbers of messages to be fetched.
Message sequence numbers (MSNs) are more easily compacted Message sequence numbers (MSNs) are more easily compacted
@ -152,57 +158,55 @@ class IMAPFolder(BaseFolder):
Arguments: Arguments:
- imapobj: instance of IMAPlib - imapobj: instance of IMAPlib
- min_date (optional): a time_struct; only fetch messages newer than this
- min_uid (optional): only fetch messages with UID >= min_uid
This function should be called with at MOST one of min_date OR
min_uid set but not BOTH.
Returns: range(s) for messages or None if no messages Returns: range(s) for messages or None if no messages
are to be fetched.""" are to be fetched."""
res_type, imapdata = imapobj.select(self.getfullname(), True, True) def search(search_conditions):
if imapdata == [None] or imapdata[0] == '0': """Actually request the server with the specified conditions.
# Empty folder, no need to populate message list
return None
# By default examine all messages in this folder Returns: range(s) for messages or None if no messages
msgsToFetch = '1:*' are to be fetched."""
res_type, res_data = imapobj.search(None, search_conditions)
maxage = self.config.getdefaultint(
"Account %s"% self.accountname, "maxage", -1)
maxsize = self.config.getdefaultint(
"Account %s"% self.accountname, "maxsize", -1)
# Build search condition
if (maxage != -1) | (maxsize != -1):
search_cond = "(";
if(maxage != -1):
#find out what the oldest message is that we should look at
oldest_struct = time.gmtime(time.time() - (60*60*24*maxage))
if oldest_struct[0] < 1900:
raise OfflineImapError("maxage setting led to year %d. "
"Abort syncing."% oldest_struct[0],
OfflineImapError.ERROR.REPO)
search_cond += "SINCE %02d-%s-%d"% (
oldest_struct[2],
MonthNames[oldest_struct[1]],
oldest_struct[0])
if(maxsize != -1):
if(maxage != -1): # There are two conditions, add space
search_cond += " "
search_cond += "SMALLER %d"% maxsize
search_cond += ")"
res_type, res_data = imapobj.search(None, search_cond)
if res_type != 'OK': if res_type != 'OK':
raise OfflineImapError("SEARCH in folder [%s]%s failed. " raise OfflineImapError("SEARCH in folder [%s]%s failed. "
"Search string was '%s'. Server responded '[%s] %s'"% ( "Search string was '%s'. Server responded '[%s] %s'"% (
self.getrepository(), self, search_cond, res_type, res_data), self.getrepository(), self, search_cond, res_type, res_data),
OfflineImapError.ERROR.FOLDER) OfflineImapError.ERROR.FOLDER)
return res_data[0].split()
# Resulting MSN are separated by space, coalesce into ranges res_type, imapdata = imapobj.select(self.getfullname(), True, True)
msgsToFetch = imaputil.uid_sequence(res_data[0].split()) if imapdata == [None] or imapdata[0] == '0':
# Empty folder, no need to populate message list.
return None
return msgsToFetch conditions = []
# 1. min_uid condition.
if min_uid != None:
conditions.append("UID %d:*"% min_uid)
# 2. date condition.
elif min_date != None:
# Find out what the oldest message is that we should look at.
conditions.append("SINCE %02d-%s-%d"% (
min_date[2], MonthNames[min_date[1]], min_date[0]))
# 3. maxsize condition.
maxsize = self.getmaxsize()
if maxsize != None:
conditions.append("SMALLER %d"% maxsize)
if len(conditions) >= 1:
# Build SEARCH command.
search_cond = "(%s)"% ' '.join(conditions)
search_result = search(search_cond)
return imaputil.uid_sequence(search_result)
# By default consider all messages in this folder.
return '1:*'
# Interface from BaseFolder # Interface from BaseFolder
def msglist_item_initializer(self, uid): def msglist_item_initializer(self, uid):
@ -210,19 +214,21 @@ class IMAPFolder(BaseFolder):
# Interface from BaseFolder # Interface from BaseFolder
def cachemessagelist(self): def cachemessagelist(self, min_date=None, min_uid=None):
self.ui.loadmessagelist(self.repository, self)
self.messagelist = {} self.messagelist = {}
imapobj = self.imapserver.acquireconnection() imapobj = self.imapserver.acquireconnection()
try: try:
msgsToFetch = self._msgs_to_fetch(imapobj) msgsToFetch = self._msgs_to_fetch(
imapobj, min_date=min_date, min_uid=min_uid)
if not msgsToFetch: if not msgsToFetch:
return # No messages to sync return # No messages to sync
# Get the flags and UIDs for these. single-quotes prevent # Get the flags and UIDs for these. single-quotes prevent
# imaplib2 from quoting the sequence. # imaplib2 from quoting the sequence.
res_type, response = imapobj.fetch("'%s'"% res_type, response = imapobj.fetch("'%s'"%
msgsToFetch, '(FLAGS UID)') msgsToFetch, '(FLAGS UID INTERNALDATE)')
if res_type != 'OK': if res_type != 'OK':
raise OfflineImapError("FETCHING UIDs in folder [%s]%s failed. " raise OfflineImapError("FETCHING UIDs in folder [%s]%s failed. "
"Server responded '[%s] %s'"% (self.getrepository(), self, "Server responded '[%s] %s'"% (self.getrepository(), self,
@ -247,6 +253,7 @@ class IMAPFolder(BaseFolder):
flags = imaputil.flagsimap2maildir(options['FLAGS']) flags = imaputil.flagsimap2maildir(options['FLAGS'])
rtime = imaplibutil.Internaldate2epoch(messagestr) rtime = imaplibutil.Internaldate2epoch(messagestr)
self.messagelist[uid] = {'uid': uid, 'flags': flags, 'time': rtime} self.messagelist[uid] = {'uid': uid, 'flags': flags, 'time': rtime}
self.ui.messagelistloaded(self.repository, self, self.getmessagecount())
def dropmessagelistcache(self): def dropmessagelistcache(self):
self.messagelist = {} self.messagelist = {}

View File

@ -92,25 +92,17 @@ class MaildirFolder(BaseFolder):
token.""" token."""
return 42 return 42
# Checks to see if the given message is within the maximum age according def _iswithintime(self, messagename, date):
# to the maildir name which should begin with a timestamp """Check to see if the given message is newer than date (a
def _iswithinmaxage(self, messagename, maxage): time_struct) according to the maildir name which should begin
# In order to have the same behaviour as SINCE in an IMAP search with a timestamp."""
# we must convert this to the oldest time and then strip off hrs/mins
# from that day.
oldest_time_utc = time.time() - (60*60*24*maxage)
oldest_time_struct = time.gmtime(oldest_time_utc)
oldest_time_today_seconds = ((oldest_time_struct[3] * 3600) \
+ (oldest_time_struct[4] * 60) \
+ oldest_time_struct[5])
oldest_time_utc -= oldest_time_today_seconds
timestampmatch = re_timestampmatch.search(messagename) timestampmatch = re_timestampmatch.search(messagename)
if not timestampmatch: if not timestampmatch:
return True return True
timestampstr = timestampmatch.group() timestampstr = timestampmatch.group()
timestamplong = long(timestampstr) timestamplong = long(timestampstr)
if(timestamplong < oldest_time_utc): if(timestamplong < time.mktime(date)):
return False return False
else: else:
return True return True
@ -151,18 +143,21 @@ class MaildirFolder(BaseFolder):
flags = set((c for c in flagmatch.group(1) if not c.islower())) flags = set((c for c in flagmatch.group(1) if not c.islower()))
return prefix, uid, fmd5, flags return prefix, uid, fmd5, flags
def _scanfolder(self): def _scanfolder(self, min_date=None, min_uid=None):
"""Cache the message list from a Maildir. """Cache the message list from a Maildir.
If min_date is set, this finds the min UID of all messages newer than
min_date and uses it as the real cutoff for considering messages.
This handles the edge cases where the date is much earlier than messages
with similar UID's (e.g. the UID was reassigned much later).
Maildir flags are: R (replied) S (seen) T (trashed) D (draft) F Maildir flags are: R (replied) S (seen) T (trashed) D (draft) F
(flagged). (flagged).
:returns: dict that can be used as self.messagelist. :returns: dict that can be used as self.messagelist.
""" """
maxage = self.config.getdefaultint("Account " + self.accountname, maxsize = self.getmaxsize()
"maxage", None)
maxsize = self.config.getdefaultint("Account " + self.accountname,
"maxsize", None)
retval = {} retval = {}
files = [] files = []
nouidcounter = -1 # Messages without UIDs get negative UIDs. nouidcounter = -1 # Messages without UIDs get negative UIDs.
@ -171,12 +166,11 @@ class MaildirFolder(BaseFolder):
files.extend((dirannex, filename) for files.extend((dirannex, filename) for
filename in os.listdir(fulldirname)) filename in os.listdir(fulldirname))
date_excludees = {}
for dirannex, filename in files: for dirannex, filename in files:
# We store just dirannex and filename, ie 'cur/123...' # We store just dirannex and filename, ie 'cur/123...'
filepath = os.path.join(dirannex, filename) filepath = os.path.join(dirannex, filename)
# Check maxage/maxsize if this message should be considered. # Check maxsize if this message should be considered.
if maxage and not self._iswithinmaxage(filename, maxage):
continue
if maxsize and (os.path.getsize(os.path.join( if maxsize and (os.path.getsize(os.path.join(
self.getfullname(), filepath)) > maxsize): self.getfullname(), filepath)) > maxsize):
continue continue
@ -193,16 +187,43 @@ class MaildirFolder(BaseFolder):
nouidcounter -= 1 nouidcounter -= 1
else: else:
uid = long(uidmatch.group(1)) uid = long(uidmatch.group(1))
# 'filename' is 'dirannex/filename', e.g. cur/123,U=1,FMD5=1:2,S if min_uid != None and uid > 0 and uid < min_uid:
retval[uid] = self.msglist_item_initializer(uid) continue
retval[uid]['flags'] = flags if min_date != None and not self._iswithintime(filename, min_date):
retval[uid]['filename'] = filepath # Keep track of messages outside of the time limit, because they
# still might have UID > min(UIDs of within-min_date). We hit
# this case for maxage if any message had a known/valid datetime
# and was re-uploaded because the UID in the filename got lost
# (e.g. local copy/move). On next sync, it was assigned a new
# UID from the server and will be included in the SEARCH
# condition. So, we must re-include them later in this method
# in order to avoid inconsistent lists of messages.
date_excludees[uid] = self.msglist_item_initializer(uid)
date_excludees[uid]['flags'] = flags
date_excludees[uid]['filename'] = filepath
else:
# 'filename' is 'dirannex/filename', e.g. cur/123,U=1,FMD5=1:2,S
retval[uid] = self.msglist_item_initializer(uid)
retval[uid]['flags'] = flags
retval[uid]['filename'] = filepath
if min_date != None:
# Re-include messages with high enough uid's.
positive_uids = filter(lambda uid: uid > 0, retval)
if positive_uids:
min_uid = min(positive_uids)
for uid in date_excludees.keys():
if uid > min_uid:
# This message was originally excluded because of
# its date. It is re-included now because we want all
# messages with UID > min_uid.
retval[uid] = date_excludees[uid]
return retval return retval
# Interface from BaseFolder # Interface from BaseFolder
def quickchanged(self, statusfolder): def quickchanged(self, statusfolder):
"""Returns True if the Maildir has changed""" """Returns True if the Maildir has changed
self.cachemessagelist()
Assumes cachemessagelist() has already been called """
# Folder has different uids than statusfolder => TRUE. # Folder has different uids than statusfolder => TRUE.
if sorted(self.getmessageuidlist()) != \ if sorted(self.getmessageuidlist()) != \
sorted(statusfolder.getmessageuidlist()): sorted(statusfolder.getmessageuidlist()):
@ -219,9 +240,12 @@ class MaildirFolder(BaseFolder):
return {'flags': set(), 'filename': '/no-dir/no-such-file/'} return {'flags': set(), 'filename': '/no-dir/no-such-file/'}
# Interface from BaseFolder # Interface from BaseFolder
def cachemessagelist(self): def cachemessagelist(self, min_date=None, min_uid=None):
if self.ismessagelistempty(): if self.ismessagelistempty():
self.messagelist = self._scanfolder() self.ui.loadmessagelist(self.repository, self)
self.messagelist = self._scanfolder(min_date=min_date,
min_uid=min_uid)
self.ui.messagelistloaded(self.repository, self, self.getmessagecount())
# Interface from BaseFolder # Interface from BaseFolder
def getmessagelist(self): def getmessagelist(self):

View File

@ -94,9 +94,10 @@ class MappedIMAPFolder(IMAPFolder):
OfflineImapError.ERROR.MESSAGE), None, exc_info()[2] OfflineImapError.ERROR.MESSAGE), None, exc_info()[2]
# Interface from BaseFolder # Interface from BaseFolder
def cachemessagelist(self): def cachemessagelist(self, min_date=None, min_uid=None):
self._mb.cachemessagelist() self._mb.cachemessagelist(min_date=min_date, min_uid=min_uid)
reallist = self._mb.getmessagelist() reallist = self._mb.getmessagelist()
self.messagelist = self._mb.messagelist
self.maplock.acquire() self.maplock.acquire()
try: try: