tobias/docker-offlineimap
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Daniel Jacobowitz patches

Browse Source 
fixes deb#433732

Date: Sun, 30 Sep 2007 13:54:56 -0400
From: Daniel Jacobowitz <drow@false.org>
To: offlineimap@complete.org
Subject: Assorted patches

Here's the result of a lazy Sunday hacking on offlineimap.  Sorry for
not breaking this into multiple patches.  They're mostly logically
independent so just ask if that would make a difference.
First, a new -q (quick) option.  The quick option means to only update
folders that seem to have had significant changes.  For Maildir, any
change to any message UID or flags is significant, because checking
the flags doesn't add a significant cost.  For IMAP, only a change to
the total number of messages or a change in the UID of the most recent
message is significant.  This should catch everything except for
flags changes.

The difference in bandwidth is astonishing: a quick sync takes 80K
instead of 5.3MB, and 28 seconds instead of 90.

There's a configuration variable that lets you say every tenth sync
should update flags, but let all the intervening ones be lighter.


Second, a fix to the UID validity problems many people have been
reporting with Courier.  As discussed in Debian bug #433732, I changed
the UID validity check to use SELECT unless the server complains that
the folder is read-only.  This avoids the Courier bug (see the Debian
log for more details).  This won't fix existing validity errors, you
need to remove the local status and validity files by hand and resync.


Third, some speedups in Maildir checking.  It's still pretty slow
due to a combination of poor performance in os.listdir (never reads
more than 4K of directory entries at a time) and some semaphore that
leads to lots of futex wake operations, but at least this saves
20% or so of the CPU time running offlineimap on a single folder:

Time with quick refresh and md5 in loop: 4.75s user 0.46s system 12%
cpu 41.751 total
Time with quick refresh and md5 out of loop: 4.38s user 0.50s system
14% cpu 34.799 total
Time using string compare to check folder: 4.11s user 0.47s system 13%
cpu 34.788 total


And fourth, some display fixes for Curses.Blinkenlights.  I made
warnings more visible, made the new quick sync message cyan, and
made all not explicitly colored messages grey.  That last one was
really bugging me.  Any time OfflineIMAP printed a warning in
this UI, it had even odds of coming out black on black!


Anyway, I hope these are useful.  I'm happy to revise them if you see
a problem.

-- 
Daniel Jacobowitz
CodeSourcery
This commit is contained in:
John Goerzen
2007-10-01 22:20:37 +01:00
parent 6949a31164
commit 3305d8cd4d
10 changed files with 142 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
offlineimap/folder/IMAP.py
View File

@ -41,6 +41,16 @@ class IMAPFolder(BaseFolder):
self.randomgenerator = random.Random()
BaseFolder.__init__(self)
def selectro(self, imapobj):
"""Select this folder when we do not need write access.
Prefer SELECT to EXAMINE if we can, since some servers
(Courier) do not stabilize UID validity until the folder is
selected."""
try:
imapobj.select(self.getfullname())
except imapobj.readonly:
imapobj.select(self.getfullname(), readonly = 1)
def getaccountname(self):
return self.accountname
@ -60,11 +70,52 @@ class IMAPFolder(BaseFolder):
imapobj = self.imapserver.acquireconnection()
try:
# Primes untagged_responses
imapobj.select(self.getfullname(), readonly = 1)
self.selectro(imapobj)
return long(imapobj.untagged_responses['UIDVALIDITY'][0])
finally:
self.imapserver.releaseconnection(imapobj)
def quickchanged(self, statusfolder):
# An IMAP folder has definitely changed if the number of
# messages or the UID of the last message have changed. Otherwise
# only flag changes could have occurred.
imapobj = self.imapserver.acquireconnection()
try:
# Primes untagged_responses
imapobj.select(self.getfullname(), readonly = 1, force = 1)
try:
# Some mail servers do not return an EXISTS response if
# the folder is empty.
maxmsgid = long(imapobj.untagged_responses['EXISTS'][0])
except KeyError:
return True
# Different number of messages than last time?
if maxmsgid != len(statusfolder.getmessagelist()):
return True
if maxmsgid < 1:
# No messages; return
return False
# Now, get the UID for the last message.
response = imapobj.fetch('%d' % maxmsgid, '(UID)')[1]
finally:
self.imapserver.releaseconnection(imapobj)
# Discard the message number.
messagestr = string.split(response[0], maxsplit = 1)[1]
options = imaputil.flags2hash(messagestr)
if not options.has_key('UID'):
return True
uid = long(options['UID'])
saveduids = statusfolder.getmessagelist().keys()
saveduids.sort()
if uid != saveduids[-1]:
return True
return False
def cachemessagelist(self):
imapobj = self.imapserver.acquireconnection()
self.messagelist = {}

24
offlineimap/folder/Maildir.py
View File

@ -22,7 +22,6 @@ from offlineimap.ui import UIBase
from threading import Lock
import os.path, os, re, time, socket, md5
foldermatchre = re.compile(',FMD5=([0-9a-f]{32})')
uidmatchre = re.compile(',U=(\d+)')
flagmatchre = re.compile(':.*2,([A-Z]+)')
@ -78,16 +77,16 @@ class MaildirFolder(BaseFolder):
files = []
nouidcounter = -1 # Messages without UIDs get
# negative UID numbers.
foldermd5 = md5.new(self.getvisiblename()).hexdigest()
folderstr = ',FMD5=' + foldermd5
for dirannex in ['new', 'cur']:
fulldirname = os.path.join(self.getfullname(), dirannex)
files.extend([os.path.join(fulldirname, filename) for
filename in os.listdir(fulldirname)])
for file in files:
messagename = os.path.basename(file)
foldermatch = foldermatchre.search(messagename)
if (not foldermatch) or \
md5.new(self.getvisiblename()).hexdigest() \
!= foldermatch.group(1):
foldermatch = messagename.find(folderstr) != -1
if not foldermatch:
# If there is no folder MD5 specified, or if it mismatches,
# assume it is a foreign (new) message and generate a
# negative uid for it
@ -111,8 +110,21 @@ class MaildirFolder(BaseFolder):
'filename': file}
return retval
def quickchanged(self, statusfolder):
self.cachemessagelist()
savedmessages = statusfolder.getmessagelist()
if len(self.messagelist) != len(savedmessages):
return True
for uid in self.messagelist.keys():
if uid not in savedmessages:
return True
if self.messagelist[uid]['flags'] != savedmessages[uid]['flags']:
return True
return False
def cachemessagelist(self):
self.messagelist = self._scanfolder()
if self.messagelist is None:
self.messagelist = self._scanfolder()
def getmessagelist(self):
return self.messagelist