Dear All,
Attached is the patch that I have developed to provide maxage and
maxsize options. You can thus sync only the last x days of messages and
exclude large messages. All details in the attached git file.
Regards,
-Mike
-- Attached file included as plaintext by Ecartis --
-- File: submit
From 04fead2b46a79675a5b29de6f2b4088b9c9448e5 Mon Sep 17 00:00:00 2001
From: mike <mike@mikelaptop.(none)>
Date: Sun, 16 Aug 2009 17:00:49 +0430
Subject: [PATCH] Patch to provide maxage and maxsize account options to exclude old/large messages
This is designed to make offlineimap even better for low bandwidth connections.
maxage allows you to specify a number of days and only messages within
that range will be considered by offlineimap for the sync. This can be
useful if you would like to start using offlineimap with a large
existing account and do not want to import large archives of mail.
maxsize allows you to specify the maximum size of a message to consider so
that you can exclude messages with large attachments etc.
In both cases the cachemessagelist function of the folder was modified to ignore
messages that do not meet the criteria. If the criteria are not specified
then the existing code will be executed the same as before. If a message
does not meet the criteria it will be as though this message does not exist
- offlineimap will completely ignore it. It will not have flags updated,
it will not be deleted, it will not be considered at all.
When operating against an IMAP repository a server side search function
is used. This of course requires support for server side search.
I have tested this with either option, no options etc. against IMAP, Maildir
and Gmail. I have run variations of this patch here for the last 3 weeks or
so syncing about 4 accounts normally.
This reverts commit 220db8a77d.
From: Daniel Jacobowitz
Date: July 17, 2009
The documentation may have been unclear, but quick never worked the
way this patch changes it. My installed offlineimap does one sync
every $autorefresh minutes, and the $quick'th one is a full sync. All
the others are quick. Now it's going to do a full sync every
$autorefresh minutes and a ludicrous number of $quick syncs.
-- Daniel Jacobowitz CodeSourcery
Hello John,
The attached patch makes quick synchronization work again (according to
the documentation), so there are <quick> quick synchronizations
performed between full synchronizations.
Thanks,
Michal
* fixes rh bz #510036
Signed-off-by: Christoph Höger <choeger@cs.tu-berlin.de>
[ per jgoerzen: rh bz #510036 is Kerberos issue. also Closes: #535794 in Debian ]
Closes: #535160
Bug#535160: offlineimap fails on encfs (FUSE) due to hard linking
Package: offlineimap
Version: 6.0.3
Severity: normal
Tags: patch
Hi,
if the localrepository is stored on an encfs, offlineimap fails as follows:
OfflineIMAP 6.0.3
Copyright (C) 2002 - 2008 John Goerzen <jgoerzen@complete.org>
This software comes with ABSOLUTELY NO WARRANTY; see the file
COPYING for details. This is free software, and you are welcome
to distribute it under the conditions laid out in COPYING.
***** Processing account pleione
Copying folder structure from IMAP to Maildir
Establishing connection to tunnel:eval `keychain --quiet --eval` && ssh -q mymailhost /usr/sbin/dovecot --exec-mail imap.
Syncing Drafts: IMAP -> Maildir
Syncing INBOX: IMAP -> Maildir
----------8<----------
Syncing net: IMAP -> Maildir
Copy message 360 IMAP[net] -> Maildir[net], LocalStatus[net]
Syncing personal: IMAP -> Maildir
Thread 'Copy message 360 from net' terminated with exception:
Traceback (most recent call last):
File "/var/lib/python-support/python2.4/offlineimap/threadutil.py", line 149, in run
Thread.run(self)
File "threading.py", line 422, in run
self.__target(*self.__args, **self.__kwargs)
File "/var/lib/python-support/python2.4/offlineimap/folder/Base.py", line 282, in copymessageto
newuid = object.savemessage(uid, message, flags, rtime)
File "/var/lib/python-support/python2.4/offlineimap/folder/Maildir.py", line 198, in savemessage
os.path.join(tmpdir, messagename))
OSError: [Errno 1] Operation not permitted
Last 4 debug messages logged for Copy message 360 from net prior to exception:
----------8<----------
As encfs does not support hard linking, the rename of the temporary
message file fails.
This is the only occurrence of hard linking in offlineimap and can be
fixed by using an atomic `os.rename` instead of `os.link`+`os.unlink`.
I have been using the attached patch for more than 1.5 years and have
never again had any problems with offlineimap on encfs.
Regards,
Peter
On Tue, Apr 21, 11:19:00 -0500, John Goerzen wrote:
> I think the one loss of functionality we have here is that it doesn't
> check if a given account has already been listed before adding to the
> list. Should be a simple tweak. If you could tweak that and test, I'd
> apply a new patch.
Good catch. I attach an updated patch which I've tested and it appears
to work fine, including not syncing two accounts twice.
Thanks,
--
Jonny Lamb, UK
jonny@debian.org
From 7f348ee116bba64f7330e28d4e7b2c015910a890 Mon Sep 17 00:00:00 2001
From: Jonny Lamb <jonny@debian.org>
Date: Tue, 05 May 2009 20:45:17 +0100
Subject: [PATCH] Respect order of general.accounts config setting.
This makes the order of account synchronisation the same as the order of
the general.accounts setting by using a list instead of a dict, which
was actually pointless as the value of each dict item was never even
looked at.
Signed-off-by: Jonny Lamb <jonny@debian.org>
Patch from Eric Dorland
Closes: #470875
From: Wouter Verhelst
Subject: kerberos authentication works only the first time
Date: Fri, 14 Mar 2008 09:28:37 +0100
Package: offlineimap
Version: 5.99.8
Severity: normal
Hi,
I have the "autorefresh" configuration option specified in my
.offlineimaprc, and am now using the kerberos authentication.
However, this kerberos authentication seems to work only the first
time
offlineimap tries to fetch mails. The next time, it fails with this
output:
Thread 'Account sync Test' terminated with exception:
Traceback (most recent call last):
File "/var/lib/python-support/python2.4/offlineimap/threadutil.py",
line 153, in run
Thread.run(self)
File "/usr/lib/python2.4/threading.py", line 422, in run
self.__target(*self.__args, **self.__kwargs)
File "/var/lib/python-support/python2.4/offlineimap/accounts.py",
line 119, in syncrunner
self.sync()
File "/var/lib/python-support/python2.4/offlineimap/accounts.py",
line 148, in sync
remoterepos.syncfoldersto(localrepos, [statusrepos])
File
"/var/lib/python-support/python2.4/offlineimap/repository/Base.py",
line 135, in syncfoldersto
srcfolders = src.getfolders()
File
"/var/lib/python-support/python2.4/offlineimap/repository/IMAP.py",
line 192, in getfolders
listresult = imapobj.list(directory =
self.imapserver.reference)[1]
File "/usr/lib/python2.4/imaplib.py", line 469, in list
typ, dat = self._simple_command(name, directory, pattern)
File "/usr/lib/python2.4/imaplib.py", line 1028, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "/usr/lib/python2.4/imaplib.py", line 787, in _command
raise self.error(
error: command LIST illegal in state NONAUTH
Paul Hinze <paul.t.hinze@gmail.com> on 2008-12-22 at 19:16:
> I did a bit of debugging and asking around in #python and it turns out
> that because Maildir.py uses a conditional import of md5 from either
> hashlib or the md5 module, the md5.new() call is not always correct.
> The attached patch should fix this problem.
Thanks to Daniel Rall for pointing out that the attachment got stripped
off my last message. Patch follows.
Paul
----
From e01fdfbf62c61c31b514e826e6cc00fb98d88808 Mon Sep 17 00:00:00 2001
From: Paul Hinze <paul.t.hinze@gmail.com>
Date: Mon, 22 Dec 2008 18:42:44 -0600
Subject: [PATCH] Fixing md5 import when hashlib is available
Thanks to habnabit and Rhamphoryncus in #python for help debugging
Here's the way I'd like to use offlineimap on my laptop:
1. Have a regular cron job running infrequently. The cron job
checks to see
if I'm online, plugged in, and that no other copy of offlineimap is
running. If
all of these conditions are satisfied, it runs offlineimap just once:
"offlineimap -o -u Noninteractive.Quiet"
2. When I start up mutt, I do it by calling a wrapper script that
delays
until cron-started copies of offlineimap have finished, then starts
offlineimap
on its regular, stay-alive and keep checking schedule. When I quit
mutt, the
wrapper script tells offlineimap to stop.
This way I get frequent regular checks while I have mutt running, but
I don't
waste my battery/cpu checking frequently for mail when I'm not
interested in
it.
To make this work, though, it'd be nicer if it were easier to tell
offlineimap,
from the outside, things like "terminate cleanly now" and "when you've
finished
synching, then terminate instead of sleeping and synching again."
OK, to put my money where my mouth is, I attach two patches against
offlineimap
6.0.3.
The first, "cleanup.patch", cleans up a few spots that tend to throw
exceptions
for me as offlineimap is exiting from a KeyboardInterrupt.
The second adds signaling capabilities to offlineimap.
* sending a SIGTERM tells offlineimap to terminate immediately but
cleanly,
just as if "q" had been pressed in the GUI interface
* sending a SIGUSR1 tells every account to do a full sync asap: if
it's
sleeping, then wake up and do the sync now. If it's mid-sync, then
re-synch
any folders whose syncing has already been started or completed, and
continue
to synch the other, queued but not-yet-synched folders.
* sending a SIGHUP tells every account to die as soon as it can (but
not
immediately: only after finishing any synch it's now engaged in)
* sending a SIGUSR2 tells every account to do a full sync asap (as
with
SIGUSR1), then die
It's tricky to mix signals with threads, but I think I've done this
correctly.
I've been using it now for a few weeks without any obvious
problems. But I'm passing it
on so that others can review the code and test it out on their
systems. I developed the
patch when I was running Python 2.5.2, but to my knowledge I don't use
any Python 2.5-specific
code. Now I'm using the patch with Python 2.6.
Although I said "without any obvious problems," let me confess that
I'm
seeing offlineimap regularly choke when I do things like this: start
up
my offlineimap-wrapped copy of mutt, wait a while, put the machine to
sleep (not sure if offlineimap is active in the background or idling),
move to a different spot, wake the machine up again and it acquires a
new network, sometimes a wired network instead of wifi. Offlineimap
doesn't like that so much. I don't yet have any reason to think the
problems here come from my patches. But I'm just acknowledging them,
so
that if others are able to use offlineimap without any difficulty in
situations like I described, then maybe the fault is with my patches.
From Jim Pryor
The first, "cleanup.patch", cleans up a few spots that tend to throw
exceptions for me as offlineimap is exiting from a KeyboardInterrupt.
Frankly, the original code doesn't even make much sense, and
moreover it's not forward compatible with python 2.6, and
furthermore:
DeprecationWarning: socket.ssl() is deprecated.
So, this commit is just a temporary fix.
Signed-off-by: Michael Witten <mfwitten@mit.edu>
module threading should be taken out back and shot.
Condition.wait() is a busywait loop that has negative implications for
battery consumption on laptops.
Queue.get() in blocking mode blocks SIGINT from being delivered.
Argh, argh, argh.
Closes: #493801.
This should improve power-management abilities some more
The catch is that we can't wait any longer for the kathread to
terminate. We were waiting for this in some cases. This is probably
not a big deal.
fixes deb#434074
fixes#66
This patch adds sslclientcert and sslclientkey configuration
options which are passed through to imaplib in order to allow
the use of client certificates for authentication.
Tue Nov 13 14:44:17 CST 2007 Mark Hymers <mhy@debian.org>
Noted thie code:
statusfolder =
statusrepos.getfolder(remotefolder.getvisiblename().\
replace(remoterepos.getsep(),
statusrepos.getsep()))
in accounts.py. Should have been using the sep of the LocalStatus all
along.
refs deb#479798, #68
fixes#20
04/27/2008 09:34 AM - Vincent Beffara
* File darwin.patch added
OK, I finally installed an IMAPS server on my iMac, so I could test a
little further. For some reason, fixing the read() method in
imaplibutil.py did not seem to work (it hung on connecting to the
server) - however, modifying the file imapserver.py similarly to the
non-SSL case fixed the crash.
I also reduced the chunk size to 100k instead of 1M, as 1M seemed to
still trigger the memory error in some cases. Ah, and I added a
platform test, so that the patch does essentially nothing on
non-Darwin machines ...
So, still no guarantee, but the attached patch works for me. Any
comments ?
(If noone here screams in horror at my code, I will include the patch
into the Fink package and see what happens there.)
I created new IMAP folders on my server. In some moment I found offlineimap
crashed. The reason were empty LocalStatus files. From looking at the code I
have no explanation how those files came into existence.
* John Goerzen (jgoerzen@complete.org) wrote:
> tages 359213 help
> thanks
>
> Hi folks,
>
> Unfortunately, I lack both the expertise to add Kerberos
> authentication and a way to test this. A patch would be very welcome
> :-)
Well it was a bit unclear but there's a patch for offlineimap attached
to Guido's message:
http://bugs.debian.org/cgi-bin/bugreport.cgi?msg=10;filename=support-GSSAPI-via-pykerberos.diff;att=1;bug=359213
I've tried it out and it works great! Except that if the Kerberos
layer throws an exception it wouldn't fail over to plain
authentication properly. Attached is a slightly reworked patch that I
tested it in the following situations:
1. python-kerberos uninstalled, with kerberos credentials.
Prompted for password authentication as expected.
2. python-kerberos installed, with kerberos credentials.
Keberos authentication was successful and my mail was fetched
without password prompting. Yeehaw!
3. python-kerberos installed, with no kerberos credentials.
Prompted for password authentication as expected.
I think those are all the interesting situations. So I think this is
ready to go in, with a Suggest on python-kerberos (>=
1.0+mk080218-1).
--
Eric Dorland <eric@kuroneko.ca>
ICQ: #61138586, Jabber: hooty@jabber.com
refs deb#469598
Bug report received in Debian. User is having occasional trouble with Gmail
driver crashing, indicates problem may have started after switching to Gmail.
Gmail driver merged to tree in 81b86fb74 on 2008-01-03 and recently released
in beta tarballs.
BT shows:
File "/var/lib/python-support/python2.4/offlineimap/folder/Gmail.py", line 102, in processmessagesflags
attributehash = imaputil.flags2hash(imaputil.imapsplit(result)[1])
File "/var/lib/python-support/python2.4/offlineimap/imaputil.py", line 88, in imapsplit
for i in range(len(imapstring)):
TypeError: len() of unsized object
imap debug log shows:
imap: 29:58.16 > BEAL75 UID STORE 4887 +FLAGS (\Deleted)
imap: 29:58.47 < BEAL75 OK Success
imap: 29:58.47 matched r'(?P<tag>BEAL\d+) (?P<type>[A-Z]+) (?P<data>.*)' => ('BEAL75', 'OK', 'Success')
imap: imapsplit() called with input: None
imap: imapsplit() got a non-string input; working around.
looking at code for Gmail.py processmessagesflags, comments from Ricardo
indicate it was copied from the same function in folder/IMAP.py.
However, folder/IMAP.py has checks for this, added 2002-07-12 in
5342dacc & 817a10ce. Suspect that Gmail author believed those checks
superfluous for Gmail.
Copied checks from folder/IMAP.py to folder/Gmail.py.
This reverts commit 71a76d9a61.
Zak Smith reported a problem at:
self.processmessagesflags('+', uidlist, flags)
File "/usr/lib/python2.5/site-packages/offlineimap/folder/IMAP.py",
line 372, in processmessagesflags
myrights = imapobj.myrights(self.getfullname())[1][0].split()[1]
IndexError: list index out of range
Conflicts:
offlineimap/folder/IMAP.py
Added WrappedIMAP4_SSL class to help fix up performance of SSL
Standard imaplib.py is really bad with this, since it reads one
character at a time.
Reported by Aaron Kaplan at
http://lists.complete.org/offlineimap@complete.org/2008/01/msg00012.html.gz
He wrote:
I just noticed that the version of offlineimap I've been using
(3.99.17) is well over four years old. How time flies. I haven't
had any problems with it, but out of curiosity I decided to pull in
5.99.2 from the fedora repository. It turns out to take
consistently over twice as long as the old version to sync the same
account. Is this expected?
He tracked it down at
http://lists.complete.org/offlineimap@complete.org/2008/02/msg00012.html.gz
The following changeset is the one responsible for the difference in
speed I was noticing between the imaplib.py that was packaged with
older versions of offlineimap and the one that comes with python:
* /offlineimap/head: changeset 169
More optimizations -- this time fix readline() to not work
character-by-character!
closes#22
from pistore in OfflineIMAP #22:
When an IMAP flag update is performed for multiple messages, some IMAP
servers (e.g. Exchange) return the UID attribute only for some of the
FETCH untagged responses, as shown in the following log:
21:19.04 > DCKF8 UID STORE 66050,50613,52164,40043,40055,25874 +FLAGS
(\Deleted)
21:19.36 < * 35 FETCH (FLAGS (\Seen \Deleted) UID 25874)
21:19.36 < * 321 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 322 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 560 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 581 FETCH (FLAGS (\Seen \Deleted) UID 52164)
21:19.62 < * 1022 FETCH (FLAGS (\Seen \Deleted))
21:19.62 < DCKF8 OK STORE completed.
Function IMAPFolder.processmessagesflags is able to manage the servers
which return the UID and the servers which do not return it, but is
not able to deal with the mixed behavior shown above.
The problem is that the fragment of function
IMAPFolder.processmessagesflags that handles the responses with UID
attribute uses variable flags to store the list of flags of the
message in the IMAP format ("flags = attributehashFLAGS?"), while the
fragment that handles the responses without UID expects variable
"flags" to contain the list of modified flags passed to the function
in Maildir format ("self.messagelist[uid]flags?.append(flag)").
As a consequence, the wrong list of flags is used for the messages
without UID, leading to the addition of "strange" flags to the Maildir
messages:
Syncing messages IMAP[INBOX] -> Maildir[.]
Adding flags to 4 messages on Maildir[.]
Adding flags e to 4 messages on Maildir[.]
Adding flags d to 4 messages on Maildir[.]
Adding flags ) to 4 messages on Maildir[.]
Adding flags ( to 4 messages on Maildir[.]
Adding flags l to 4 messages on Maildir[.]
Adding flags n to 4 messages on Maildir[.]
Adding flags t to 4 messages on Maildir[.]
Adding flags \ to 4 messages on Maildir[.]
Adding flags D to 4 messages on Maildir[.]
Deleting flags T to 4 messages on Maildir[.]
Adding flags to 4 messages on LocalStatus[.]
Adding flags e to 4 messages on LocalStatus[.]
Adding flags d to 4 messages on LocalStatus[.]
Adding flags ) to 4 messages on LocalStatus[.]
Adding flags ( to 4 messages on LocalStatus[.]
Adding flags l to 4 messages on LocalStatus[.]
Adding flags n to 4 messages on LocalStatus[.]
Adding flags t to 4 messages on LocalStatus[.]
Adding flags \ to 4 messages on LocalStatus[.]
Adding flags D to 4 messages on LocalStatus[.]
Deleting flags T to 4 messages on LocalStatus[.]
Fix: use a different variable to store IMAP flags when managing
messages corresponding to responses with UID attribute, e.g.:
*** IMAP.py.orig Wed Aug 22 18:23:17 2007
--- IMAP.py Wed Aug 22 18:22:38 2007
*************** class IMAPFolder(BaseFolder):
*** 340,348 ****
if not ('UID' in attributehash and 'FLAGS' in
attributehash):
# Compensate for servers that don't return a UID
attribute.
continue
! flags = attributehash['FLAGS']
uid = long(attributehash['UID'])
! self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(flags)
try:
needupdate.remove(uid)
except ValueError: # Let it slide if it's not
in the list
--- 340,348 ----
if not ('UID' in attributehash and 'FLAGS' in
attributehash):
# Compensate for servers that don't return a UID
attribute.
continue
! lflags = attributehash['FLAGS']
uid = long(attributehash['UID'])
! self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(lflags)
try:
needupdate.remove(uid)
except ValueError: # Let it slide if it's not
in the list
02/03/08 14:04:35 changed by js
* attachment flags-fix.patch added.
Delete 02/03/08 14:05:24 changed by js ¶
Unfortunately I have to fetch some of my mail from an Exchange server
(Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1)
and I can confirm that the analysis of the problem is correct, and the
patch given here fixes the problem.
Looking at the code of the processmessagesflags() method I think it
generally is a bug that the "flags" parameter is reused as a local
variable, since the final "for uid in needupdate:" loop needs the
original value of "flags". This only worked by accident.
I'm attaching a unidiff version of the patch which applies cleanly
against Debian unstable's offlineimap 5.99.4.
New repository/folder classes to support "real deletion" of messages
thorugh Gmail's IMAP interface: to really delete a message in Gmail,
one has to move it to the Trash folder, rather than EXPUNGE it.
