Commit Graph

123 Commits

Author SHA1 Message Date
John Goerzen
08a579657a Fix handling of servers that return UIDs in some FETCH responses
closes #22

from pistore in OfflineIMAP #22:

When an IMAP flag update is performed for multiple messages, some IMAP
servers (e.g. Exchange) return the UID attribute only for some of the
FETCH untagged responses, as shown in the following log:

21:19.04 > DCKF8 UID STORE 66050,50613,52164,40043,40055,25874 +FLAGS
(\Deleted)
21:19.36 < * 35 FETCH (FLAGS (\Seen \Deleted) UID 25874)
21:19.36 < * 321 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 322 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 560 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 581 FETCH (FLAGS (\Seen \Deleted) UID 52164)
21:19.62 < * 1022 FETCH (FLAGS (\Seen \Deleted))
21:19.62 < DCKF8 OK STORE completed.

Function IMAPFolder.processmessagesflags is able to manage the servers
which return the UID and the servers which do not return it, but is
not able to deal with the mixed behavior shown above.

The problem is that the fragment of function
IMAPFolder.processmessagesflags that handles the responses with UID
attribute uses variable flags to store the list of flags of the
message in the IMAP format ("flags = attributehashFLAGS?"), while the
fragment that handles the responses without UID expects variable
"flags" to contain the list of modified flags passed to the function
in Maildir format ("self.messagelist[uid]flags?.append(flag)").

As a consequence, the wrong list of flags is used for the messages
without UID, leading to the addition of "strange" flags to the Maildir
messages:

Syncing messages IMAP[INBOX] -> Maildir[.]
Adding flags   to 4 messages  on Maildir[.]
Adding flags e to 4 messages  on Maildir[.]
Adding flags d to 4 messages  on Maildir[.]
Adding flags ) to 4 messages  on Maildir[.]
Adding flags ( to 4 messages  on Maildir[.]
Adding flags l to 4 messages  on Maildir[.]
Adding flags n to 4 messages  on Maildir[.]
Adding flags t to 4 messages  on Maildir[.]
Adding flags \ to 4 messages  on Maildir[.]
Adding flags D to 4 messages  on Maildir[.]
Deleting flags T to 4 messages on Maildir[.]
Adding flags   to 4 messages  on LocalStatus[.]
Adding flags e to 4 messages  on LocalStatus[.]
Adding flags d to 4 messages  on LocalStatus[.]
Adding flags ) to 4 messages  on LocalStatus[.]
Adding flags ( to 4 messages  on LocalStatus[.]
Adding flags l to 4 messages  on LocalStatus[.]
Adding flags n to 4 messages  on LocalStatus[.]
Adding flags t to 4 messages  on LocalStatus[.]
Adding flags \ to 4 messages  on LocalStatus[.]
Adding flags D to 4 messages  on LocalStatus[.]
Deleting flags T to 4 messages on LocalStatus[.]

Fix: use a different variable to store IMAP flags when managing
messages corresponding to responses with UID attribute, e.g.:

*** IMAP.py.orig        Wed Aug 22 18:23:17 2007
--- IMAP.py     Wed Aug 22 18:22:38 2007
*************** class IMAPFolder(BaseFolder):
*** 340,348 ****
              if not ('UID' in attributehash and 'FLAGS' in
              attributehash):
                  # Compensate for servers that don't return a UID
                  attribute.
                  continue
!             flags = attributehash['FLAGS']
              uid = long(attributehash['UID'])
!             self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(flags)
              try:
                  needupdate.remove(uid)
              except ValueError:          # Let it slide if it's not
              in the list
--- 340,348 ----
              if not ('UID' in attributehash and 'FLAGS' in
              attributehash):
                  # Compensate for servers that don't return a UID
                  attribute.
                  continue
!             lflags = attributehash['FLAGS']
              uid = long(attributehash['UID'])
!             self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(lflags)
              try:
                  needupdate.remove(uid)
              except ValueError:          # Let it slide if it's not
              in the list

02/03/08 14:04:35 changed by js

    * attachment flags-fix.patch added.

Delete 02/03/08 14:05:24 changed by js ¶

Unfortunately I have to fetch some of my mail from an Exchange server
(Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1)
and I can confirm that the analysis of the problem is correct, and the
patch given here fixes the problem.

Looking at the code of the processmessagesflags() method I think it
generally is a bug that the "flags" parameter is reused as a local
variable, since the final "for uid in needupdate:" loop needs the
original value of "flags". This only worked by accident.

I'm attaching a unidiff version of the patch which applies cleanly
against Debian unstable's offlineimap 5.99.4.
2008-03-02 22:25:05 -06:00
Florian Friesdorf
08f22be8ea completed: * fixes behaviour when changing flags, without corresp. rights (s,d,w) 2007-07-19 17:52:29 +01:00
Florian Friesdorf
71a76d9a61 fixes behaviour when changing flags, without corresp. rights (s,d,w) 2007-07-12 05:08:47 +01:00
Florian Friesdorf
c305d63e00 fix behaviour for delete/expunge, when lacking rights
This patch maneuvers around python imaplib's mysterious read-only detection
algorithm and correctly calls the UI's deletetoreadonly(), when trying to
delete/expunge in a mailbox without having the necessary rights.
2007-07-12 04:44:11 +01:00
John Goerzen
3305d8cd4d Daniel Jacobowitz patches
fixes deb#433732

Date: Sun, 30 Sep 2007 13:54:56 -0400
From: Daniel Jacobowitz <drow@false.org>
To: offlineimap@complete.org
Subject: Assorted patches

Here's the result of a lazy Sunday hacking on offlineimap.  Sorry for
not breaking this into multiple patches.  They're mostly logically
independent so just ask if that would make a difference.
First, a new -q (quick) option.  The quick option means to only update
folders that seem to have had significant changes.  For Maildir, any
change to any message UID or flags is significant, because checking
the flags doesn't add a significant cost.  For IMAP, only a change to
the total number of messages or a change in the UID of the most recent
message is significant.  This should catch everything except for
flags changes.

The difference in bandwidth is astonishing: a quick sync takes 80K
instead of 5.3MB, and 28 seconds instead of 90.

There's a configuration variable that lets you say every tenth sync
should update flags, but let all the intervening ones be lighter.


Second, a fix to the UID validity problems many people have been
reporting with Courier.  As discussed in Debian bug #433732, I changed
the UID validity check to use SELECT unless the server complains that
the folder is read-only.  This avoids the Courier bug (see the Debian
log for more details).  This won't fix existing validity errors, you
need to remove the local status and validity files by hand and resync.


Third, some speedups in Maildir checking.  It's still pretty slow
due to a combination of poor performance in os.listdir (never reads
more than 4K of directory entries at a time) and some semaphore that
leads to lots of futex wake operations, but at least this saves
20% or so of the CPU time running offlineimap on a single folder:

Time with quick refresh and md5 in loop: 4.75s user 0.46s system 12%
cpu 41.751 total
Time with quick refresh and md5 out of loop: 4.38s user 0.50s system
14% cpu 34.799 total
Time using string compare to check folder: 4.11s user 0.47s system 13%
cpu 34.788 total


And fourth, some display fixes for Curses.Blinkenlights.  I made
warnings more visible, made the new quick sync message cyan, and
made all not explicitly colored messages grey.  That last one was
really bugging me.  Any time OfflineIMAP printed a warning in
this UI, it had even odds of coming out black on black!


Anyway, I hope these are useful.  I'm happy to revise them if you see
a problem.

-- 
Daniel Jacobowitz
CodeSourcery
2007-10-01 22:20:37 +01:00
John Goerzen
2323fdcdb3 Additional date validity check
patch from Mike Gerber

Two times today I have found my offlineimap to have died with this same
situation. It appears as if certain types of messages (both spam in my
situation), cause offlineimap to choke. When it does it cannot proceed.
This means that when I run offlineimap, it pulls in messages from some
folders, then it hits the folder with the bad message and dies, leaving
undownloaded mail on the server. The only fix to this problem is to find
the problem message on the server and remove it by hand. This isn't such
a huge deal for me, since I run the server, but other people have to
come to me to ask me to delete these messages, and until I do they
cannot download their email.

I have captured the output by running script during one of these
incidents, this has been attached. Additionally, I have also attach the 
problematic message.

The patch seems to work for me, might need some Python wizard and better 
testing, though.

fixes deb#396443
2007-08-01 02:25:05 +01:00
Florian Friesdorf
3e7899a9dc IMAP.py: fixed cannot concatenate 'str' and 'list' in assert
r[1] is a list. In case it contains more than one 'str' they are concatenated
using '. '. In processmessagesflags the join is tested, for savemessagesflags I
assume that it is also needed.
2007-07-07 04:51:02 +01:00
John Goerzen
abf9648fd8 Clean up imaplib imports 2007-07-04 18:53:48 +01:00
John Goerzen
1b9fa9ff9a Switch to imaplibutil Internaldate2Epoch 2007-07-04 18:51:57 +01:00
John Goerzen
9d7df0e21a Fix up date parsing to use message date if no rtime is available 2006-08-22 02:13:39 +01:00
John Goerzen
03488ba81b Sync INTERNALDATE <-> mtime
The attached patch adds syncing the INTERNALDATE of IMAP folders with
the mtime of messages in maildir folders.
I want this to happen, because I'm running a dovecot over the maildirs
synced by offlineimap, and that uses the mtime as the INTERNALDATE.
When using mutt to view messages I generally sort based on the received
date, which for IMAP folders is the INTERNALDATE.

Since this is the first real coding I've done in Python the patch may
need to be cleaned up some, but it's working pretty well for me.  I've
added new messages to each side, and the received date has been
preserved going both ways.
2006-08-22 02:09:36 +01:00
John Goerzen
39a18fef60 Update FSF address 2006-08-12 05:15:55 +01:00
Adam Spiers
5fc0e1ae42 helpful comments 2006-05-16 04:40:23 +01:00
Adam Spiers
b39845c488 stop UID FETCH 0 errors 2006-05-16 04:31:44 +01:00
Adam Spiers
2e52bcdafe failure to obtain uid indicated by savemessage_searchforheader return value <= 0, not ValueError 2006-05-16 04:30:48 +01:00
Adam Spiers
720511f3f1 add missing ) 2006-05-16 04:27:57 +01:00
Adam Spiers
f2d4c668d5 return 0 rather than raise exception to indicate that message was saved with unknown UID 2006-05-16 03:34:46 +01:00
Adam Spiers
f11a49f263 make savemessage_searchforheader more robust 2006-05-15 03:51:12 +01:00
John Goerzen
c0d031e760 Remove parens for SEARCH command
Patch from Adam Spiers

Reported that parens were causing trouble for Groupwise IMAP server
2006-05-01 20:40:55 +01:00
John Goerzen
1f25b5393b [324827] Fixed handling of invalid dates
Patch from Nikita V. Youshchenko

From: "Nikita V. Youshchenko" 
To: Debian Bug Tracking System
Subject: offlineimap: exception on mail with broken headers (+fix)
Date: Wed, 24 Aug 2005 13:41:08 +0400

Package: offlineimap
Version: 4.0.10
Severity: normal
Tags: patch

Recently I've got an exception (see below) while using offlineimap.
Exception was probably caused by invalid Date header of (likely spam)
message:
Date: Sat, 20 Aug 2005 4294967295:43:18 -0700
2005-08-24 19:01:42 +01:00
John Goerzen
8137e53c14 Added code to limit command line length for very large mailboxes
Keywords: 


(jgoerzen@complete.org--projects/offlineimap--head--1.0--patch-4)
2004-11-16 23:41:09 +01:00
John Goerzen
440ee1708d Attempt to fix problem with getting back the wrong UID from APPENDUID
Keywords: 


(jgoerzen@complete.org--projects/offlineimap--head--1.0--patch-1)
2004-10-18 20:00:29 +01:00
John Goerzen
d839be3c61 Step 2 of SVN to arch tree conversion 2005-04-16 20:33:35 +01:00