Commit Graph

182 Commits

Author SHA1 Message Date
Sebastian Spaeth
4c558c1b69 Error proof IMAP.APPEND against dropped connections
Make sure that when a connection is dropped during append, we really
discard the broken connection and get a new one, retrying. We retry
indefinitely on the specific abort() Exception, as this is what imaplib2
suggests us to do.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
fb8017991c Rework undocumented listjoin to create UID sequences
This function was badly named and completely undocumented. Rework it to
avoid copying the full UID list using an iterator. Make it possible to
hand it a list of UIDs as strings rather than implicitely relying on the
fact that they are numeric already. Document the code.

The behavior off the function itself remained otherwise unchanged.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
3302940382 Proper error handling for SEARCH and FETCH failures from the server
SEARCH and FETCH were never checking that the IMAP server actually
returned OK. Throw OfflineImapErrors at severity FOLDER in case one of
them fails.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
f755c8b423 Use range 1:* if we want to examine all messages in a folder
Some code cleanup. If we want to examine all messages of a folder, don't
try to find out how many there are and request a long list of all of them,
but simply request 1:*. This obliviates us from the need to force a select
even if we already had the folder selected and it requires us to send a
few less bytes over the wire.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
8532a69458 Clean up the maxsize maxage code
Do away with the wrapping of this code in a try...except KeyError, as
this code cannot conceivably throw a KeyError. Even if it could, it
should be documented why we should simply return() in this case.

Shorten some of the variable names and minor code cleanup while taking
the git blame anyway.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
06bfff7c04 Coalesce SEARCH uid list into sequence set
Rather than passing in huge lists of continuous numbers which eventually
overflow the maximum command line length, we coalesce number ranges
before passing the UID sequence to SEARCH. This should do away with the
error that has been reported with busy mailing lists and 'maxage'.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Nicolas Sebrecht
8c730cf509 Merge branch 'ss/folder.imap-dont-import-copy' into next
Conflicts:
	offlineimap/folder/IMAP.py

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:57:02 +02:00
Sebastian Spaeth
63b9dcd896 folder.IMAP: Don't import copy
It is not needed. list(ALIST) will create a new copy of the list just
fine.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:53:44 +02:00
Sebastian Spaeth
2cf6155282 Error proof IMAP.APPEND against dropped connections
Make sure that when a connection is dropped during append, we really
discard the broken connection and get a new one, retrying. We retry
indefinitely on the specific abort() Exception, as this is what imaplib2
suggests us to do.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:51:49 +02:00
Sebastian Spaeth
0906d0db70 IMAP.cachemessagelist(): Protect against empty folders
When invoked with FETCH 1:* (UID), imaplib returns [None] for empty
folders. We need to protect against this case and simply 'continue' here.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:07:54 +02:00
Sebastian Spaeth
dc103ab9ce Protect IMAP.getmessage() against dropped connections
If a connection is dropped for some reason while fetching a message, the
imapobj.uid command throws an imapbj.abort() Exception which means we are
supposed to retry. Implement a fail loop that drops the connection, gets a
new one and attempts the command another time.

Remove obsolete comment that we need to catch nonexisting messages. We do
now.

GMail seems to drop connections left and right. This patch is a response to
the reported mail "4E5F8D8C.1020005@gmail.com" by zeek
<ezekiel.das@gmail.com>.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:00:48 +02:00
Sebastian Spaeth
0a25408199 Rework undocumented listjoin to create UID sequences
This function was badly named and completely undocumented. Rework it to
avoid copying the full UID list using an iterator. Make it possible to
hand it a list of UIDs as strings rather than implicitely relying on the
fact that they are numeric already. Document the code.

The behavior off the function itself remained otherwise unchanged.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:55:28 +02:00
Sebastian Spaeth
466ded04d9 Make flags a set rather than a list
As this is essentially what it is, a set of values. This allows as
to do set arithmetics to see, e.g. the intersection of 2 flag sets
rather than clunkily having to do:

for flag in newflags:
  if flag not in oldflags:
    oldflags.append(flag)

Also some more code documenting.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 18:56:10 +02:00
Vladimir Marek
b6ac1aecb1 Another way of locating UID of just saved message
It works by fetching all headers of new messages from IMAP server and
searching for our X-OfflineIMAP marker by using regular expression.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-17 20:58:41 +02:00
Sebastian Spaeth
fe388400c4 Better error message when FETCH fails.
We were not including the full server reply into our error message. Fix
that so we get better error logs.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:16:29 +02:00
Sebastian Spaeth
194aa1db3c Catch 'BAD' replies on append()ing a message
append() raises an Exception, in case the IMAP server replies with 'BAD'
(but not when it responds with 'NO') but we were not catching that. Do
catch the situation and also raise an OfflineImapError at MESSAGE
severity, so that we can continue with the next message.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:05:08 +02:00
Sebastian Spaeth
f6b9c68333 IMAP: Don't use assert() in folder.savemessage()
We simply assert()ed that APPENDing a message returned OK, but in some
cases (e.g. Google chat messages) APPEND might return BAD or NO too. We
should be throwing an OfflineImapError here at MESSAGE level, so that we
can continue to sync all other messages, and still give the user some
details on what went wrong at the end of the sync run.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-11 19:14:28 +02:00
Vladimir Marek
e58399ac0b Fixed typo
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-27 18:54:53 +02:00
Vladimir Marek
d5cbdc4c0e Handle when UID can't be found on saved messages
Message was stored to dstfolder, but we can't find it's UID. This means we can't
link current message to the one created in IMAP. So we just delete local message
and on next run we'll sync it back. Also fixed imap.savemessage description.

This was broken by e20d8b9679.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-27 18:54:53 +02:00
Vladimir Marek
86e9c7442b Include message header at better place
It's not enough to place header after first newline, since this might break
multiline rfc0822 folded long header lines. Those are difined as CRLF followed
by white space. Instead we'll search for two successive CRLF sequences which
mark end of mail headers and place our header just before that.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-27 18:54:52 +02:00
Haojun Bao
9d95d7bc62 folder/IMAP: fix typo with maxsize and maxage.
Signed-off-by: Bao Haojun <baohaojun@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-27 18:26:43 +02:00
Sebastian Spaeth
856982a4e6 Throw OfflineImapError when we try to request an inexistant message
During a sync run, someone might remove or move IMAP messages. As we
only cache the list of UIDs in the beginning, we might be requesting
UIDs that don't exist anymore. Protect folder.IMAP.getmessage() against
the response that we get when we ask for unknown UIDs.

Also, if the server responds with anything else than "OK", (eg. Gmail
seems to be saying frequently ['NO', 'Dave I can't let you do that now']
:-) so we should also be throwing OfflineImapErrors here rather than
AssertionErrors.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-15 22:17:04 +02:00
Sebastian Spaeth
e8b633b884 folder/IMAP: Remove buggy duplicate assignment
we do:
  for msgid in imapdata:
      maxmsgid = max(long(msgid), maxmsgid)
and then basically immediately:
   maxmsgid = long(imapdata[0])

throwing away the first assignment although the first method of
assigning is the correct one. The second had been forgotten to be
removed when we introduced the above iteration. This bug would fix a
regression with those broken ZIMBRA servers that send multiple EXISTS
replies.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:52:24 +02:00
Sebastian Spaeth
846070b240 Fix typos in months names
All months names are 3-letter abbreviated, but accidentally June and
July slipped through. Thanks to the heads up by
Philipp Kern <pkern@debian.org>.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:20:25 +02:00
Sebastian Spaeth
c70a621709 Limit msg body length in debug output
We were outputting full message bodies to the debug log (often stderr),
and then again (as they go over the imaplib2 wire, imaplib logs
everything too). Not only is quite a privacy issue when sending in debug
logs but it can also freeze a console for quite some time. Plus it
bloats debug logs A LOT.

Only output the first and last 100 bytes of each message body to the
debug log (we still get the full body from imaplib2 logging). This
limits privacy issues when handing the log to someone else, but usually
still contains all the interesting bits that we want to see in a log.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-09 20:58:46 +02:00
Sebastian Spaeth
b3a383d151 accounts: handle OfflineImapError severity FOLDER
Throw an OfflineImapError when SELECTing a folder is unsuccessful and
bail out with a FOLDER serverity. In accounts.py catch all
OfflineImapErrors and either just log the error and skip the folder or
bubble it up if it's severe.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 13:56:04 +02:00
Sebastian Spaeth
f4081985dc Prettify and use new uidexists() helper function
Make the folder classes use uidexists() more. Add some code
documentation while going through.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:50:22 +02:00
Nicolas Sebrecht
0bc1c9583e Merge branch 'ss/folder-cleanup-getmessage' into next
Conflicts:
	offlineimap/folder/IMAP.py
2011-04-27 22:52:17 +02:00
Sebastian Spaeth
1ff628bd5c folder/IMAP: cleanup getmessage()
Add some comments how the data structures actually look like.
Describe the function properly, and make sure we only hold on to the
data connection as quickly as possible.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 22:49:18 +02:00
Sebastian Spaeth
36eb37b47d IMAP: reduce quickchanged() checks
For each folder we were making a second IMAP request asking for the
latest UID and compared that with the highest UID in our
statusfolder. This catched the case that 1 mail has been deleted by
someone else and another one has arrived since we checked, so that the
total number of mails appears to not have changed.

We don't capture anymore this case in the quickchanged() case.

It improves my performance from 8 to about 7.5 seconds per check (with lots of
variation) and we would benefit even more in the IMAP<->IMAP case as we do one
additional IMAP lookup per folder on each side then.

Do cleanups on whitespaces while in this file.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-25 12:58:47 +02:00
Sebastian Spaeth
58220fd8e7 Replace calls to getmessagelist() to alternatives
getmessagelist() is slow for the mapped UID case, so replace some of its
occurences with calls that are optimized for this case, ie
getmessagecount() and uidexists().

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-and-tested-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 17:53:14 +02:00
David Favro
954655b7ec Fixed bug: wrong number of arguments to debug() [IMAP.py].
Signed-off-by: David Favro <offlineimap@meta-dynamic.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 16:49:21 +02:00
Sebastian Spaeth
09515f8f90 Increase compatability with Gmail
When uploading a new message to Gmail we need to find out the UID it
assigned it, but Gmail does not advertize the UIDPLUS extension (in all
cases) and it fails to find the email that we just uploaded when
searching for it. This prevented us effectively from uploading to
gmail.

See analysis in
http://lists.alioth.debian.org/pipermail/offlineimap-project/2011-March/001449.html
for details on what is going wrong.

This patch increases compatability with Gmail by checking for APPENDUID
responses to an APPEND action even if the server did not claim to
support it. This restores the capability to upload messages to the
*broken* Gmail IMAP implementation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-03-25 18:42:16 +01:00
Sebastian Spaeth
fd28c5a2d3 folder/IMAP: savemessage() should just save flags if uid already exists
As the LocalStatus and UIDMap backend already did: If the uid already
exists for savemessage(), only modify the flags and don't append a new
message.

We don't invoke savemessage() on messages that already exist in our sync
logic, so this has no change on our current behavior. But it makes
backends befave more consistent with each other.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 18:05:16 +01:00
Ethan Glasser-Camp
1bf4bee5e6 Update to match semantics of new imaplib2
The biggest change here is that imapobj.untagged_responses is no
longer a dictionary, but a list. To access it, I use the semi-private
_get_untagged_response method.

* offlineimap/folder/IMAP.py (IMAPFolder.quickchanged,
  IMAPFolder.cachemessagelist): imaplib2 now explicitly removes its
  EXISTS response on select(), so instead we use the return values from
  select() to get the number of messages.

* offlineimap/imapserver.py (UsefulIMAPMixIn.select): imaplib2 now
  stores untagged_responses for different mailboxes, which confuses us
  because it seems like our mailboxes are "still" in read-only mode when
  we just re-opened them.  Additionally, we have to return the value
  from imaplib2's select() so that the above thing works.

* offlineimap/imapserver.py (UsefulIMAPMixIn._mesg): imaplib2 now
  calls _mesg with the name of a thread, so we display this
  information in debug output. This requires a corresponding change to
  imaplibutil.new_mesg.

* offlineimap/imaplibutil.py: We override IMAP4_SSL.open, whose
  default arguments have changed, so update the default arguments. We
  also subclass imaplib.IMAP4 in a few different places, which now
  relies on having a read_fd file descriptor to poll on.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
f9413226b8 Import imaplib2 instead of imaplib
imaplib2 has slightly different semantics than standard imaplib, so
this patch will break the build, but I thought it was helpful to have it as
a separate commit.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Sebastian Spaeth
2946a1ea5d Convert rfc822 module to email module
The rfc822 module has been deprecated since python 2.3, and conversion to
the email module is straightforward, so let us do that. rfc822 is
completely gone in python3.

This also fixes a bug that led to offlineimap abortion (but that code path
is apparently usually not exercised so I did not notice:
rfc822|email.utils.parsedate return a tuple which has no named attributes,
but we were using them later in that function. So pass the tuple into a
struct_time() to get named attributes.

While reading the docs, I noticed that email.parsedate returns invalid
daylight savings information (is_dst attribute), and we are using it
anyway. Oh well, the imap server might think the mails are off by an hour
at worst.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-13 20:30:35 +01:00
Sebastian Spaeth
3b8e1f91cd Implement APPENDUID support
Rather than inserting our own home-grown header, everytime we save a
message to an IMAP server, we check if we suport the UIDPLUS extension
which provides us with an APPENDUID reply. Use that to find the new UID
if possible, but keep the old way if we don't have that extension.

If a folder is read-only, return the uid that we have passed in per API
description in folder.Base.py

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
96db608e4d Remove convoluted assert statements
The working horse of the savemessage() function, imaplib.append() was
hidden away in an assert statement. Pull the real functions out of the
asserts and simply assert on the return values. This looks less
convoluted and makes this easier to understand in my opinion.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
d22c762385 Factor out the date guessing/retrieving
savemessage was too long and complex. Factor out the date guessing part
of the function and put it into a function of its own. The logic of the
date guessing is the same, however, we do not use the
imaplib.Time2InternalDate() function as it is buggy
(http://bugs.python.org/issue11024) and returns localized patches. So we
create INTERNALDATE ourselves and pass it to append() as a string.

This commit fixes a bug that international users used to pass an invalid
date to the IMAP server, which the server will either ignore or complain
about.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
419f27418e Simplify & document savemessage_getnewheader
savemessage_getnewheader was an undocmented, cryptic and overengineered
function. It generates a new unique value that can be used as a mail
header to be inserted. For this it used LOTS of randomness sources: hash
of the mail content, hash of the folder name, hash of the repository
name, the current time, a random() value, and the offlineimap version string.
All we need is something random. So reduce this to hash of content
appended by a random integer. Sufficient and somewhat faster to calculate.

Rename the function to actually describe accurately what it does or
would you have guessed that savemessage_getnewheader() did nothing more
than returning ('X-OfflineIMAP', <randomstring> )? Rename to
generate_randomheader() to make it clearer what this is all about.

Also document the function, describing what it does, and what it returns.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:09 +01:00
Sebastian Spaeth
6398bf7795 Don't require the string module
There is no need for using the string module if all we want is to split
a string at the white space. All pythons since at least 2.4 can do that.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-02-21 13:20:21 +01:00
Nicolas Sebrecht
9afb432053 folder/IMAP.py: accord __version__ with import
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-27 19:45:14 +01:00
Nicolas Sebrecht
a9c63078a2 Merge branch 'master' into next 2011-01-20 20:12:02 +01:00
Knut Anders Hatlen
9ad1810e43 understand multiple EXISTS replies from servers like Zimbra
Lars Thalmann <web@larsthalmann.com> writes:

> It seems some servers (Zimbra?) respond to imap SELECT requests with
> multiple EXISTS lines:
>
> ? SELECT INBOX
> * 500 EXISTS
> * 0 RECENT
> * 1000 EXISTS
> * 0 RECENT
> * 1500 EXISTS
> * 0 RECENT
> * 1567 EXISTS
> * 0 RECENT

Signed-off-by: Knut Anders Hatlen <kahatlen@gmail.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-20 20:06:46 +01:00
Nicolas Sebrecht
ee8a00cce2 Merge branch 'ss/ui' into next
Conflicts:
	offlineimap/folder/IMAP.py

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-05 19:31:08 +01:00
Sebastian Spaeth
de618d84f3 Save ui as self.ui in folder/Base class
Rather than regetting the ui with UIBase.getglobalui() all the time, we get it once in the base class and let all derivative classes just make use of self.ui rather than refetching the ui all the time, this makes for a bit less code and shorter lines.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-05 19:24:00 +01:00
Sebastian Spaeth
0b5b38d298 Define version constants etc in __init__.py
Move central constant definitions into __init__.py.  This does away
with version.py which contained nothing else and __init__.py is where
things like __VERSION__ are usually defined.

This commit also changes code to use offlineimap.__version__ rather
than offlineimap.version.__version__ as was before. Cleaned up some
duplicate or unneeded imports while touching those, formatting import
statements per PEP8 (one import per row).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2010-12-22 19:45:01 +01:00
Johannes Stezenbach
573d7ed52a Change X-OfflineIMAP header format
Change the X-OfflineIMAP header to work around possible Exchange MAPI
table overflow problem described in
http://article.gmane.org/gmane.mail.imap.offlineimap.general/1699
(It is unknown whether this problem still exits in current
Exchange versions, but let's assume the worst.)

The X-OfflineIMAP header is neccessary with some IMAP servers to
reliably determine the UID of a new messages uploaded to the server
by using the "UID SEARCH HEADER name string" command.  Since this
command compares header name and value it is sufficient to have
a unique header value and a non-unique header name.

Note that a message can have more than one X-OfflineIMAP header if
the message was copied between IMAP folders multiple times.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2010-12-09 19:52:48 +01:00
John Goerzen
1148df7685 Merge branch 'maxage'
Conflicts:
	offlineimap/folder/IMAP.py
2009-08-16 09:29:00 -05:00
Mike Dawson
cde94e5047 Patch for maxage and maxsize options
Dear All,
Attached is the patch that I have developed to provide maxage and
maxsize options.  You can thus sync only the last x days of messages and
exclude large messages.  All details in the attached git file.

Regards,

-Mike

-- Attached file included as plaintext by Ecartis --
-- File: submit

From 04fead2b46a79675a5b29de6f2b4088b9c9448e5 Mon Sep 17 00:00:00 2001
From: mike <mike@mikelaptop.(none)>
Date: Sun, 16 Aug 2009 17:00:49 +0430
Subject: [PATCH] Patch to provide maxage and maxsize account options to exclude old/large messages

This is designed to make offlineimap even better for low bandwidth connections.

maxage allows you to specify a number of days and only messages within
that range will be considered by offlineimap for the sync.  This can be
useful if you would like to start using offlineimap with a large
existing account and do not want to import large archives of mail.

maxsize allows you to specify the maximum size of a message to consider so
that you can exclude messages with large attachments etc.

In both cases the cachemessagelist function of the folder was modified to ignore
messages that do not meet the criteria.  If the criteria are not specified
then the existing code will be executed the same as before.  If a message
does not meet the criteria it will be as though this message does not exist
- offlineimap will completely ignore it.  It will not have flags updated,
it will not be deleted, it will not be considered at all.

When operating against an IMAP repository a server side search function
is used.  This of course requires support for server side search.

I have tested this with either option, no options etc. against IMAP, Maildir
and Gmail.  I have run variations of this patch here for the last 3 weeks or
so syncing about 4 accounts normally.
2009-08-16 09:27:49 -05:00
John Goerzen
2433dc9f06 Merge branches 'master' and 'master' of ssh://jpgarch@complete.org/~jpgarch/git/offlineimap 2009-08-12 15:25:26 -05:00
John Goerzen
1f8024a70e [imaplib2 removal] Revert "Implementation of IMAP IDLE"
This reverts commit 3847d0ba9d.
2009-08-12 14:49:58 -05:00
John Goerzen
e66fd5f51a Don't request INTERNALDATE when we don't need it. Patch from Marc MERLIN. 2009-07-16 20:57:07 -05:00
James Bunton
3847d0ba9d Implementation of IMAP IDLE
- Use a newer version of imaplib
 - Hijack the keepalive process to send IDLE instead of NOOP
2009-02-10 11:27:48 +11:00
John Goerzen
6e6720d1b0 Update for 5.99.7 2008-03-04 08:32:29 -06:00
John Goerzen
3cb3a675b8 Revert "completed: * fixes behaviour when changing flags, without corresp. rights (s,d,w)"
This reverts commit 08f22be8ea.

Conflicts:

	offlineimap/folder/IMAP.py
2008-03-04 08:15:48 -06:00
John Goerzen
cabd8d1735 Revert "fix behaviour for delete/expunge, when lacking rights"
This reverts commit c305d63e00.

Zak Smith reported it continued to be a problem
2008-03-04 08:13:48 -06:00
John Goerzen
9a168c573e Revert "fixes behaviour when changing flags, without corresp. rights (s,d,w)"
This reverts commit 71a76d9a61.

Zak Smith reported a problem at:

      self.processmessagesflags('+', uidlist, flags)
  File "/usr/lib/python2.5/site-packages/offlineimap/folder/IMAP.py",
  line 372, in processmessagesflags
    myrights = imapobj.myrights(self.getfullname())[1][0].split()[1]
IndexError: list index out of range

Conflicts:

	offlineimap/folder/IMAP.py
2008-03-03 21:20:53 -06:00
John Goerzen
08a579657a Fix handling of servers that return UIDs in some FETCH responses
closes #22

from pistore in OfflineIMAP #22:

When an IMAP flag update is performed for multiple messages, some IMAP
servers (e.g. Exchange) return the UID attribute only for some of the
FETCH untagged responses, as shown in the following log:

21:19.04 > DCKF8 UID STORE 66050,50613,52164,40043,40055,25874 +FLAGS
(\Deleted)
21:19.36 < * 35 FETCH (FLAGS (\Seen \Deleted) UID 25874)
21:19.36 < * 321 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 322 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 560 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 581 FETCH (FLAGS (\Seen \Deleted) UID 52164)
21:19.62 < * 1022 FETCH (FLAGS (\Seen \Deleted))
21:19.62 < DCKF8 OK STORE completed.

Function IMAPFolder.processmessagesflags is able to manage the servers
which return the UID and the servers which do not return it, but is
not able to deal with the mixed behavior shown above.

The problem is that the fragment of function
IMAPFolder.processmessagesflags that handles the responses with UID
attribute uses variable flags to store the list of flags of the
message in the IMAP format ("flags = attributehashFLAGS?"), while the
fragment that handles the responses without UID expects variable
"flags" to contain the list of modified flags passed to the function
in Maildir format ("self.messagelist[uid]flags?.append(flag)").

As a consequence, the wrong list of flags is used for the messages
without UID, leading to the addition of "strange" flags to the Maildir
messages:

Syncing messages IMAP[INBOX] -> Maildir[.]
Adding flags   to 4 messages  on Maildir[.]
Adding flags e to 4 messages  on Maildir[.]
Adding flags d to 4 messages  on Maildir[.]
Adding flags ) to 4 messages  on Maildir[.]
Adding flags ( to 4 messages  on Maildir[.]
Adding flags l to 4 messages  on Maildir[.]
Adding flags n to 4 messages  on Maildir[.]
Adding flags t to 4 messages  on Maildir[.]
Adding flags \ to 4 messages  on Maildir[.]
Adding flags D to 4 messages  on Maildir[.]
Deleting flags T to 4 messages on Maildir[.]
Adding flags   to 4 messages  on LocalStatus[.]
Adding flags e to 4 messages  on LocalStatus[.]
Adding flags d to 4 messages  on LocalStatus[.]
Adding flags ) to 4 messages  on LocalStatus[.]
Adding flags ( to 4 messages  on LocalStatus[.]
Adding flags l to 4 messages  on LocalStatus[.]
Adding flags n to 4 messages  on LocalStatus[.]
Adding flags t to 4 messages  on LocalStatus[.]
Adding flags \ to 4 messages  on LocalStatus[.]
Adding flags D to 4 messages  on LocalStatus[.]
Deleting flags T to 4 messages on LocalStatus[.]

Fix: use a different variable to store IMAP flags when managing
messages corresponding to responses with UID attribute, e.g.:

*** IMAP.py.orig        Wed Aug 22 18:23:17 2007
--- IMAP.py     Wed Aug 22 18:22:38 2007
*************** class IMAPFolder(BaseFolder):
*** 340,348 ****
              if not ('UID' in attributehash and 'FLAGS' in
              attributehash):
                  # Compensate for servers that don't return a UID
                  attribute.
                  continue
!             flags = attributehash['FLAGS']
              uid = long(attributehash['UID'])
!             self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(flags)
              try:
                  needupdate.remove(uid)
              except ValueError:          # Let it slide if it's not
              in the list
--- 340,348 ----
              if not ('UID' in attributehash and 'FLAGS' in
              attributehash):
                  # Compensate for servers that don't return a UID
                  attribute.
                  continue
!             lflags = attributehash['FLAGS']
              uid = long(attributehash['UID'])
!             self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(lflags)
              try:
                  needupdate.remove(uid)
              except ValueError:          # Let it slide if it's not
              in the list

02/03/08 14:04:35 changed by js

    * attachment flags-fix.patch added.

Delete 02/03/08 14:05:24 changed by js 

Unfortunately I have to fetch some of my mail from an Exchange server
(Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1)
and I can confirm that the analysis of the problem is correct, and the
patch given here fixes the problem.

Looking at the code of the processmessagesflags() method I think it
generally is a bug that the "flags" parameter is reused as a local
variable, since the final "for uid in needupdate:" loop needs the
original value of "flags". This only worked by accident.

I'm attaching a unidiff version of the patch which applies cleanly
against Debian unstable's offlineimap 5.99.4.
2008-03-02 22:25:05 -06:00
Florian Friesdorf
08f22be8ea completed: * fixes behaviour when changing flags, without corresp. rights (s,d,w) 2007-07-19 17:52:29 +01:00
Florian Friesdorf
71a76d9a61 fixes behaviour when changing flags, without corresp. rights (s,d,w) 2007-07-12 05:08:47 +01:00
Florian Friesdorf
c305d63e00 fix behaviour for delete/expunge, when lacking rights
This patch maneuvers around python imaplib's mysterious read-only detection
algorithm and correctly calls the UI's deletetoreadonly(), when trying to
delete/expunge in a mailbox without having the necessary rights.
2007-07-12 04:44:11 +01:00
John Goerzen
3305d8cd4d Daniel Jacobowitz patches
fixes deb#433732

Date: Sun, 30 Sep 2007 13:54:56 -0400
From: Daniel Jacobowitz <drow@false.org>
To: offlineimap@complete.org
Subject: Assorted patches

Here's the result of a lazy Sunday hacking on offlineimap.  Sorry for
not breaking this into multiple patches.  They're mostly logically
independent so just ask if that would make a difference.
First, a new -q (quick) option.  The quick option means to only update
folders that seem to have had significant changes.  For Maildir, any
change to any message UID or flags is significant, because checking
the flags doesn't add a significant cost.  For IMAP, only a change to
the total number of messages or a change in the UID of the most recent
message is significant.  This should catch everything except for
flags changes.

The difference in bandwidth is astonishing: a quick sync takes 80K
instead of 5.3MB, and 28 seconds instead of 90.

There's a configuration variable that lets you say every tenth sync
should update flags, but let all the intervening ones be lighter.


Second, a fix to the UID validity problems many people have been
reporting with Courier.  As discussed in Debian bug #433732, I changed
the UID validity check to use SELECT unless the server complains that
the folder is read-only.  This avoids the Courier bug (see the Debian
log for more details).  This won't fix existing validity errors, you
need to remove the local status and validity files by hand and resync.


Third, some speedups in Maildir checking.  It's still pretty slow
due to a combination of poor performance in os.listdir (never reads
more than 4K of directory entries at a time) and some semaphore that
leads to lots of futex wake operations, but at least this saves
20% or so of the CPU time running offlineimap on a single folder:

Time with quick refresh and md5 in loop: 4.75s user 0.46s system 12%
cpu 41.751 total
Time with quick refresh and md5 out of loop: 4.38s user 0.50s system
14% cpu 34.799 total
Time using string compare to check folder: 4.11s user 0.47s system 13%
cpu 34.788 total


And fourth, some display fixes for Curses.Blinkenlights.  I made
warnings more visible, made the new quick sync message cyan, and
made all not explicitly colored messages grey.  That last one was
really bugging me.  Any time OfflineIMAP printed a warning in
this UI, it had even odds of coming out black on black!


Anyway, I hope these are useful.  I'm happy to revise them if you see
a problem.

-- 
Daniel Jacobowitz
CodeSourcery
2007-10-01 22:20:37 +01:00
John Goerzen
2323fdcdb3 Additional date validity check
patch from Mike Gerber

Two times today I have found my offlineimap to have died with this same
situation. It appears as if certain types of messages (both spam in my
situation), cause offlineimap to choke. When it does it cannot proceed.
This means that when I run offlineimap, it pulls in messages from some
folders, then it hits the folder with the bad message and dies, leaving
undownloaded mail on the server. The only fix to this problem is to find
the problem message on the server and remove it by hand. This isn't such
a huge deal for me, since I run the server, but other people have to
come to me to ask me to delete these messages, and until I do they
cannot download their email.

I have captured the output by running script during one of these
incidents, this has been attached. Additionally, I have also attach the 
problematic message.

The patch seems to work for me, might need some Python wizard and better 
testing, though.

fixes deb#396443
2007-08-01 02:25:05 +01:00
Florian Friesdorf
3e7899a9dc IMAP.py: fixed cannot concatenate 'str' and 'list' in assert
r[1] is a list. In case it contains more than one 'str' they are concatenated
using '. '. In processmessagesflags the join is tested, for savemessagesflags I
assume that it is also needed.
2007-07-07 04:51:02 +01:00
John Goerzen
abf9648fd8 Clean up imaplib imports 2007-07-04 18:53:48 +01:00
John Goerzen
1b9fa9ff9a Switch to imaplibutil Internaldate2Epoch 2007-07-04 18:51:57 +01:00
John Goerzen
9d7df0e21a Fix up date parsing to use message date if no rtime is available 2006-08-22 02:13:39 +01:00
John Goerzen
03488ba81b Sync INTERNALDATE <-> mtime
The attached patch adds syncing the INTERNALDATE of IMAP folders with
the mtime of messages in maildir folders.
I want this to happen, because I'm running a dovecot over the maildirs
synced by offlineimap, and that uses the mtime as the INTERNALDATE.
When using mutt to view messages I generally sort based on the received
date, which for IMAP folders is the INTERNALDATE.

Since this is the first real coding I've done in Python the patch may
need to be cleaned up some, but it's working pretty well for me.  I've
added new messages to each side, and the received date has been
preserved going both ways.
2006-08-22 02:09:36 +01:00
John Goerzen
39a18fef60 Update FSF address 2006-08-12 05:15:55 +01:00
Adam Spiers
5fc0e1ae42 helpful comments 2006-05-16 04:40:23 +01:00
Adam Spiers
b39845c488 stop UID FETCH 0 errors 2006-05-16 04:31:44 +01:00
Adam Spiers
2e52bcdafe failure to obtain uid indicated by savemessage_searchforheader return value <= 0, not ValueError 2006-05-16 04:30:48 +01:00
Adam Spiers
720511f3f1 add missing ) 2006-05-16 04:27:57 +01:00
Adam Spiers
f2d4c668d5 return 0 rather than raise exception to indicate that message was saved with unknown UID 2006-05-16 03:34:46 +01:00
Adam Spiers
f11a49f263 make savemessage_searchforheader more robust 2006-05-15 03:51:12 +01:00
John Goerzen
c0d031e760 Remove parens for SEARCH command
Patch from Adam Spiers

Reported that parens were causing trouble for Groupwise IMAP server
2006-05-01 20:40:55 +01:00
John Goerzen
1f25b5393b [324827] Fixed handling of invalid dates
Patch from Nikita V. Youshchenko

From: "Nikita V. Youshchenko" 
To: Debian Bug Tracking System
Subject: offlineimap: exception on mail with broken headers (+fix)
Date: Wed, 24 Aug 2005 13:41:08 +0400

Package: offlineimap
Version: 4.0.10
Severity: normal
Tags: patch

Recently I've got an exception (see below) while using offlineimap.
Exception was probably caused by invalid Date header of (likely spam)
message:
Date: Sat, 20 Aug 2005 4294967295:43:18 -0700
2005-08-24 19:01:42 +01:00
John Goerzen
8137e53c14 Added code to limit command line length for very large mailboxes
Keywords: 


(jgoerzen@complete.org--projects/offlineimap--head--1.0--patch-4)
2004-11-16 23:41:09 +01:00
John Goerzen
440ee1708d Attempt to fix problem with getting back the wrong UID from APPENDUID
Keywords: 


(jgoerzen@complete.org--projects/offlineimap--head--1.0--patch-1)
2004-10-18 20:00:29 +01:00
John Goerzen
d839be3c61 Step 2 of SVN to arch tree conversion 2005-04-16 20:33:35 +01:00