Commit Graph

229 Commits

Author SHA1 Message Date
Dmitrijs Ledkovs
e8c40a9285 Merge branch 'master' into next 2013-07-10 02:49:52 +01:00
Adam Spiers
aa5202396c fix typos 2013-03-27 12:42:32 +00:00
Eygene Ryabinkin
611f6e89c0 IMAP class: don't suggest multithreading in single-threaded mode
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-02-11 23:34:28 +04:00
Tobias Thierer
caef9a72fc Fix IMAP folder throwing away time zone when parsing email Date headers
Fix imapfolder.getmessageinternaldate misparsing the Date:
header from emails due to a bug or surprising behaviour by
email.utils.parsedate. This is because email.utils.parsedate's
return value contains the unadjusted hour value from the string
parsed but does not include information about the time zone in
which it is specified. For example (Python 2.7.3):

$ python -c "import email.utils;
  print email.utils.parsedate('Mon, 20 Nov 1995 19:12:08 -0500')"
 (1995, 11, 20, 19, 12, 8, 0, 1, -1)

(the -1 is the isdst field); the -0500 time zone is completely
ignored, so e.g. the same input with time "19:12:08 +0300" has
the same result. When passed to time.struct_time as allowed per
the parsedate documentation, this time is interpreted in GMT and
thus deviates from the correct value by the timezone offset
(in this example, -5 hours).

I consider this a bug in email.utils.parsedate: In my opinion,
since the return value of the parsetime doesn't include a timezone,
it should be expressed in terms of UTC rather than in terms of the
time zone from the Date header; the existence of
email.utils.parsedate_tz, to which I've switched, indicates that
maybe the authors were aware of this problem.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-10-03 11:16:13 +02:00
Sebastian Spaeth
36156fa985 Remove the APPENDUID hack, previously introduced
As Gmail was only announcing the presence of the UIDPLUS extension
after we logged in, and we were then only getting server
capabilities before, a hack was introduced that checked the
existence of an APPENDUID reply, even if the server did not claim
to support it.

However, John Wiegley reports problems, where the APPENDUID would
be None, and we attempt to go this path (it seems that imaplib2
returns [None] if there is no such reply, so our test here for "!="
might fail. Given that this is an undocumented imaplib2 function
anyway, and we do fetch gmail capabilities after authentication,
this hack should no longer be necessary.

We had problems there earlier, where imapobj.response() would
return [None] although we had received a APPENDUID response from
the server, this might need more debugging and testing.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-06-06 10:13:42 +02:00
Sebastian Spaeth
61e754c65e Do not try to release IMAP connection twice
Reported by sharat87 in https://github.com/spaetz/offlineimap/pull/38,
he would often get an unhandled Exception when trying to
releaseconnection() a connection that was not in the pool of
connections.

The reason this could happen is that when folder.IMAP.quickchanged()
raises an Exception in select(), we would release the connection in the
"except" handling, and than release the same connection in the "finally"
clause, which led to the error. The right thing is to only release the
connection once, of course.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-04-21 13:32:14 +02:00
Sebastian Spaeth
0e6b4ae798 Revert "Clean up and improve APPENDUID handling"
This reverts commit 4d47f7bf3c.

This is one of two candidates for introducing the instabilities that
John Wiegley observed. We need to reintroduce with careful testing only.

The original patch has been mostly reverted.
2012-02-24 08:35:59 +01:00
Sebastian Spaeth
bc73c11239 Revert "Don't CHECK imapserver after each APPEND"
This reverts commit 47390e03d6.

It is one of two potential candidates for the APPENDUID
regression that John Wiegley reported. We need to examine this
carefully before reintroducing this patch.

Resolved Changelog.draft.rst conflict.
2012-02-24 08:31:31 +01:00
Sebastian Spaeth
5ef69e95c0 Prevent modifications on a folder level to occur in dry-run
Prevent savemessage(), and savemessageflags() to occur in dryrun mode in
all backends. Still need to protect against deletemessage().

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 13:17:05 +01:00
Sebastian Spaeth
7da50e638d folder/IMAP: better error when savemessage fails
If we cannot identify the new UID after a sendmessage(), log a better error
message, including the server response for better debugging.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-16 16:49:06 +01:00
Sebastian Spaeth
5c598d7e74 dict.has_key(a) --> a in dict
has_key() is gone in python3, so use a more modern syntax here.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
e6e708ec78 Don't use sort() on dict values()
This won't work in python3 anymore, so just use sorted() when needed.
In one case, we could remove the sort() completely as were were sanity checking
one line above, that we only having one UID as response which makes sorting
unneeded.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
b33f2452f0 Use "from . import" for relative imports
Will fail in python3 otherwise.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
81fc20c7ca Remove python<2.6 import workarounds (set & ssl)
'set' is builtin since python2.6, so remove the imports. Also 'ssl' exists
since 2.6 and has everything we need, so no need for conditional import
tests here anymore.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
c5468ae599 raise Exception, "text" --> raise Exception("text")
To have the code work in python3, we need to convert all occurences of

raise Exception, "text" to be proper functions. This style also adheres to PEP8.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
0844d27f9f except Ex, e: --> except Ex as e:
Nudge us towards python3 compatability by converting deprecated python2 syntax.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
47390e03d6 Don't CHECK imapserver after each APPEND
Most servers support the UIDPLUS extension, and we don't have to search
headers after each uploaded message. There is no need to CHECK the imap
server after each message when there is no need to search headers.

I have not measured the performance impact on real world servers, but
this lets us do less unneeded work in the common case.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 14:43:12 +01:00
Sebastian Spaeth
4d47f7bf3c Clean up and improve APPENDUID handling
We were not cleaning out possibly existing APPENDUID messages before
APPENDing a new message. In case an old message were still hanging
around, this could *possibly* lead to retrieving and old UID. Things
should have been fine, but we do want to play safe here.

Also, make use of the "official" imaplib2 .response() command rather
than the internal _get_untagged_response() function.

Remove the hack that we would be looking for APPENDUID responses even if
the server claimed not to support the UIDPLUS ext. We now poll server
CAPABILITIES after login, and Gmail does provide us with the UIDPLUS
capability after login.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 14:33:16 +01:00
Sebastian Spaeth
9453e1d955 Fix getuidvalidity crash (UIDVALIDITY returning None)
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.

1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
   from the imaplib2 result stack, so subsequent calls would return "None".
   As various functions can invoke this, this led to some errors that we
   avoid by caching the current UIDVALIDITY value in the Folder instance.

There are more simplifications and improvements to be made.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:02:13 +01:00
Sebastian Spaeth
3284e010ff Revert "use .response() rather _get_untagged_response()"
Recently the internal function use of imaplib2's _get_untagged_response()
was switched to use the public documented .response() function (which
should return the same data). However within a few fays we received reports
that both uses of a) the UIDVALIDITY fetching and b) the APPENDUID fetching
returned [None] as data although the IMAP log definitely shows that data
was returned. Revert to using the undocumented internal imaplib2 function,
that seemed to have worked without problems. This needs to be taken up to
the imaplib2 developer.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-09 09:51:43 +01:00
Sebastian Spaeth
7184ec28cc Sanity check return value of UIDVALIDTY response
We have a reported case where response('UIDVALIDITY') returned [None]
which results in an ugly non-intuitive crash. Sanity check and report
something nicer.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 12:26:47 +01:00
Sebastian Spaeth
50de2174bf Allow to pass 'force' arg to selectro() to enforce a new select
Pass through the 'force' argument from selectro() to select() so that it
can also enforce a new SELECT even if we already are on that folder.

Also change the default parameter from '0' to 'False' to make clear that
this is a Bool.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 11:29:54 +01:00
Sebastian Spaeth
3a1eab7383 Remove a stray debug output that slipped in the previous commit.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 01:31:24 +01:00
Sebastian Spaeth
ec63b4fe6b replace imaplib internal _get_untagged_response with public functions
We were using the internal imaplib2 _get_untagged_response() functions a
few times. Replace 3 of these calls with 2 calls to the public function
response() rather than fudging with internals that could change anytime.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 01:28:20 +01:00
Sebastian Spaeth
8fc7227189 SEVERE: Fix getting wrong UID back on IMAP upload
This change looks harmless, but it fixes a severe bugfix, potentially
leading to data loss! It fixes the "on n new uploads, it will redownload
n-1, n-2, n-3,... messages during the next syncs" condition, and this is
what happens:

If there are more than one Mails to upload to a server, we do that by
repeatedly invoking folder.IMAP.savemessage(). If the server supports
the UIDPLUS extension we query the resulting UID by doing a:

imapobj._get_untagged_response('APPENDUID', True)

and that is exactly the problem. The "True" part causes the reply to
remain in the "response stack" of the imaplib2 library. When we do
the same call on a subsequent message and the connection is still on the
same folder, we will get the same UID response back (imaplib2 only looks
for the first matching response and returns that). The only time we
clear the response stack, is when the IMAP connection SELECTS a
different folder.

This means that when we upload 10 messages, the IMAP server gives us
always the same UID (that of the first one) back. And trying to write
out 10 different messages with the same UID will confuse OfflineIMAP.

This is the reason why we saw the ongoing UPLOADING/DOWNLOADING behavior
that people reported. And this is the reason why we saw the
inconsistency in the UID mapping in the IMAP<->IMAP case.

I urge everyone to upgrade ASAP. Sorry for that, I don't know why the
problem only became prevalent in the recent few releases as this code
has been there for quite a while.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 21:57:48 +01:00
Sebastian Spaeth
78a37f27ef WIP, revamp how we treat top-level dirs
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 19:08:48 +01:00
Sebastian Spaeth
de537dc09c Implement change_message_uid
Previously, assigning a new UID to a mapped IMAP or Maildir repository
was done by loading the "local" item, saving it under a new UID and
deleting the old one. This involved lots of disk activity for nothing
more than an effective file rename in Maildirs, and lots of network
usage in the MappedUID cases.

We do this on every upload from a local to a remote item, so that can
potentially be quite expensive. This patch lets backends that support it
(Maildir, MappedUID) efficiently rename the file rather than having to
read the mail content, write it out as a new file and delete the old
file. This speeds up uploads from Maildir and the MappedUID server.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 19:08:48 +01:00
Sebastian Spaeth
656405616d Drop connection if it might be bad on APPEND
1) Differentiate error messages between imaplib.abort and imaplib.error
exceptions in the log.

2) Drop connections in the case of imapobj.error, it also might denote a
   broken connection.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 10:36:30 +01:00
Sebastian Spaeth
33b4a16dac Fix mbox.select(foldername) readonly parameter comparison
The default parameter value was "None", and we were comparing that
directly to the imaplib2 value of is_readonly which is False or True, so
the comparison always returned "False".

Fix this by setting the default parameter to "False" and not
"None". Also convert all users of that function to use False/True.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 08:57:01 +01:00
Sebastian Spaeth
9578f29195 Use 'reference' value when creating an IMAP directory
A repositories 'reference value is always prefixed to the full folder
path, so we should do so when creating a new one. The code had existed
but was commented out since 2003, I guess the "reference" option is not
too often used.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-30 16:57:07 +02:00
Sebastian Spaeth
4bfc1e8227 except imapobj.abort() -> except imapobj.abort
When checking for the IMAP4.abort() exception, we need of course to
perform:

except imapobj.abort:

and not

except imapobj.abort():

Thanks to Johannes Stezenbach <js@sig21.net> for pointing to the glitch.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-27 14:08:20 +02:00
Sebastian Spaeth
e145beb394 Fix "command CHECK illegal in state AUTH"
Dave identified a case where our new dropped connection handling did
not work out correctly: we use the retry_left variable to signify
success (0=success if no exception occured).

However, we were decrementing the variable AFTER all the exception
checks, so if there was one due to a dropped connection, it
could well be that we 1) did not raise an exception (because we want to
retry), and 2) then DECREMENTED retry_left, which indicated "all is
well, no need to retry".

The code then continued to check() the append, which failed with the
above message (because we obtained a new connection which had not even
selected the current folder and we were still in mode AUTH). The fix is
of course, to fix our logic: Decrement retry_left first, THEN decide
whether to raise() (retry_left==0) or retry (retry_left>0) which would
then correctly attempt another loop. I am sorry for this newbie type of
logic error. The retry count loop was too hastily slipped in, it seems.

Reported-by: Dave Abrahams <dave@boostpro.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-26 16:13:52 +02:00
Sebastian Spaeth
953c58a9c9 Robustify (&fix) error throwing on APPEND
If APPEND raises abort(), the (typ, dat) variables will not be set, so
we should not be using it for the  OfflineImapError Exception
string. Fixing and prettifying the string formatting a bit at the same
time.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-26 15:57:35 +02:00
Sebastian Spaeth
6b2ec956cf Apply nametrans to all Foldertypes
getvisiblename() was only defined on IMAP(derived) foldertypes, but we
want it on eg. Maildirs too, so we define it centrally in Folder.Base.py
rather than only in folder.IMAP.py.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 14:23:08 +02:00
Sebastian Spaeth
0d3303ec12 Remove visiblename as parameter to IMAPFolder creation
IMAPFolder has the repository and foldername values so it can get the
transposed (aka visiblename) of a folder itself just fine. There is no
need to pass it in as an separate parameter.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:29 +02:00
Sebastian Spaeth
ee75e0921f Remove 'config' as parameter from BaseFolder & derivatives
It is possible to get the config parameter from the Repository() which is
set in BaseFolder, so we set self.config there and remove the various
methods and 'config' parameters that are superfluous.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:12 +02:00
Sebastian Spaeth
410e2d35e9 Set accountname in BaseFolder, and don't pass it in initialization
We passed in the accountname to all derivatives of BaseFolder, such as
IMAPFolder(...,repository,...,accountname), although it is perfectly
possible to get the accountname from the Repository(). So remove this
unneeded parameter. Each backend had to define getaccountname() (although
the function is hardly used and most accessed .accountname directly).

On the other hand BaseFolder was using getaccountname but it never defined
the function. So make the sane thing, remove all definitions from backends
and define accountname() once in Basefolder. It was made a property and not
just a (public) attribute, so it will show up in our developer
documentation as public API.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:12 +02:00
Sebastian Spaeth
c93cd9bb1a BaseFolder(): Save name and repository
As all Folders share these parameters, we can safely handle them in
BaseFolder. This makes sense, as BaseFolder has a getname() function
that returns self.name but nothing actually set self.name.

It also saves a few lines of code.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:12 +02:00
Sebastian Spaeth
7941ea7e7d folder.IMAP: Make use of the new connection discarding
In getmessage() we were releaseing a connection when we detected a
dropped connection, but it turns out that this was not enough, we need
to explicitely discard it when we detect a dropped one. So add the
drop_conn=True parameter that was recently introduced to force the
discarding of the dead conection.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:53:00 +02:00
Sebastian Spaeth
fe4f385e2c folder.IMAP: Improve dropped connection handling in quickchanged()
The quickchanged() function was not handling dropped connections yet. If
IMAP4.select() throws a FOLDER_RETRY error, we will now discard the
connection, reconnect and retry.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:53:00 +02:00
Sebastian Spaeth
7c83d505f8 IMAP cachefolder: Fix returning None on select
We rely on the number of mails being returned by the imapobj.select()
call, however that only happens if we "force" a real select() to occur.
Pass in the force parameter that I dropped earlier (we did not make use
of the return value when I dropped it, that is how it slipped through).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-13 18:31:28 +02:00
Sebastian Spaeth
24db42916c IMAP savemessage(): Don't loop indefinitely on failure
We were retrying indefinitely on imapobj.abort() (as that is what
imaplib2 suggests), but if the failure occurs repeatedly, we'll never
quit this loop. So implement a counter that errs out after unsuccessful
retries.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-13 18:29:22 +02:00
Sebastian Spaeth
5cbec30b3e Sanity check for maxage setting
If maxage is set too large, we would even SEARCH for negative
years. With devastating results. So implement some sanity check and err
out in case the year does not make sense.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-07 19:32:43 +02:00
Sebastian Spaeth
135f8c45cf Simplify constructing the SEARCH date
We can use Imaplib's monthnames and shorten the construction of the date
by using them rather than hardcoding them again.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-07 19:32:08 +02:00
Sebastian Spaeth
1b99c019e5 Fix handling the search results
Results are delivered in a 1-element list, and somehow I managed to drop
a [0] in the previous patches. We need to look at the element of course,
or our string splitting will fail horribly. Sorry this somehow slipped
through.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-07 19:31:37 +02:00
Sebastian Spaeth
26721c60d4 Don't cache empty IMAP folders
If a folder is empty, most servers will return EXISTS 0 and imaplib2
passes back ['0'] as return value to a select(). It returns [None] if
no EXISTS response was given by the server at all.

Attempting to fetch the UIDs of 0 emails which leads to
various error messages (One server responds with "NO No matching
messages", Gmail seems to say "BAD Bad message sequence 1:*" for some
(although it is working fine for me with Gmail, so it might behave
different for different people).

In case we get an None or 0 back, we simply stop caching messages as the
folder is empty. This should fix the various error reports that have
popped up.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-07 19:13:13 +02:00
Nicolas Sebrecht
67863017e2 Merge branch 'ss/better-error-throwing-and-id-sequence' into next
Conflicts:
	offlineimap/folder/IMAP.py
	offlineimap/imaputil.py

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:15:05 +02:00
Sebastian Spaeth
4c558c1b69 Error proof IMAP.APPEND against dropped connections
Make sure that when a connection is dropped during append, we really
discard the broken connection and get a new one, retrying. We retry
indefinitely on the specific abort() Exception, as this is what imaplib2
suggests us to do.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
fb8017991c Rework undocumented listjoin to create UID sequences
This function was badly named and completely undocumented. Rework it to
avoid copying the full UID list using an iterator. Make it possible to
hand it a list of UIDs as strings rather than implicitely relying on the
fact that they are numeric already. Document the code.

The behavior off the function itself remained otherwise unchanged.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
3302940382 Proper error handling for SEARCH and FETCH failures from the server
SEARCH and FETCH were never checking that the IMAP server actually
returned OK. Throw OfflineImapErrors at severity FOLDER in case one of
them fails.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
f755c8b423 Use range 1:* if we want to examine all messages in a folder
Some code cleanup. If we want to examine all messages of a folder, don't
try to find out how many there are and request a long list of all of them,
but simply request 1:*. This obliviates us from the need to force a select
even if we already had the folder selected and it requires us to send a
few less bytes over the wire.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
8532a69458 Clean up the maxsize maxage code
Do away with the wrapping of this code in a try...except KeyError, as
this code cannot conceivably throw a KeyError. Even if it could, it
should be documented why we should simply return() in this case.

Shorten some of the variable names and minor code cleanup while taking
the git blame anyway.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
06bfff7c04 Coalesce SEARCH uid list into sequence set
Rather than passing in huge lists of continuous numbers which eventually
overflow the maximum command line length, we coalesce number ranges
before passing the UID sequence to SEARCH. This should do away with the
error that has been reported with busy mailing lists and 'maxage'.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Nicolas Sebrecht
8c730cf509 Merge branch 'ss/folder.imap-dont-import-copy' into next
Conflicts:
	offlineimap/folder/IMAP.py

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:57:02 +02:00
Sebastian Spaeth
63b9dcd896 folder.IMAP: Don't import copy
It is not needed. list(ALIST) will create a new copy of the list just
fine.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:53:44 +02:00
Sebastian Spaeth
2cf6155282 Error proof IMAP.APPEND against dropped connections
Make sure that when a connection is dropped during append, we really
discard the broken connection and get a new one, retrying. We retry
indefinitely on the specific abort() Exception, as this is what imaplib2
suggests us to do.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:51:49 +02:00
Sebastian Spaeth
0906d0db70 IMAP.cachemessagelist(): Protect against empty folders
When invoked with FETCH 1:* (UID), imaplib returns [None] for empty
folders. We need to protect against this case and simply 'continue' here.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:07:54 +02:00
Sebastian Spaeth
dc103ab9ce Protect IMAP.getmessage() against dropped connections
If a connection is dropped for some reason while fetching a message, the
imapobj.uid command throws an imapbj.abort() Exception which means we are
supposed to retry. Implement a fail loop that drops the connection, gets a
new one and attempts the command another time.

Remove obsolete comment that we need to catch nonexisting messages. We do
now.

GMail seems to drop connections left and right. This patch is a response to
the reported mail "4E5F8D8C.1020005@gmail.com" by zeek
<ezekiel.das@gmail.com>.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:00:48 +02:00
Sebastian Spaeth
0a25408199 Rework undocumented listjoin to create UID sequences
This function was badly named and completely undocumented. Rework it to
avoid copying the full UID list using an iterator. Make it possible to
hand it a list of UIDs as strings rather than implicitely relying on the
fact that they are numeric already. Document the code.

The behavior off the function itself remained otherwise unchanged.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:55:28 +02:00
Sebastian Spaeth
466ded04d9 Make flags a set rather than a list
As this is essentially what it is, a set of values. This allows as
to do set arithmetics to see, e.g. the intersection of 2 flag sets
rather than clunkily having to do:

for flag in newflags:
  if flag not in oldflags:
    oldflags.append(flag)

Also some more code documenting.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 18:56:10 +02:00
Vladimir Marek
b6ac1aecb1 Another way of locating UID of just saved message
It works by fetching all headers of new messages from IMAP server and
searching for our X-OfflineIMAP marker by using regular expression.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-17 20:58:41 +02:00
Sebastian Spaeth
fe388400c4 Better error message when FETCH fails.
We were not including the full server reply into our error message. Fix
that so we get better error logs.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:16:29 +02:00
Sebastian Spaeth
194aa1db3c Catch 'BAD' replies on append()ing a message
append() raises an Exception, in case the IMAP server replies with 'BAD'
(but not when it responds with 'NO') but we were not catching that. Do
catch the situation and also raise an OfflineImapError at MESSAGE
severity, so that we can continue with the next message.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:05:08 +02:00
Sebastian Spaeth
f6b9c68333 IMAP: Don't use assert() in folder.savemessage()
We simply assert()ed that APPENDing a message returned OK, but in some
cases (e.g. Google chat messages) APPEND might return BAD or NO too. We
should be throwing an OfflineImapError here at MESSAGE level, so that we
can continue to sync all other messages, and still give the user some
details on what went wrong at the end of the sync run.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-11 19:14:28 +02:00
Vladimir Marek
e58399ac0b Fixed typo
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-27 18:54:53 +02:00
Vladimir Marek
d5cbdc4c0e Handle when UID can't be found on saved messages
Message was stored to dstfolder, but we can't find it's UID. This means we can't
link current message to the one created in IMAP. So we just delete local message
and on next run we'll sync it back. Also fixed imap.savemessage description.

This was broken by e20d8b9679.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-27 18:54:53 +02:00
Vladimir Marek
86e9c7442b Include message header at better place
It's not enough to place header after first newline, since this might break
multiline rfc0822 folded long header lines. Those are difined as CRLF followed
by white space. Instead we'll search for two successive CRLF sequences which
mark end of mail headers and place our header just before that.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-27 18:54:52 +02:00
Haojun Bao
9d95d7bc62 folder/IMAP: fix typo with maxsize and maxage.
Signed-off-by: Bao Haojun <baohaojun@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-27 18:26:43 +02:00
Sebastian Spaeth
856982a4e6 Throw OfflineImapError when we try to request an inexistant message
During a sync run, someone might remove or move IMAP messages. As we
only cache the list of UIDs in the beginning, we might be requesting
UIDs that don't exist anymore. Protect folder.IMAP.getmessage() against
the response that we get when we ask for unknown UIDs.

Also, if the server responds with anything else than "OK", (eg. Gmail
seems to be saying frequently ['NO', 'Dave I can't let you do that now']
:-) so we should also be throwing OfflineImapErrors here rather than
AssertionErrors.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-15 22:17:04 +02:00
Sebastian Spaeth
e8b633b884 folder/IMAP: Remove buggy duplicate assignment
we do:
  for msgid in imapdata:
      maxmsgid = max(long(msgid), maxmsgid)
and then basically immediately:
   maxmsgid = long(imapdata[0])

throwing away the first assignment although the first method of
assigning is the correct one. The second had been forgotten to be
removed when we introduced the above iteration. This bug would fix a
regression with those broken ZIMBRA servers that send multiple EXISTS
replies.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:52:24 +02:00
Sebastian Spaeth
846070b240 Fix typos in months names
All months names are 3-letter abbreviated, but accidentally June and
July slipped through. Thanks to the heads up by
Philipp Kern <pkern@debian.org>.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:20:25 +02:00
Sebastian Spaeth
c70a621709 Limit msg body length in debug output
We were outputting full message bodies to the debug log (often stderr),
and then again (as they go over the imaplib2 wire, imaplib logs
everything too). Not only is quite a privacy issue when sending in debug
logs but it can also freeze a console for quite some time. Plus it
bloats debug logs A LOT.

Only output the first and last 100 bytes of each message body to the
debug log (we still get the full body from imaplib2 logging). This
limits privacy issues when handing the log to someone else, but usually
still contains all the interesting bits that we want to see in a log.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-09 20:58:46 +02:00
Sebastian Spaeth
b3a383d151 accounts: handle OfflineImapError severity FOLDER
Throw an OfflineImapError when SELECTing a folder is unsuccessful and
bail out with a FOLDER serverity. In accounts.py catch all
OfflineImapErrors and either just log the error and skip the folder or
bubble it up if it's severe.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 13:56:04 +02:00
Sebastian Spaeth
f4081985dc Prettify and use new uidexists() helper function
Make the folder classes use uidexists() more. Add some code
documentation while going through.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:50:22 +02:00
Nicolas Sebrecht
0bc1c9583e Merge branch 'ss/folder-cleanup-getmessage' into next
Conflicts:
	offlineimap/folder/IMAP.py
2011-04-27 22:52:17 +02:00
Sebastian Spaeth
1ff628bd5c folder/IMAP: cleanup getmessage()
Add some comments how the data structures actually look like.
Describe the function properly, and make sure we only hold on to the
data connection as quickly as possible.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 22:49:18 +02:00
Sebastian Spaeth
36eb37b47d IMAP: reduce quickchanged() checks
For each folder we were making a second IMAP request asking for the
latest UID and compared that with the highest UID in our
statusfolder. This catched the case that 1 mail has been deleted by
someone else and another one has arrived since we checked, so that the
total number of mails appears to not have changed.

We don't capture anymore this case in the quickchanged() case.

It improves my performance from 8 to about 7.5 seconds per check (with lots of
variation) and we would benefit even more in the IMAP<->IMAP case as we do one
additional IMAP lookup per folder on each side then.

Do cleanups on whitespaces while in this file.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-25 12:58:47 +02:00
Sebastian Spaeth
58220fd8e7 Replace calls to getmessagelist() to alternatives
getmessagelist() is slow for the mapped UID case, so replace some of its
occurences with calls that are optimized for this case, ie
getmessagecount() and uidexists().

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-and-tested-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 17:53:14 +02:00
David Favro
954655b7ec Fixed bug: wrong number of arguments to debug() [IMAP.py].
Signed-off-by: David Favro <offlineimap@meta-dynamic.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 16:49:21 +02:00
Sebastian Spaeth
09515f8f90 Increase compatability with Gmail
When uploading a new message to Gmail we need to find out the UID it
assigned it, but Gmail does not advertize the UIDPLUS extension (in all
cases) and it fails to find the email that we just uploaded when
searching for it. This prevented us effectively from uploading to
gmail.

See analysis in
http://lists.alioth.debian.org/pipermail/offlineimap-project/2011-March/001449.html
for details on what is going wrong.

This patch increases compatability with Gmail by checking for APPENDUID
responses to an APPEND action even if the server did not claim to
support it. This restores the capability to upload messages to the
*broken* Gmail IMAP implementation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-03-25 18:42:16 +01:00
Sebastian Spaeth
fd28c5a2d3 folder/IMAP: savemessage() should just save flags if uid already exists
As the LocalStatus and UIDMap backend already did: If the uid already
exists for savemessage(), only modify the flags and don't append a new
message.

We don't invoke savemessage() on messages that already exist in our sync
logic, so this has no change on our current behavior. But it makes
backends befave more consistent with each other.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 18:05:16 +01:00
Ethan Glasser-Camp
1bf4bee5e6 Update to match semantics of new imaplib2
The biggest change here is that imapobj.untagged_responses is no
longer a dictionary, but a list. To access it, I use the semi-private
_get_untagged_response method.

* offlineimap/folder/IMAP.py (IMAPFolder.quickchanged,
  IMAPFolder.cachemessagelist): imaplib2 now explicitly removes its
  EXISTS response on select(), so instead we use the return values from
  select() to get the number of messages.

* offlineimap/imapserver.py (UsefulIMAPMixIn.select): imaplib2 now
  stores untagged_responses for different mailboxes, which confuses us
  because it seems like our mailboxes are "still" in read-only mode when
  we just re-opened them.  Additionally, we have to return the value
  from imaplib2's select() so that the above thing works.

* offlineimap/imapserver.py (UsefulIMAPMixIn._mesg): imaplib2 now
  calls _mesg with the name of a thread, so we display this
  information in debug output. This requires a corresponding change to
  imaplibutil.new_mesg.

* offlineimap/imaplibutil.py: We override IMAP4_SSL.open, whose
  default arguments have changed, so update the default arguments. We
  also subclass imaplib.IMAP4 in a few different places, which now
  relies on having a read_fd file descriptor to poll on.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
f9413226b8 Import imaplib2 instead of imaplib
imaplib2 has slightly different semantics than standard imaplib, so
this patch will break the build, but I thought it was helpful to have it as
a separate commit.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Sebastian Spaeth
2946a1ea5d Convert rfc822 module to email module
The rfc822 module has been deprecated since python 2.3, and conversion to
the email module is straightforward, so let us do that. rfc822 is
completely gone in python3.

This also fixes a bug that led to offlineimap abortion (but that code path
is apparently usually not exercised so I did not notice:
rfc822|email.utils.parsedate return a tuple which has no named attributes,
but we were using them later in that function. So pass the tuple into a
struct_time() to get named attributes.

While reading the docs, I noticed that email.parsedate returns invalid
daylight savings information (is_dst attribute), and we are using it
anyway. Oh well, the imap server might think the mails are off by an hour
at worst.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-13 20:30:35 +01:00
Sebastian Spaeth
3b8e1f91cd Implement APPENDUID support
Rather than inserting our own home-grown header, everytime we save a
message to an IMAP server, we check if we suport the UIDPLUS extension
which provides us with an APPENDUID reply. Use that to find the new UID
if possible, but keep the old way if we don't have that extension.

If a folder is read-only, return the uid that we have passed in per API
description in folder.Base.py

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
96db608e4d Remove convoluted assert statements
The working horse of the savemessage() function, imaplib.append() was
hidden away in an assert statement. Pull the real functions out of the
asserts and simply assert on the return values. This looks less
convoluted and makes this easier to understand in my opinion.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
d22c762385 Factor out the date guessing/retrieving
savemessage was too long and complex. Factor out the date guessing part
of the function and put it into a function of its own. The logic of the
date guessing is the same, however, we do not use the
imaplib.Time2InternalDate() function as it is buggy
(http://bugs.python.org/issue11024) and returns localized patches. So we
create INTERNALDATE ourselves and pass it to append() as a string.

This commit fixes a bug that international users used to pass an invalid
date to the IMAP server, which the server will either ignore or complain
about.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
419f27418e Simplify & document savemessage_getnewheader
savemessage_getnewheader was an undocmented, cryptic and overengineered
function. It generates a new unique value that can be used as a mail
header to be inserted. For this it used LOTS of randomness sources: hash
of the mail content, hash of the folder name, hash of the repository
name, the current time, a random() value, and the offlineimap version string.
All we need is something random. So reduce this to hash of content
appended by a random integer. Sufficient and somewhat faster to calculate.

Rename the function to actually describe accurately what it does or
would you have guessed that savemessage_getnewheader() did nothing more
than returning ('X-OfflineIMAP', <randomstring> )? Rename to
generate_randomheader() to make it clearer what this is all about.

Also document the function, describing what it does, and what it returns.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:09 +01:00
Sebastian Spaeth
6398bf7795 Don't require the string module
There is no need for using the string module if all we want is to split
a string at the white space. All pythons since at least 2.4 can do that.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-02-21 13:20:21 +01:00
Nicolas Sebrecht
9afb432053 folder/IMAP.py: accord __version__ with import
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-27 19:45:14 +01:00
Nicolas Sebrecht
a9c63078a2 Merge branch 'master' into next 2011-01-20 20:12:02 +01:00
Knut Anders Hatlen
9ad1810e43 understand multiple EXISTS replies from servers like Zimbra
Lars Thalmann <web@larsthalmann.com> writes:

> It seems some servers (Zimbra?) respond to imap SELECT requests with
> multiple EXISTS lines:
>
> ? SELECT INBOX
> * 500 EXISTS
> * 0 RECENT
> * 1000 EXISTS
> * 0 RECENT
> * 1500 EXISTS
> * 0 RECENT
> * 1567 EXISTS
> * 0 RECENT

Signed-off-by: Knut Anders Hatlen <kahatlen@gmail.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-20 20:06:46 +01:00
Nicolas Sebrecht
ee8a00cce2 Merge branch 'ss/ui' into next
Conflicts:
	offlineimap/folder/IMAP.py

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-05 19:31:08 +01:00
Sebastian Spaeth
de618d84f3 Save ui as self.ui in folder/Base class
Rather than regetting the ui with UIBase.getglobalui() all the time, we get it once in the base class and let all derivative classes just make use of self.ui rather than refetching the ui all the time, this makes for a bit less code and shorter lines.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-05 19:24:00 +01:00
Sebastian Spaeth
0b5b38d298 Define version constants etc in __init__.py
Move central constant definitions into __init__.py.  This does away
with version.py which contained nothing else and __init__.py is where
things like __VERSION__ are usually defined.

This commit also changes code to use offlineimap.__version__ rather
than offlineimap.version.__version__ as was before. Cleaned up some
duplicate or unneeded imports while touching those, formatting import
statements per PEP8 (one import per row).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2010-12-22 19:45:01 +01:00
Johannes Stezenbach
573d7ed52a Change X-OfflineIMAP header format
Change the X-OfflineIMAP header to work around possible Exchange MAPI
table overflow problem described in
http://article.gmane.org/gmane.mail.imap.offlineimap.general/1699
(It is unknown whether this problem still exits in current
Exchange versions, but let's assume the worst.)

The X-OfflineIMAP header is neccessary with some IMAP servers to
reliably determine the UID of a new messages uploaded to the server
by using the "UID SEARCH HEADER name string" command.  Since this
command compares header name and value it is sufficient to have
a unique header value and a non-unique header name.

Note that a message can have more than one X-OfflineIMAP header if
the message was copied between IMAP folders multiple times.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2010-12-09 19:52:48 +01:00
John Goerzen
1148df7685 Merge branch 'maxage'
Conflicts:
	offlineimap/folder/IMAP.py
2009-08-16 09:29:00 -05:00
Mike Dawson
cde94e5047 Patch for maxage and maxsize options
Dear All,
Attached is the patch that I have developed to provide maxage and
maxsize options.  You can thus sync only the last x days of messages and
exclude large messages.  All details in the attached git file.

Regards,

-Mike

-- Attached file included as plaintext by Ecartis --
-- File: submit

From 04fead2b46a79675a5b29de6f2b4088b9c9448e5 Mon Sep 17 00:00:00 2001
From: mike <mike@mikelaptop.(none)>
Date: Sun, 16 Aug 2009 17:00:49 +0430
Subject: [PATCH] Patch to provide maxage and maxsize account options to exclude old/large messages

This is designed to make offlineimap even better for low bandwidth connections.

maxage allows you to specify a number of days and only messages within
that range will be considered by offlineimap for the sync.  This can be
useful if you would like to start using offlineimap with a large
existing account and do not want to import large archives of mail.

maxsize allows you to specify the maximum size of a message to consider so
that you can exclude messages with large attachments etc.

In both cases the cachemessagelist function of the folder was modified to ignore
messages that do not meet the criteria.  If the criteria are not specified
then the existing code will be executed the same as before.  If a message
does not meet the criteria it will be as though this message does not exist
- offlineimap will completely ignore it.  It will not have flags updated,
it will not be deleted, it will not be considered at all.

When operating against an IMAP repository a server side search function
is used.  This of course requires support for server side search.

I have tested this with either option, no options etc. against IMAP, Maildir
and Gmail.  I have run variations of this patch here for the last 3 weeks or
so syncing about 4 accounts normally.
2009-08-16 09:27:49 -05:00
John Goerzen
2433dc9f06 Merge branches 'master' and 'master' of ssh://jpgarch@complete.org/~jpgarch/git/offlineimap 2009-08-12 15:25:26 -05:00
John Goerzen
1f8024a70e [imaplib2 removal] Revert "Implementation of IMAP IDLE"
This reverts commit 3847d0ba9d.
2009-08-12 14:49:58 -05:00
John Goerzen
e66fd5f51a Don't request INTERNALDATE when we don't need it. Patch from Marc MERLIN. 2009-07-16 20:57:07 -05:00
James Bunton
3847d0ba9d Implementation of IMAP IDLE
- Use a newer version of imaplib
 - Hijack the keepalive process to send IDLE instead of NOOP
2009-02-10 11:27:48 +11:00
John Goerzen
6e6720d1b0 Update for 5.99.7 2008-03-04 08:32:29 -06:00
John Goerzen
3cb3a675b8 Revert "completed: * fixes behaviour when changing flags, without corresp. rights (s,d,w)"
This reverts commit 08f22be8ea.

Conflicts:

	offlineimap/folder/IMAP.py
2008-03-04 08:15:48 -06:00
John Goerzen
cabd8d1735 Revert "fix behaviour for delete/expunge, when lacking rights"
This reverts commit c305d63e00.

Zak Smith reported it continued to be a problem
2008-03-04 08:13:48 -06:00
John Goerzen
9a168c573e Revert "fixes behaviour when changing flags, without corresp. rights (s,d,w)"
This reverts commit 71a76d9a61.

Zak Smith reported a problem at:

      self.processmessagesflags('+', uidlist, flags)
  File "/usr/lib/python2.5/site-packages/offlineimap/folder/IMAP.py",
  line 372, in processmessagesflags
    myrights = imapobj.myrights(self.getfullname())[1][0].split()[1]
IndexError: list index out of range

Conflicts:

	offlineimap/folder/IMAP.py
2008-03-03 21:20:53 -06:00
John Goerzen
08a579657a Fix handling of servers that return UIDs in some FETCH responses
closes #22

from pistore in OfflineIMAP #22:

When an IMAP flag update is performed for multiple messages, some IMAP
servers (e.g. Exchange) return the UID attribute only for some of the
FETCH untagged responses, as shown in the following log:

21:19.04 > DCKF8 UID STORE 66050,50613,52164,40043,40055,25874 +FLAGS
(\Deleted)
21:19.36 < * 35 FETCH (FLAGS (\Seen \Deleted) UID 25874)
21:19.36 < * 321 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 322 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 560 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 581 FETCH (FLAGS (\Seen \Deleted) UID 52164)
21:19.62 < * 1022 FETCH (FLAGS (\Seen \Deleted))
21:19.62 < DCKF8 OK STORE completed.

Function IMAPFolder.processmessagesflags is able to manage the servers
which return the UID and the servers which do not return it, but is
not able to deal with the mixed behavior shown above.

The problem is that the fragment of function
IMAPFolder.processmessagesflags that handles the responses with UID
attribute uses variable flags to store the list of flags of the
message in the IMAP format ("flags = attributehashFLAGS?"), while the
fragment that handles the responses without UID expects variable
"flags" to contain the list of modified flags passed to the function
in Maildir format ("self.messagelist[uid]flags?.append(flag)").

As a consequence, the wrong list of flags is used for the messages
without UID, leading to the addition of "strange" flags to the Maildir
messages:

Syncing messages IMAP[INBOX] -> Maildir[.]
Adding flags   to 4 messages  on Maildir[.]
Adding flags e to 4 messages  on Maildir[.]
Adding flags d to 4 messages  on Maildir[.]
Adding flags ) to 4 messages  on Maildir[.]
Adding flags ( to 4 messages  on Maildir[.]
Adding flags l to 4 messages  on Maildir[.]
Adding flags n to 4 messages  on Maildir[.]
Adding flags t to 4 messages  on Maildir[.]
Adding flags \ to 4 messages  on Maildir[.]
Adding flags D to 4 messages  on Maildir[.]
Deleting flags T to 4 messages on Maildir[.]
Adding flags   to 4 messages  on LocalStatus[.]
Adding flags e to 4 messages  on LocalStatus[.]
Adding flags d to 4 messages  on LocalStatus[.]
Adding flags ) to 4 messages  on LocalStatus[.]
Adding flags ( to 4 messages  on LocalStatus[.]
Adding flags l to 4 messages  on LocalStatus[.]
Adding flags n to 4 messages  on LocalStatus[.]
Adding flags t to 4 messages  on LocalStatus[.]
Adding flags \ to 4 messages  on LocalStatus[.]
Adding flags D to 4 messages  on LocalStatus[.]
Deleting flags T to 4 messages on LocalStatus[.]

Fix: use a different variable to store IMAP flags when managing
messages corresponding to responses with UID attribute, e.g.:

*** IMAP.py.orig        Wed Aug 22 18:23:17 2007
--- IMAP.py     Wed Aug 22 18:22:38 2007
*************** class IMAPFolder(BaseFolder):
*** 340,348 ****
              if not ('UID' in attributehash and 'FLAGS' in
              attributehash):
                  # Compensate for servers that don't return a UID
                  attribute.
                  continue
!             flags = attributehash['FLAGS']
              uid = long(attributehash['UID'])
!             self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(flags)
              try:
                  needupdate.remove(uid)
              except ValueError:          # Let it slide if it's not
              in the list
--- 340,348 ----
              if not ('UID' in attributehash and 'FLAGS' in
              attributehash):
                  # Compensate for servers that don't return a UID
                  attribute.
                  continue
!             lflags = attributehash['FLAGS']
              uid = long(attributehash['UID'])
!             self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(lflags)
              try:
                  needupdate.remove(uid)
              except ValueError:          # Let it slide if it's not
              in the list

02/03/08 14:04:35 changed by js

    * attachment flags-fix.patch added.

Delete 02/03/08 14:05:24 changed by js 

Unfortunately I have to fetch some of my mail from an Exchange server
(Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1)
and I can confirm that the analysis of the problem is correct, and the
patch given here fixes the problem.

Looking at the code of the processmessagesflags() method I think it
generally is a bug that the "flags" parameter is reused as a local
variable, since the final "for uid in needupdate:" loop needs the
original value of "flags". This only worked by accident.

I'm attaching a unidiff version of the patch which applies cleanly
against Debian unstable's offlineimap 5.99.4.
2008-03-02 22:25:05 -06:00
Florian Friesdorf
08f22be8ea completed: * fixes behaviour when changing flags, without corresp. rights (s,d,w) 2007-07-19 17:52:29 +01:00
Florian Friesdorf
71a76d9a61 fixes behaviour when changing flags, without corresp. rights (s,d,w) 2007-07-12 05:08:47 +01:00
Florian Friesdorf
c305d63e00 fix behaviour for delete/expunge, when lacking rights
This patch maneuvers around python imaplib's mysterious read-only detection
algorithm and correctly calls the UI's deletetoreadonly(), when trying to
delete/expunge in a mailbox without having the necessary rights.
2007-07-12 04:44:11 +01:00
John Goerzen
3305d8cd4d Daniel Jacobowitz patches
fixes deb#433732

Date: Sun, 30 Sep 2007 13:54:56 -0400
From: Daniel Jacobowitz <drow@false.org>
To: offlineimap@complete.org
Subject: Assorted patches

Here's the result of a lazy Sunday hacking on offlineimap.  Sorry for
not breaking this into multiple patches.  They're mostly logically
independent so just ask if that would make a difference.
First, a new -q (quick) option.  The quick option means to only update
folders that seem to have had significant changes.  For Maildir, any
change to any message UID or flags is significant, because checking
the flags doesn't add a significant cost.  For IMAP, only a change to
the total number of messages or a change in the UID of the most recent
message is significant.  This should catch everything except for
flags changes.

The difference in bandwidth is astonishing: a quick sync takes 80K
instead of 5.3MB, and 28 seconds instead of 90.

There's a configuration variable that lets you say every tenth sync
should update flags, but let all the intervening ones be lighter.


Second, a fix to the UID validity problems many people have been
reporting with Courier.  As discussed in Debian bug #433732, I changed
the UID validity check to use SELECT unless the server complains that
the folder is read-only.  This avoids the Courier bug (see the Debian
log for more details).  This won't fix existing validity errors, you
need to remove the local status and validity files by hand and resync.


Third, some speedups in Maildir checking.  It's still pretty slow
due to a combination of poor performance in os.listdir (never reads
more than 4K of directory entries at a time) and some semaphore that
leads to lots of futex wake operations, but at least this saves
20% or so of the CPU time running offlineimap on a single folder:

Time with quick refresh and md5 in loop: 4.75s user 0.46s system 12%
cpu 41.751 total
Time with quick refresh and md5 out of loop: 4.38s user 0.50s system
14% cpu 34.799 total
Time using string compare to check folder: 4.11s user 0.47s system 13%
cpu 34.788 total


And fourth, some display fixes for Curses.Blinkenlights.  I made
warnings more visible, made the new quick sync message cyan, and
made all not explicitly colored messages grey.  That last one was
really bugging me.  Any time OfflineIMAP printed a warning in
this UI, it had even odds of coming out black on black!


Anyway, I hope these are useful.  I'm happy to revise them if you see
a problem.

-- 
Daniel Jacobowitz
CodeSourcery
2007-10-01 22:20:37 +01:00
John Goerzen
2323fdcdb3 Additional date validity check
patch from Mike Gerber

Two times today I have found my offlineimap to have died with this same
situation. It appears as if certain types of messages (both spam in my
situation), cause offlineimap to choke. When it does it cannot proceed.
This means that when I run offlineimap, it pulls in messages from some
folders, then it hits the folder with the bad message and dies, leaving
undownloaded mail on the server. The only fix to this problem is to find
the problem message on the server and remove it by hand. This isn't such
a huge deal for me, since I run the server, but other people have to
come to me to ask me to delete these messages, and until I do they
cannot download their email.

I have captured the output by running script during one of these
incidents, this has been attached. Additionally, I have also attach the 
problematic message.

The patch seems to work for me, might need some Python wizard and better 
testing, though.

fixes deb#396443
2007-08-01 02:25:05 +01:00
Florian Friesdorf
3e7899a9dc IMAP.py: fixed cannot concatenate 'str' and 'list' in assert
r[1] is a list. In case it contains more than one 'str' they are concatenated
using '. '. In processmessagesflags the join is tested, for savemessagesflags I
assume that it is also needed.
2007-07-07 04:51:02 +01:00
John Goerzen
abf9648fd8 Clean up imaplib imports 2007-07-04 18:53:48 +01:00
John Goerzen
1b9fa9ff9a Switch to imaplibutil Internaldate2Epoch 2007-07-04 18:51:57 +01:00
John Goerzen
9d7df0e21a Fix up date parsing to use message date if no rtime is available 2006-08-22 02:13:39 +01:00
John Goerzen
03488ba81b Sync INTERNALDATE <-> mtime
The attached patch adds syncing the INTERNALDATE of IMAP folders with
the mtime of messages in maildir folders.
I want this to happen, because I'm running a dovecot over the maildirs
synced by offlineimap, and that uses the mtime as the INTERNALDATE.
When using mutt to view messages I generally sort based on the received
date, which for IMAP folders is the INTERNALDATE.

Since this is the first real coding I've done in Python the patch may
need to be cleaned up some, but it's working pretty well for me.  I've
added new messages to each side, and the received date has been
preserved going both ways.
2006-08-22 02:09:36 +01:00
John Goerzen
39a18fef60 Update FSF address 2006-08-12 05:15:55 +01:00
Adam Spiers
5fc0e1ae42 helpful comments 2006-05-16 04:40:23 +01:00
Adam Spiers
b39845c488 stop UID FETCH 0 errors 2006-05-16 04:31:44 +01:00
Adam Spiers
2e52bcdafe failure to obtain uid indicated by savemessage_searchforheader return value <= 0, not ValueError 2006-05-16 04:30:48 +01:00
Adam Spiers
720511f3f1 add missing ) 2006-05-16 04:27:57 +01:00
Adam Spiers
f2d4c668d5 return 0 rather than raise exception to indicate that message was saved with unknown UID 2006-05-16 03:34:46 +01:00
Adam Spiers
f11a49f263 make savemessage_searchforheader more robust 2006-05-15 03:51:12 +01:00
John Goerzen
c0d031e760 Remove parens for SEARCH command
Patch from Adam Spiers

Reported that parens were causing trouble for Groupwise IMAP server
2006-05-01 20:40:55 +01:00
John Goerzen
1f25b5393b [324827] Fixed handling of invalid dates
Patch from Nikita V. Youshchenko

From: "Nikita V. Youshchenko" 
To: Debian Bug Tracking System
Subject: offlineimap: exception on mail with broken headers (+fix)
Date: Wed, 24 Aug 2005 13:41:08 +0400

Package: offlineimap
Version: 4.0.10
Severity: normal
Tags: patch

Recently I've got an exception (see below) while using offlineimap.
Exception was probably caused by invalid Date header of (likely spam)
message:
Date: Sat, 20 Aug 2005 4294967295:43:18 -0700
2005-08-24 19:01:42 +01:00
John Goerzen
8137e53c14 Added code to limit command line length for very large mailboxes
Keywords: 


(jgoerzen@complete.org--projects/offlineimap--head--1.0--patch-4)
2004-11-16 23:41:09 +01:00
John Goerzen
440ee1708d Attempt to fix problem with getting back the wrong UID from APPENDUID
Keywords: 


(jgoerzen@complete.org--projects/offlineimap--head--1.0--patch-1)
2004-10-18 20:00:29 +01:00
John Goerzen
d839be3c61 Step 2 of SVN to arch tree conversion 2005-04-16 20:33:35 +01:00