Commit Graph

1648 Commits

Author SHA1 Message Date
Sebastian Spaeth
06bfff7c04 Coalesce SEARCH uid list into sequence set
Rather than passing in huge lists of continuous numbers which eventually
overflow the maximum command line length, we coalesce number ranges
before passing the UID sequence to SEARCH. This should do away with the
error that has been reported with busy mailing lists and 'maxage'.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Nicolas Sebrecht
a21bb949af Merge branch 'ss/improve-collapsing' into next 2011-09-06 19:59:28 +02:00
Nicolas Sebrecht
8c730cf509 Merge branch 'ss/folder.imap-dont-import-copy' into next
Conflicts:
	offlineimap/folder/IMAP.py

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:57:02 +02:00
Sebastian Spaeth
63b9dcd896 folder.IMAP: Don't import copy
It is not needed. list(ALIST) will create a new copy of the list just
fine.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:53:44 +02:00
Sebastian Spaeth
2cf6155282 Error proof IMAP.APPEND against dropped connections
Make sure that when a connection is dropped during append, we really
discard the broken connection and get a new one, retrying. We retry
indefinitely on the specific abort() Exception, as this is what imaplib2
suggests us to do.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:51:49 +02:00
Sebastian Spaeth
9c678c9d6b Allow Imapserver.releaseconnection() to drop a connection
If a connection is broken, we want to have it really dropped and not be
reused. So far, we are checking the .Terminate attribute for this, but
according to the imaplib2 author, it is only set on normal shutdown and
it is an undocumented attribute whose meaning could change any time.

This patch introduces the parameter drop_conn which allows to tell
releaseconnection() that we really want to connection being dropped from
the pool of available connections and properly destroy it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:51:49 +02:00
Sebastian Spaeth
0906d0db70 IMAP.cachemessagelist(): Protect against empty folders
When invoked with FETCH 1:* (UID), imaplib returns [None] for empty
folders. We need to protect against this case and simply 'continue' here.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:07:54 +02:00
Nicolas Sebrecht
ff9b941f42 Merge branch 'ss/protect-select-for-idle' into next 2011-09-06 19:01:11 +02:00
Sebastian Spaeth
dc103ab9ce Protect IMAP.getmessage() against dropped connections
If a connection is dropped for some reason while fetching a message, the
imapobj.uid command throws an imapbj.abort() Exception which means we are
supposed to retry. Implement a fail loop that drops the connection, gets a
new one and attempts the command another time.

Remove obsolete comment that we need to catch nonexisting messages. We do
now.

GMail seems to drop connections left and right. This patch is a response to
the reported mail "4E5F8D8C.1020005@gmail.com" by zeek
<ezekiel.das@gmail.com>.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:00:48 +02:00
Sebastian Spaeth
f1555ec893 Catch terminated connections on the IDLE SELECT operation
Handle the case gracefully where a server has closed an IMAP connection
that we want to use for IDLEing. Simply have it dropped and get a new one
in this case. THis should get rid of the errors reported by John Wiegley
in mail id:"m2sjohd16t.fsf@gmail.com".

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:00:21 +02:00
Sebastian Spaeth
61fffd5c5f imaplibutil: Have SELECT throw an OfflineImapError when the connection died
Imapserver.acquireconnection will succeed even whent the server connection
has been terminated and the first IMAP operation will throw an exception.

Often this is the folder SELECT operation (e.g. after an idle timeout), as
has been reported by John Wiegley. Catch this case and throw an
OfflineImapError with severity FOLDER_RETRY to notify consumers that they
are supposed to retry.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:00:21 +02:00
Sebastian Spaeth
7bc45507cb Introduce new error level FOLDER_RETRY
In cases where processing a folder failed, but a retry might well succeed
e.g. when the server connection had simply timed out and was disconnected,
we can throw a FOLDER_RETRY (which is less severe than FOLDER).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:00:21 +02:00
Sebastian Spaeth
7aa4c49ba4 Improve documentation on folderincludes
Just minor wording improvements.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-03 17:17:34 +02:00
Sebastian Spaeth
fe1391084b Improve repo.Base.py:syncfoldersto parameter/var names
Name parameter that hands us a Status Repository 'status_repo' and not
'copyfolders' as was before:

a) make it clear that we pass in a repository and not folder
instances. That was very confusing before.
b) We were always only using one 'copyfolders' item anyway, so let us
not make it a list.

Go through the list and make the variable nameing consistent:
dst_repo rather than dest (is it a folder?) to match the status_repo
naming scheme.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-03 17:16:42 +02:00
Sebastian Spaeth
54d3307499 imaputil: make uid_sequence sort all items to improve collapsing
To preserve previous behavior and save a few CPU cycles, we were not sorting UID
lists and only collapsed them if they were alreay sorted.  Vincent pointed out
that this is not always the case and unsorted lists lead to non-optimally
collapsing.

Force lists to numeric types and sort them before collapsing.

Reported-by: Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-02 22:02:51 +02:00
Sebastian Spaeth
f0869e5c0f Fix sqlite upgrade code to use correct API call
We were using self.getfolderbasename(self.name) but the API is simply
getfolderbasename(). Fix this glitch.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-02 21:59:35 +02:00
Sebastian Spaeth
8cce178fb8 Improve developer documentation
Well, this patch... improves it. :)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-30 22:19:16 +02:00
Sebastian Spaeth
4db5913492 Use SafeConfigParser for the configuration
SafeConfigParser is very similar to the currently used ConfigParser but
it supports interpolation. This means values can contain format strings
which refer to other values in the same section, or values in a special
DEFAULT section. For example:

[My Section]
foodir: %(dir)s/whatever
dir=frob

would resolve the %(dir)s to the value of dir (frob in this case). All reference expansions are done on demand.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-30 22:16:35 +02:00
Nicolas Sebrecht
f10e3a58fc MANUAL: fix typo about variable declaration in configuration file
I included a wrong change in commit dcfdf2ade7 writen by Sebastian.

As he said:

	> I think the first colon should be an equals sign...  :-)

	Actually no, this was on purpose.

	http://docs.python.org/library/configparser.html:

	[My Section]
	foodir: %(dir)s/whatever
	dir=frob
	long: this value continues
   	in the next line

Reported-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-30 22:05:44 +02:00
Sebastian Spaeth
dcfdf2ade7 MANUAL.rst: folderfilter->nametrans typo
We really wanted nametrans here, also simplify the lambda per
Dan Christensen's suggestion.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-29 21:42:43 +02:00
Sebastian Spaeth
3333723ce7 Improve our MANUAL
Make it contain real use cases and more explanations. We probably need
to add more of the FAQ entries to the MANUAL and point to relevant FAQ
entries from the manual.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-26 21:07:55 +02:00
Sebastian Spaeth
654e3ab9dc De-EXPERIMENTALIZE the sqlite backend and IDLE folders
Both have been in a stable release and we have never gotten negative
feedback about them. I have been using sqlite exclusively for some time
and people seem actively to use the IDLE folders without major problems.

This patch removes the scary: "THIS IS EXPERIMENTAL" warning, that I
know has kept some people from using it. Do note that the plaintext
backend is still the default even with this patch.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-26 21:07:09 +02:00
Sebastian Spaeth
1fc64f4c19 Improve folderfilters documentation
The text was confusing and not very helpful. Do away with the bad
folderfilters examples and describe a bit more what they are for.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-26 21:06:10 +02:00
Sebastian Spaeth
dcc50efc4f Don't call set().sort()
as that method doesn't exist on sets. Rather call the inbuilt
sorted(flags). This fixes Exceptions being thrown when using the sqlite
backend.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:59:48 +02:00
Sebastian Spaeth
a6480d4959 Fix string formatting
We were omitting an '%' where we needed it. Also include the traceback
information where it belongs in the new ui.error infrastructure.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:58:55 +02:00
Sebastian Spaeth
2b9b6be6be Fix standard port for SSL/TLS
443 is of course the https and not the IMAPS standard port. Fix.

Thanks to Daniel Shahaf <d.s@daniel.shahaf.name> for the heads up.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:57:38 +02:00
Vladimir Marek
9ecac29aaa Maildir relative paths change was not complete
Commit e023f190b0 changed the storing of
file paths in the messagelist variable to be relative paths, but we were
using the full absolute path anyway as we missed one spot.

Adapt this and construct the full file path in the one place where we
need it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:56:24 +02:00
Sebastian Spaeth
6295d64d54 Shorten list of messages to be deleted in UI output
Rather than output the full list of messages, coalesce it into number
ranges wherever possible.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:55:28 +02:00
Sebastian Spaeth
0a25408199 Rework undocumented listjoin to create UID sequences
This function was badly named and completely undocumented. Rework it to
avoid copying the full UID list using an iterator. Make it possible to
hand it a list of UIDs as strings rather than implicitely relying on the
fact that they are numeric already. Document the code.

The behavior off the function itself remained otherwise unchanged.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:55:28 +02:00
Vladimir Marek
8de326b29f Catch correct type of exception
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 19:05:16 +02:00
Vladimir Marek
df62bb61a5 Create exception when file rename fails
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 19:02:47 +02:00
Sebastian Spaeth
466ded04d9 Make flags a set rather than a list
As this is essentially what it is, a set of values. This allows as
to do set arithmetics to see, e.g. the intersection of 2 flag sets
rather than clunkily having to do:

for flag in newflags:
  if flag not in oldflags:
    oldflags.append(flag)

Also some more code documenting.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 18:56:10 +02:00
Sebastian Spaeth
373e7cdbc1 Allow empty foldernames
Empty foldernames (as they could be created through nametrans) were
failing as the uidvalidity and status files names as determined by
folder/Base.py:getfolderbasename() lead to invalid file names ''.

Fix this by handling empty file names and translating them to '.' which
leads to the special file name 'dot'. (this special value existed before
and was not invented by this patch)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 18:37:00 +02:00
Sebastian Spaeth
4f1cd05fdc Fix copyingmessage() in Machine UI
I changed the API to pass in a folder rather than a list of folders, but
used getnicename() on the wrong object. It is not used on the folder but
on the ui object. Fix this and give the variable somewhat better names.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 18:25:04 +02:00
Vladimir Marek
c2fc81dd3d Make syncfoldersto to accept a single folder only
It is just historic relict

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-17 21:02:21 +02:00
Vladimir Marek
b6ac1aecb1 Another way of locating UID of just saved message
It works by fetching all headers of new messages from IMAP server and
searching for our X-OfflineIMAP marker by using regular expression.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-17 20:58:41 +02:00
Sebastian Spaeth
6f9b171ffd Don't pass a list to ui.copyingmessage()
We only copy to a single folder anyway, so clean up the code to only
pass in a single folder.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-17 20:41:32 +02:00
Vladimir Marek
38b1d7b085 Replaced tabs with spaces to unify python sources
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-16 22:33:19 +02:00
Sebastian Spaeth
8d858a0b78 Disable certificate verification if there is no certificate
imaplib2 always attempts to verify a certificate if a verification
callback function is passed in, even the certificate is None
specified. Disable the verification excplictly by setting the
verification function to None in that case.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-16 22:22:47 +02:00
Nicolas Sebrecht
82bf23ee57 Merge branch 'ss/import-errno' into next
Conflicts:
	offlineimap/imapserver.py
2011-08-15 14:46:52 +02:00
Sebastian Spaeth
4a0d17b566 Import errno
we need errno.CONNREFUSED, but through some merging mishaps(?) the part
that actually imported errno was missing. Import the errno module.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 13:58:44 +02:00
Nicolas Sebrecht
2749374a93 Merge branch 'ss/imaplib2-2.28' into next
Conflicts:
	Changelog.draft.rst
2011-08-15 13:57:37 +02:00
Sebastian Spaeth
3a91e296f0 Adapt the code to work with the new imaplib2
imaplib renamed self.sslobj to self.sock and our overriden open()
functions were failing for that reason when updating imaplib2 to
v2.28. It turns out that all of our custom initializations are being
done by stock imaplib2 now anyway, so there is no need to override them
anymore. This lets us simplify the code we have to worry about.

Move the verifycert() function to the imapserver.py file, it is now a
callback function that is being handed to imaplib from there, so it
makes sense to also define it in our imapserver function...
(this also lets us easily make use of the verifycert function in the
starttls case in the future)

TODO: we need to examine if and why we still need to override the
select() function, it is the only reason why we still wrap the IMAP4
classes.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:21:38 +02:00
Sebastian Spaeth
1c0c19ad86 imaplib2: bump to upstream version 2.28 (rev 8228a0f)
Imaplib2 2.28 can deal with ID sequences, such as 1:*, so we need to
bump upstream in order to make use of these features.

Note that this revision will not run correctly as it requires
adaptations to our code, which happens in the next commit.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:21:38 +02:00
Sebastian Spaeth
fe388400c4 Better error message when FETCH fails.
We were not including the full server reply into our error message. Fix
that so we get better error logs.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:16:29 +02:00
Sebastian Spaeth
9f23fea74e Fix error handling in folder.Base.copymessageto()
This is a bug fix on several levels. 1) We were lacking the import of
OfflineImapError. 2) OfflineImap.ERROR.MESSAGE was misspelled as ERROR.Message.
3) COntinuing with the next message only worked in single-thread mode
(using debug) and not in multi-thread mode. The reason is that we were
invoking a new thread and catching Exceptions in the main thread. But
python immediately aborts if an Exception bubbles up to a thread start.
This was fixed by catching exceptions directly in copymessageto() which
is the new thread, rather than catching them in the main thread.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:16:28 +02:00
Sebastian Spaeth
634b6cd49e Don't make Blinkenlight statuschar configurable
Allowing to specify the char to use in the BLinkenlights is a bit over
the top and bloats our default offlineimap.conf. The dot is just fine,
so let us settle for it and cut the example config file by an unneeded
section.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:10:20 +02:00
Sebastian Spaeth
194aa1db3c Catch 'BAD' replies on append()ing a message
append() raises an Exception, in case the IMAP server replies with 'BAD'
(but not when it responds with 'NO') but we were not catching that. Do
catch the situation and also raise an OfflineImapError at MESSAGE
severity, so that we can continue with the next message.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:05:08 +02:00
Nicolas Sebrecht
e6dee14a84 Merge branch 'vm/doc-handle-UID' into next 2011-08-14 17:29:14 +02:00
Nicolas Sebrecht
b896e6f13b Merge branch 'ss/remove-processmessagesflags-function' into next 2011-08-14 17:23:49 +02:00