This eases testing of option values inside the code. This instance
is implemented as the read-only copy of the obtained 'options' object,
so callers won't be able to modify its contents.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
The problem lies in the fact that offlineimap.folder.Base's method
syncmessagesto_copy() uses threaded code everytime it is suggested by
the derived class's suggeststhreads() (currently, only IMAP does this
suggestion), but offlineimap/init.py will not spawn the
exitnotifymonitorloop() from offlineimap.threadutil.
The root cause is that ExitNotifyThread-derived threads need
offlineimap.threadutil's exitnotifymonitorloop() to be running the
cleaner for the exitthreads Queue(), because it fills the queue via
the run() method from this class: it wants to put() itself to the
Queue on exit, so when no exitnotifymonitorloop() is running, the
queue will fill up. And if this thread is an instance of
InstanceLimitedThread that hits the limit on the number of threads,
then it will hold the instancelimitedsems[] semaphore will prevent
other InstanceLimitedThread()s of the same name to pass its start()
method.
The fix is to avoid using threaded code if we're running
single-threaded.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Obtained-from: X-Ryl669 <boite.pour.spam@gmail.com>
Fix imapfolder.getmessageinternaldate misparsing the Date:
header from emails due to a bug or surprising behaviour by
email.utils.parsedate. This is because email.utils.parsedate's
return value contains the unadjusted hour value from the string
parsed but does not include information about the time zone in
which it is specified. For example (Python 2.7.3):
$ python -c "import email.utils;
print email.utils.parsedate('Mon, 20 Nov 1995 19:12:08 -0500')"
(1995, 11, 20, 19, 12, 8, 0, 1, -1)
(the -1 is the isdst field); the -0500 time zone is completely
ignored, so e.g. the same input with time "19:12:08 +0300" has
the same result. When passed to time.struct_time as allowed per
the parsedate documentation, this time is interpreted in GMT and
thus deviates from the correct value by the timezone offset
(in this example, -5 hours).
I consider this a bug in email.utils.parsedate: In my opinion,
since the return value of the parsetime doesn't include a timezone,
it should be expressed in terms of UTC rather than in terms of the
time zone from the Date header; the existence of
email.utils.parsedate_tz, to which I've switched, indicates that
maybe the authors were aware of this problem.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than later when retrieving them by IMAP, we determine the
"sync_this" value of a folder on Folder() initialization.
This also fixes a bug where folder structure was not propagated when
a folder was filtered out but included in the folderincludes list.
Patch by Dave, split and modified slightly by Sebastian
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
While looking at the code to investigate if an why we sometimes don't
seem to honor the write lock, I made it use the more modern "with lock:"
pattern.
Still have not found out how we could ever be using 2 instances of the
LocalStatusFolder for the same folder.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
As Gmail was only announcing the presence of the UIDPLUS extension
after we logged in, and we were then only getting server
capabilities before, a hack was introduced that checked the
existence of an APPENDUID reply, even if the server did not claim
to support it.
However, John Wiegley reports problems, where the APPENDUID would
be None, and we attempt to go this path (it seems that imaplib2
returns [None] if there is no such reply, so our test here for "!="
might fail. Given that this is an undocumented imaplib2 function
anyway, and we do fetch gmail capabilities after authentication,
this hack should no longer be necessary.
We had problems there earlier, where imapobj.response() would
return [None] although we had received a APPENDUID response from
the server, this might need more debugging and testing.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reported by sharat87 in https://github.com/spaetz/offlineimap/pull/38,
he would often get an unhandled Exception when trying to
releaseconnection() a connection that was not in the pool of
connections.
The reason this could happen is that when folder.IMAP.quickchanged()
raises an Exception in select(), we would release the connection in the
"except" handling, and than release the same connection in the "finally"
clause, which led to the error. The right thing is to only release the
connection once, of course.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This reverts commit 4d47f7bf3c.
This is one of two candidates for introducing the instabilities that
John Wiegley observed. We need to reintroduce with careful testing only.
The original patch has been mostly reverted.
This reverts commit 47390e03d6.
It is one of two potential candidates for the APPENDUID
regression that John Wiegley reported. We need to examine this
carefully before reintroducing this patch.
Resolved Changelog.draft.rst conflict.
Prevent savemessage(), and savemessageflags() to occur in dryrun mode in
all backends. Still need to protect against deletemessage().
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Bail out with a better Exception and error text. The whole mapped
UID situation needs to be improved though.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If we cannot identify the new UID after a sendmessage(), log a better error
message, including the server response for better debugging.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This allows to compare folders directly with strings. It also allows
constructs such as "if 'moo' in repo.getfolders()".
See the code documentation for the exact behavior (it basically is equal if
it's the same instance *or* a string matching the untranslated folder name.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
There is no need to cast 0 to 'long' even if we want to compare it to long
numbers in modern pythons.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This won't work in python3 anymore, so just use sorted() when needed.
In one case, we could remove the sort() completely as were were sanity checking
one line above, that we only having one UID as response which makes sorting
unneeded.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
'set' is builtin since python2.6, so remove the imports. Also 'ssl' exists
since 2.6 and has everything we need, so no need for conditional import
tests here anymore.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
To have the code work in python3, we need to convert all occurences of
raise Exception, "text" to be proper functions. This style also adheres to PEP8.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
It can lead to potential dataloss (see recent commit log where I added a
scary warning about it to offlineimap.conf).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Most servers support the UIDPLUS extension, and we don't have to search
headers after each uploaded message. There is no need to CHECK the imap
server after each message when there is no need to search headers.
I have not measured the performance impact on real world servers, but
this lets us do less unneeded work in the common case.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We were not cleaning out possibly existing APPENDUID messages before
APPENDing a new message. In case an old message were still hanging
around, this could *possibly* lead to retrieving and old UID. Things
should have been fine, but we do want to play safe here.
Also, make use of the "official" imaplib2 .response() command rather
than the internal _get_untagged_response() function.
Remove the hack that we would be looking for APPENDUID responses even if
the server claimed not to support the UIDPLUS ext. We now poll server
CAPABILITIES after login, and Gmail does provide us with the UIDPLUS
capability after login.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.
1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
from the imaplib2 result stack, so subsequent calls would return "None".
As various functions can invoke this, this led to some errors that we
avoid by caching the current UIDVALIDITY value in the Folder instance.
There are more simplifications and improvements to be made.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When deleting many (eg 2000) mails using the SQLITE backend, this takes
a long time durig which OfflineImap can not be aborted via
CTRL-C. Thinking it had frozen permanently, I killed it hard, leaving a
corrupted db journal (which leads to awkwards complaints by OLI on
subsequent starts!). That shows that delete performance is critical and
needs improvement.
We were iterating through the list of messages to delete and deleted
them one-by-one execute()'ing a new SQL Query for each message. This
patch improves the situation by allowing us to use executemany(), which
is -despite still being one SQL query per message- much faster. This is
because rather than performing a commit() after each mail, we now do
only one commit() after all mails have been deleted.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Recently the internal function use of imaplib2's _get_untagged_response()
was switched to use the public documented .response() function (which
should return the same data). However within a few fays we received reports
that both uses of a) the UIDVALIDITY fetching and b) the APPENDUID fetching
returned [None] as data although the IMAP log definitely shows that data
was returned. Revert to using the undocumented internal imaplib2 function,
that seemed to have worked without problems. This needs to be taken up to
the imaplib2 developer.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We have a reported case where response('UIDVALIDITY') returned [None]
which results in an ugly non-intuitive crash. Sanity check and report
something nicer.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Pass through the 'force' argument from selectro() to select() so that it
can also enforce a new SELECT even if we already are on that folder.
Also change the default parameter from '0' to 'False' to make clear that
this is a Bool.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Do not read in custom maildir flags, or we would try to sync them over
the wire. The next step will be to merge flag writes with existing
custom flags, so we don't lose information.
The long term goal will be to attempt to sync flags to the other side,
of course.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We were using the internal imaplib2 _get_untagged_response() functions a
few times. Replace 3 of these calls with 2 calls to the public function
response() rather than fudging with internals that could change anytime.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
These were needed for python <2.6 compatability, but since we depend on
python 2.6 now, these can go.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This change looks harmless, but it fixes a severe bugfix, potentially
leading to data loss! It fixes the "on n new uploads, it will redownload
n-1, n-2, n-3,... messages during the next syncs" condition, and this is
what happens:
If there are more than one Mails to upload to a server, we do that by
repeatedly invoking folder.IMAP.savemessage(). If the server supports
the UIDPLUS extension we query the resulting UID by doing a:
imapobj._get_untagged_response('APPENDUID', True)
and that is exactly the problem. The "True" part causes the reply to
remain in the "response stack" of the imaplib2 library. When we do
the same call on a subsequent message and the connection is still on the
same folder, we will get the same UID response back (imaplib2 only looks
for the first matching response and returns that). The only time we
clear the response stack, is when the IMAP connection SELECTS a
different folder.
This means that when we upload 10 messages, the IMAP server gives us
always the same UID (that of the first one) back. And trying to write
out 10 different messages with the same UID will confuse OfflineIMAP.
This is the reason why we saw the ongoing UPLOADING/DOWNLOADING behavior
that people reported. And this is the reason why we saw the
inconsistency in the UID mapping in the IMAP<->IMAP case.
I urge everyone to upgrade ASAP. Sorry for that, I don't know why the
problem only became prevalent in the recent few releases as this code
has been there for quite a while.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
