The example illustrations were slightly cryptic;
modify them to be more obvious.
Fix case #2 (message starts with two line breaks)
where additional line break was shown and coded.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
The parameter's value is a string representing the linebreak,
and can sometimes contain just '\n', in which case naming it
crlf is slightly misleading.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
For cases like
http://article.gmane.org/gmane.mail.imap.offlineimap.general/6468
it is beneficial to see that folder name was translated and the result
of this translation on a single line: having log like
{{{
Folder Boring/Wreck [acc: tmarble@info9.net]:
Syncing Boring/Breck: Gmail -> Maildir
}}}
with translated name on the "Folder" line and original one on the
"Syncing" line isn't very intuitive.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
For servers without UIDPLUS we are inserting additional header
just after transformation '\n' -> CRLF was done. addmessageheaders()
was written to work with just '\n' as the separator, so X-OfflineIMAP
header wasn't preceeded by the CRLF, but just by '\n'.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
private methods prevent them from being overriden on derived classes. In
GmailFolder we need to override copymessageto, so it can't be private.
Before this commit, copymessageto was made private in Base but not in
GmailFolder. The end result was that labels were not set when copying
the message content, and always needed to be set on the label copying
pass.
Pointyhat-to: Eygene Ryabinkin
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
... and not self.filterheaders. With the current code this change
is no-op (since self.filterheaders is always passed as header_list),
but it is a bug in general.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
If when we request a LocalStatus folder, the folder has to be created,
we look whether the other backend has data, and if it does we migrate
it to the new backend.
The old backend data is left untouched, so that if you change back say
from sqlite to plaintext, the older data is still there. That should
not lead to data loss, only a slower sync while the status folder gets
updated.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
* Do not inherit LocalStatusSQLiteFolder class from the plaintext
one.
* Use some functions already in BaseFolder in both, plaintext and
sqlite classes.
* Add a saveall method. The idea is that saveall dumps the entire
messagelist to disk, while save only commits the uncommited
changes. Right now, save is noop for sqlite, and equivalent to
saveall for plaintext, but it enables to be more clever on when we
commit to disk in the future.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
* Implements Status Folder format v2, with a mechanism to upgrade an
old statusfolder.
* Do not warn about Gmail and GmailMaildir needing sqlite backend
anymore.
* Clean repository.LocalStatus reusing some code from
folder.LocalStatus.
* Change field separator in the plaintext file from ':' to '|'. Now
the local status stores gmail labels. If they contain field
separator character (formerly ':'), they get messed up. The new
character '|' is less likely to appear in a label.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Format headers X-Label and Keywords as a space separated list and all
other ones as comma-separated entities. This makes OfflineIMAP label
handling to be compatible with some user agents that recognise these
headers.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
When synclabels config flag is set to "yes" for the GMail repo,
offlineimap fetches the message labels along with the messages, and
embeds them into the body under the header X-Keywords (or whatever
'labelsheader' was set to), as a comma separated list.
It also adds an extra pass to savemessageto, that performs label
synchronization on existing messages from GMail to local, the same way
it is done with flags.
We also introduce GmailMaildir repository that adds functionality to
change message labels. It keeps track of messages modification time,
so one can quickly detect when the labels may have changed.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
In preparation for GMail label sync, we had split our some functionality
that will be needed further into their own functions. This also permitted
the code to look more compact and concise.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
When filterheaders is set to a comma-separated list of headers,
OfflineIMAP removes those headers from messages before uploading them
to the server.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Since we just do multiple passes for saving the message without
actually modifying its content (apart from header insertion that
is CRLF-clean), we can change line ends to the proper CRLF just
once.
And we can also get message's date only once too.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Make external API of class/module to be smaller, explicitely mark
all internal functions. Also annotate methods that are implemented
as the part of the parent class interface.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Allow people who want folder filtering to depend on the external
conditions or to make it dynamic for other reasons, to do what
they want.
New repository configuration knob 'dynamic_folderfilter' was
introduced; it defaults to 'False' that matches historical behaviour.
GitHub: #73
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
This commit fixes the case when we're invoking releaseconnection()
for a given imapobj twice.
This bug manifests itself as
{{{
ValueError: list.remove(x): x not in list
File "[...]/offlineimap/folder/IMAP.py", line 615, in savemessage
self.imapserver.releaseconnection(imapobj)
File "[...]/offlineimap/imapserver.py", line 130, in releaseconnection
self.assignedconnections.remove(connection)
}}}
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Second argument is exception traceback, not the message; without this
tracebacks like mentioned in
http://permalink.gmane.org/gmane.mail.imap.offlineimap.general/5712
were happening when this exception handling block was hit.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Global or per-repository option utime_from_message tells OfflineIMAP
to set file modification time of messages pushed from one repository
to another basing on the message's "Date" header.
This is useful if you are doing some processing/finding on your
Maildir (for example, finding messages older than 3 months),
without parsing each file/message content.
From: Cyril RUSSO <boite.pour.spam@gmail.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
IMAP servers can return `NO` responses to the `APPEND` command,
e.g. here's an example response from Groupwise's IMAP server:
NO APPEND The 1500 MB storage limit has been exceeded.
In this case, savemessage() should abort the repository sync rather
than returning UID 0 which would cause the local copy of the message
being saved to get irreversibly deleted.
Signed-off-by: Adam Spiers <offlineimap@adamspiers.org>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
They are redundant in all pruned cases and sometimes even create some
problems, e.g., when one tries to jump through paragraphs in vi.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
This eases testing of option values inside the code. This instance
is implemented as the read-only copy of the obtained 'options' object,
so callers won't be able to modify its contents.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
The problem lies in the fact that offlineimap.folder.Base's method
syncmessagesto_copy() uses threaded code everytime it is suggested by
the derived class's suggeststhreads() (currently, only IMAP does this
suggestion), but offlineimap/init.py will not spawn the
exitnotifymonitorloop() from offlineimap.threadutil.
The root cause is that ExitNotifyThread-derived threads need
offlineimap.threadutil's exitnotifymonitorloop() to be running the
cleaner for the exitthreads Queue(), because it fills the queue via
the run() method from this class: it wants to put() itself to the
Queue on exit, so when no exitnotifymonitorloop() is running, the
queue will fill up. And if this thread is an instance of
InstanceLimitedThread that hits the limit on the number of threads,
then it will hold the instancelimitedsems[] semaphore will prevent
other InstanceLimitedThread()s of the same name to pass its start()
method.
The fix is to avoid using threaded code if we're running
single-threaded.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Obtained-from: X-Ryl669 <boite.pour.spam@gmail.com>
Fix imapfolder.getmessageinternaldate misparsing the Date:
header from emails due to a bug or surprising behaviour by
email.utils.parsedate. This is because email.utils.parsedate's
return value contains the unadjusted hour value from the string
parsed but does not include information about the time zone in
which it is specified. For example (Python 2.7.3):
$ python -c "import email.utils;
print email.utils.parsedate('Mon, 20 Nov 1995 19:12:08 -0500')"
(1995, 11, 20, 19, 12, 8, 0, 1, -1)
(the -1 is the isdst field); the -0500 time zone is completely
ignored, so e.g. the same input with time "19:12:08 +0300" has
the same result. When passed to time.struct_time as allowed per
the parsedate documentation, this time is interpreted in GMT and
thus deviates from the correct value by the timezone offset
(in this example, -5 hours).
I consider this a bug in email.utils.parsedate: In my opinion,
since the return value of the parsetime doesn't include a timezone,
it should be expressed in terms of UTC rather than in terms of the
time zone from the Date header; the existence of
email.utils.parsedate_tz, to which I've switched, indicates that
maybe the authors were aware of this problem.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than later when retrieving them by IMAP, we determine the
"sync_this" value of a folder on Folder() initialization.
This also fixes a bug where folder structure was not propagated when
a folder was filtered out but included in the folderincludes list.
Patch by Dave, split and modified slightly by Sebastian
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
While looking at the code to investigate if an why we sometimes don't
seem to honor the write lock, I made it use the more modern "with lock:"
pattern.
Still have not found out how we could ever be using 2 instances of the
LocalStatusFolder for the same folder.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
As Gmail was only announcing the presence of the UIDPLUS extension
after we logged in, and we were then only getting server
capabilities before, a hack was introduced that checked the
existence of an APPENDUID reply, even if the server did not claim
to support it.
However, John Wiegley reports problems, where the APPENDUID would
be None, and we attempt to go this path (it seems that imaplib2
returns [None] if there is no such reply, so our test here for "!="
might fail. Given that this is an undocumented imaplib2 function
anyway, and we do fetch gmail capabilities after authentication,
this hack should no longer be necessary.
We had problems there earlier, where imapobj.response() would
return [None] although we had received a APPENDUID response from
the server, this might need more debugging and testing.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reported by sharat87 in https://github.com/spaetz/offlineimap/pull/38,
he would often get an unhandled Exception when trying to
releaseconnection() a connection that was not in the pool of
connections.
The reason this could happen is that when folder.IMAP.quickchanged()
raises an Exception in select(), we would release the connection in the
"except" handling, and than release the same connection in the "finally"
clause, which led to the error. The right thing is to only release the
connection once, of course.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This reverts commit 4d47f7bf3c.
This is one of two candidates for introducing the instabilities that
John Wiegley observed. We need to reintroduce with careful testing only.
The original patch has been mostly reverted.
This reverts commit 47390e03d6.
It is one of two potential candidates for the APPENDUID
regression that John Wiegley reported. We need to examine this
carefully before reintroducing this patch.
Resolved Changelog.draft.rst conflict.
Prevent savemessage(), and savemessageflags() to occur in dryrun mode in
all backends. Still need to protect against deletemessage().
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Bail out with a better Exception and error text. The whole mapped
UID situation needs to be improved though.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If we cannot identify the new UID after a sendmessage(), log a better error
message, including the server response for better debugging.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This allows to compare folders directly with strings. It also allows
constructs such as "if 'moo' in repo.getfolders()".
See the code documentation for the exact behavior (it basically is equal if
it's the same instance *or* a string matching the untranslated folder name.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
There is no need to cast 0 to 'long' even if we want to compare it to long
numbers in modern pythons.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This won't work in python3 anymore, so just use sorted() when needed.
In one case, we could remove the sort() completely as were were sanity checking
one line above, that we only having one UID as response which makes sorting
unneeded.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
'set' is builtin since python2.6, so remove the imports. Also 'ssl' exists
since 2.6 and has everything we need, so no need for conditional import
tests here anymore.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
To have the code work in python3, we need to convert all occurences of
raise Exception, "text" to be proper functions. This style also adheres to PEP8.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
It can lead to potential dataloss (see recent commit log where I added a
scary warning about it to offlineimap.conf).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Most servers support the UIDPLUS extension, and we don't have to search
headers after each uploaded message. There is no need to CHECK the imap
server after each message when there is no need to search headers.
I have not measured the performance impact on real world servers, but
this lets us do less unneeded work in the common case.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We were not cleaning out possibly existing APPENDUID messages before
APPENDing a new message. In case an old message were still hanging
around, this could *possibly* lead to retrieving and old UID. Things
should have been fine, but we do want to play safe here.
Also, make use of the "official" imaplib2 .response() command rather
than the internal _get_untagged_response() function.
Remove the hack that we would be looking for APPENDUID responses even if
the server claimed not to support the UIDPLUS ext. We now poll server
CAPABILITIES after login, and Gmail does provide us with the UIDPLUS
capability after login.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.
1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
from the imaplib2 result stack, so subsequent calls would return "None".
As various functions can invoke this, this led to some errors that we
avoid by caching the current UIDVALIDITY value in the Folder instance.
There are more simplifications and improvements to be made.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When deleting many (eg 2000) mails using the SQLITE backend, this takes
a long time durig which OfflineImap can not be aborted via
CTRL-C. Thinking it had frozen permanently, I killed it hard, leaving a
corrupted db journal (which leads to awkwards complaints by OLI on
subsequent starts!). That shows that delete performance is critical and
needs improvement.
We were iterating through the list of messages to delete and deleted
them one-by-one execute()'ing a new SQL Query for each message. This
patch improves the situation by allowing us to use executemany(), which
is -despite still being one SQL query per message- much faster. This is
because rather than performing a commit() after each mail, we now do
only one commit() after all mails have been deleted.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Recently the internal function use of imaplib2's _get_untagged_response()
was switched to use the public documented .response() function (which
should return the same data). However within a few fays we received reports
that both uses of a) the UIDVALIDITY fetching and b) the APPENDUID fetching
returned [None] as data although the IMAP log definitely shows that data
was returned. Revert to using the undocumented internal imaplib2 function,
that seemed to have worked without problems. This needs to be taken up to
the imaplib2 developer.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We have a reported case where response('UIDVALIDITY') returned [None]
which results in an ugly non-intuitive crash. Sanity check and report
something nicer.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Pass through the 'force' argument from selectro() to select() so that it
can also enforce a new SELECT even if we already are on that folder.
Also change the default parameter from '0' to 'False' to make clear that
this is a Bool.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Do not read in custom maildir flags, or we would try to sync them over
the wire. The next step will be to merge flag writes with existing
custom flags, so we don't lose information.
The long term goal will be to attempt to sync flags to the other side,
of course.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We were using the internal imaplib2 _get_untagged_response() functions a
few times. Replace 3 of these calls with 2 calls to the public function
response() rather than fudging with internals that could change anytime.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
These were needed for python <2.6 compatability, but since we depend on
python 2.6 now, these can go.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This change looks harmless, but it fixes a severe bugfix, potentially
leading to data loss! It fixes the "on n new uploads, it will redownload
n-1, n-2, n-3,... messages during the next syncs" condition, and this is
what happens:
If there are more than one Mails to upload to a server, we do that by
repeatedly invoking folder.IMAP.savemessage(). If the server supports
the UIDPLUS extension we query the resulting UID by doing a:
imapobj._get_untagged_response('APPENDUID', True)
and that is exactly the problem. The "True" part causes the reply to
remain in the "response stack" of the imaplib2 library. When we do
the same call on a subsequent message and the connection is still on the
same folder, we will get the same UID response back (imaplib2 only looks
for the first matching response and returns that). The only time we
clear the response stack, is when the IMAP connection SELECTS a
different folder.
This means that when we upload 10 messages, the IMAP server gives us
always the same UID (that of the first one) back. And trying to write
out 10 different messages with the same UID will confuse OfflineIMAP.
This is the reason why we saw the ongoing UPLOADING/DOWNLOADING behavior
that people reported. And this is the reason why we saw the
inconsistency in the UID mapping in the IMAP<->IMAP case.
I urge everyone to upgrade ASAP. Sorry for that, I don't know why the
problem only became prevalent in the recent few releases as this code
has been there for quite a while.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than always parsing the filename, we only need to do so if the flags
have actually changed, otherwise we can keep the filename.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, assigning a new UID to a mapped IMAP or Maildir repository
was done by loading the "local" item, saving it under a new UID and
deleting the old one. This involved lots of disk activity for nothing
more than an effective file rename in Maildirs, and lots of network
usage in the MappedUID cases.
We do this on every upload from a local to a remote item, so that can
potentially be quite expensive. This patch lets backends that support it
(Maildir, MappedUID) efficiently rename the file rather than having to
read the mail content, write it out as a new file and delete the old
file. This speeds up uploads from Maildir and the MappedUID server.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Various functions (such as change_message_uid) will want to construct
maildir filenames, so factor out the code into a helper.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Create a helper function that retrieves the UID, folder MD5, and Flags from
a message filename.
We need these items when we simply want to rename (=new UID) a Maildir
message file later. The new function can give us these components.
Rework, so we cache the calculation of the folder's md5 value once, it
never changes and we call it a lot.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If someone had a custom :2,a flag, adding a new flag would lead to the
invalid maildir filename ...a:2,... due to regex deficiencies not coping
with this. Fix this so we alway produce valid maildir names.
Note that custom flags are still problematic: as the syncing to the
remote IMAP server will fail, the next sync will assume that they have
been removed from the remote IMAP side and they will be removed from the
local Maildir then. We will need to think about how to handle this. At
least, with this patch we won't lose standard flags and won't produce
invalid maildir names.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The regex for catching Maildir message flags was
self.infosep + '.*2,([A-Z]+)' (infosep being ':').
The .* is bogus, as there is nothing between the : and the 2, per
maildir name specification, so remove that unneeded piece.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, we would simply bail out in an ugly way, potentially leaving
temporary files around etc, or while writing status files. Hand SIGINT
and SIGTERM as an event to the Account class, and make that bail out
cleanly at predefined points. Stopping on ctrl-c can take a few seconds
(it will e.g. finish to transfer the ongoing message), but it will shut
down cleanly.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
As reported in https://github.com/spaetz/offlineimap/pull/2, we would
fail when files are empty because file.read() would throw attribute
errors.
Fix this by removing the superfluous read() check and additionally log
some warning message.
Reported-by: Ralf Schmitt
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
All ExitNotifyThreads and InstanceLimitThreads are setDaemon(True) in their
constructor, so there is no need to do that again in the code.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
1) Differentiate error messages between imaplib.abort and imaplib.error
exceptions in the log.
2) Drop connections in the case of imapobj.error, it also might denote a
broken connection.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The default parameter value was "None", and we were comparing that
directly to the imaplib2 value of is_readonly which is False or True, so
the comparison always returned "False".
Fix this by setting the default parameter to "False" and not
"None". Also convert all users of that function to use False/True.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Only import the lock, that we actually need. Also import the with statement
for use with python 2.5. We'll need it for sure in this file.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
A repositories 'reference value is always prefixed to the full folder
path, so we should do so when creating a new one. The code had existed
but was commented out since 2003, I guess the "reference" option is not
too often used.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Output (2 of 500) when logging message copying. This required moving of
self.ui.copyingmessage into a different function where we actually have
the information about the progress handy.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When checking for the IMAP4.abort() exception, we need of course to
perform:
except imapobj.abort:
and not
except imapobj.abort():
Thanks to Johannes Stezenbach <js@sig21.net> for pointing to the glitch.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Dave identified a case where our new dropped connection handling did
not work out correctly: we use the retry_left variable to signify
success (0=success if no exception occured).
However, we were decrementing the variable AFTER all the exception
checks, so if there was one due to a dropped connection, it
could well be that we 1) did not raise an exception (because we want to
retry), and 2) then DECREMENTED retry_left, which indicated "all is
well, no need to retry".
The code then continued to check() the append, which failed with the
above message (because we obtained a new connection which had not even
selected the current folder and we were still in mode AUTH). The fix is
of course, to fix our logic: Decrement retry_left first, THEN decide
whether to raise() (retry_left==0) or retry (retry_left>0) which would
then correctly attempt another loop. I am sorry for this newbie type of
logic error. The retry count loop was too hastily slipped in, it seems.
Reported-by: Dave Abrahams <dave@boostpro.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If APPEND raises abort(), the (typ, dat) variables will not be set, so
we should not be using it for the OfflineImapError Exception
string. Fixing and prettifying the string formatting a bit at the same
time.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
nametrans rules can lead to different visiblename names for the
top-level directory, specifically both '.' and '' (the latter was
recently introduced). However, we need to be able to compare folder
names to see if we need to create a new directory or whether a directory
already exists, so we need to be able to compare a repositories
visiblename (=transposed via nametrans rule) with another folder.
To make the top-level directory comparison happen, we enforce a
top-level name of '', so that comparisons work.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
