We were not able to handle ~/... type of path configurations and we
crashed with mysterious SSL errors when no file was found at the
configured location. Expand '~' and bomb out with usable error messages
in case such a file does not exist. This will still not protect against
corrupt cacert files but it goes a long way towards user friendliness.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
imaplib2 does not use socket, so does not know about the
defaulttimeout we set based on the config. Instead, we explicitly pass
the default timeout.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
For read(), the imaplib2 version seems to work perfectly well. The
others aren't used any more, either by imaplib2, nor by us, so we may
as well get rid of them.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The biggest change here is that imapobj.untagged_responses is no
longer a dictionary, but a list. To access it, I use the semi-private
_get_untagged_response method.
* offlineimap/folder/IMAP.py (IMAPFolder.quickchanged,
IMAPFolder.cachemessagelist): imaplib2 now explicitly removes its
EXISTS response on select(), so instead we use the return values from
select() to get the number of messages.
* offlineimap/imapserver.py (UsefulIMAPMixIn.select): imaplib2 now
stores untagged_responses for different mailboxes, which confuses us
because it seems like our mailboxes are "still" in read-only mode when
we just re-opened them. Additionally, we have to return the value
from imaplib2's select() so that the above thing works.
* offlineimap/imapserver.py (UsefulIMAPMixIn._mesg): imaplib2 now
calls _mesg with the name of a thread, so we display this
information in debug output. This requires a corresponding change to
imaplibutil.new_mesg.
* offlineimap/imaplibutil.py: We override IMAP4_SSL.open, whose
default arguments have changed, so update the default arguments. We
also subclass imaplib.IMAP4 in a few different places, which now
relies on having a read_fd file descriptor to poll on.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
imaplib2 has slightly different semantics than standard imaplib, so
this patch will break the build, but I thought it was helpful to have it as
a separate commit.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This change does not do anything yet with imaplib2, merely makes it
available for future commits.
This file is identical to the one at
http://sydney.edu.au/engineering/it/~piers/python/imaplib2 .
imaplib2, written by the same guy who wrote imaplib, is very different
from imaplib itself. Calling it a modified version from the standard
distribution is misleading. It's more like a complete rewrite. As
such, it's not really possible to summarize what was changed.
The largest thing is that imaplib2 is "threaded". Instead of doing
blocking writes/reads on the socket during/after every command,
imaplib2 forks off threads to read and write to the socket based on
input and output buffers. This opens the door to asynchronous
commands (every command is potentially asynchronous, according to the
docs), and in particular IDLE, which is by definition an asynchronous
command.
The author writes: "imaplib2 can be substituted for imaplib in
existing clients with no changes in the code", but that's pretty
misleading. It might be true for certain simple users of imaplib, but
for us it's completely false. Among other things, how untagged
responses are stored in-memory is different -- instead of a hash
table, it's a list. I'm guessing this is to preserve order of
responses.
I think there are other miscellaneous improvements, like I think
imaplib2 is IPv6 safe out-of-the-box, but I haven't conducted an
extremely thorough examination of the differences :)
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The rfc822 module has been deprecated since python 2.3, and conversion to
the email module is straightforward, so let us do that. rfc822 is
completely gone in python3.
This also fixes a bug that led to offlineimap abortion (but that code path
is apparently usually not exercised so I did not notice:
rfc822|email.utils.parsedate return a tuple which has no named attributes,
but we were using them later in that function. So pass the tuple into a
struct_time() to get named attributes.
While reading the docs, I noticed that email.parsedate returns invalid
daylight savings information (is_dst attribute), and we are using it
anyway. Oh well, the imap server might think the mails are off by an hour
at worst.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The only reason we used it here was to do a
traceback.print_exc(StringIO()) to get a string of our traceback. But we
can simply use traceback.format_exc() which exists since python 2.4.
One less module (and it is in the way to python 3 compatability too)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In commit 7a2a0225 [Don't pass list to ui.adding/deletingflags] we changed the
list logic for a per folder logic but forgot to remove one instance of
"destlist" which isn't valid anymore.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
More convenient way to test if a certain uid exists and getting a list
of all uids. Also, the SQL backend will have efficient overrides for
these methods.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than always having to call len(getmessagelist.keys()) as was done
before. No functional change, just nicer looking code. Also the SQLite
backend or other backends could implement more efficient implementations.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously we would output:
Folder sync sspaeth.de[INBOX.INBOX201004]:
Syncing INBOX.INBOX201004: IMAP -> Maildir
Folder sync sspaeth.de[INBOX.INBOX201006]:
Syncing INBOX.INBOX201006: IMAP -> Maildir
Folder sync sspaeth.de[INBOX.INBOX201009]:
Syncing INBOX.INBOX201009: IMAP -> Maildir
which is very repetitive and cluttered. By naming the folder sync
threads just according to the account and not the folder, the output
looks much nicer:
Folder sync [sspaeth.de]:
Syncing INBOX.INBOX201004: IMAP -> Maildir
Syncing INBOX.INBOX201006: IMAP -> Maildir
Syncing INBOX.INBOX201009: IMAP -> Maildir
If syncing multiple accounts in parallel, we will still get headers
indicating the account:
Folder sync [sspaeth.de]:
Syncing INBOX: IMAP -> Maildir
Syncing INBOX.INBOX201006: IMAP -> Maildir
Folder sync [gmail]:
Syncing INBOX: IMAP -> Maildir
This is a small fix that makes the output much nicer in my opinion.
Also don't output the thread name if we are in the MainThread, e.g. when
we output the initial offlineimap banner.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Read() should return empty string when EOF happen, instead of looping
forever. This is the right semantics of read(), and a wrapped version
should not change it.
If you read the read(2) system call manpage, it tells you that when EOF
is seen, return value is 0; it does not say
``loop forever when EOF happen''.
After the EOF detection is patched you can see the
following exception:
WARNING: ERROR attempting to copy message 344 for account Gmail:Traceback (most recent call last):
File "/usr/lib/pymodules/python2.6/offlineimap/folder/Base.py", line 282, in copymessageto
message = self.getmessage(uid)
File "/usr/lib/pymodules/python2.6/offlineimap/folder/IMAP.py", line 216, in getmessage
initialresult = imapobj.uid('fetch', '%d' % uid, '(BODY.PEEK[])')
File "/usr/lib/python2.6/imaplib.py", line 753, in uid
typ, dat = self._simple_command(name, command, *args)
File "/usr/lib/python2.6/imaplib.py", line 1060, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "/usr/lib/python2.6/imaplib.py", line 890, in _command_complete
raise self.abort('command: %s => %s' % (name, val))
abort: command: UID => socket error: EOF
Signed-off-by: Bao Haojun <baohaojun@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We only have one "dstfolder" at a time when deleting/adding flags, so no
need to pass in a list of those to the ui functions that output the log
info.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
They were not PEP-8 formatted, and some imports were simply
unnecessary. Removed those.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This enables us to just use the folder instance in the ui output and get
a name rather than having to call getname() all the time.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The previous syncing strategy was doing more than we needed to and was a
bit underdocumented. This is an attempt to clean it up.
1) Do away with the previous different code paths depending on
whether there is a LocalStatus file or not (the isnewfolder() test). We
always use the same strategy now, which makes the strategy easier to
understand. This strategy is simply:
a) Sync remote to local folder first
b) Sync local to remote
Where each sync implies a 4 pass strategy which does basically the same
as before (explained below).
2) Don't delete messages on LOCAL which don't exist on REMOTE right at
the beginning anymore. This prevented us e.g. from keeping local
messages rather than redownloading everything once LocalStatus got
corrupted or deleted. This surprised many who put in an existing local
maildir and expected it to be synced to the remote place. Instead, the
local maildir was deleted. This is a data loss that actually occured to
people!
3) No need to separately sync the statusfolder, we update that one
simultanously with the destfolders...
3) Simplified the sync function API by only taking one destdir rather
than a list of destdirs, we never used more anyway. This makes the code
easier to read.
4) Added plenty of code comments while I was going through to make sure
the strategy is easy to understand.
-----------------------------------------
Pass1: Transfer new local messages
Upload msg with negative/no UIDs to dstfolder. dstfolder should
assign that message a new UID. Update statusfolder.
Pass2: Copy existing messages
Copy messages in self, but not statusfolder to dstfolder if not
already in dstfolder. Update statusfolder.
Pass3: Remove deleted messages
Get all UIDS in statusfolder but not self. These are messages
that we have locally deleted. Delete those from dstfolder and
statusfolder.
Pass4: Synchronize flag changes
Compare flags in self with those in statusfolder. If msg has a
valid UID and exists on dstfolder (has not e.g. been deleted
there), sync the flag change to dstfolder and statusfolder.
The user visible implications of this change should be unnoticable
except in one situation:
Blowing away LocalStatus will not require you to redownload ALL of
your mails if you still have the local Maildir. It will simply recreate
LocalStatus.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Removing this lock makes the function not threadsafe, but then it is
only ever called from one thread, the main account syncer. Also, it
doesn't make it worse than most of the other functions in that class
which are also not threadsafe.
Removing this makes the code simpler, and removes the need to import the
threading module.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The previous ui names were pretty unwieldy. Is it TTYUI.TTY or
TTY.TTYUI? Do I have to use capitals and where?
Simplify the names by making them case insensitive and by dropping
everything before the dot.
So "Curses.Blinkenlights" can now be invoked as "blinkenlights" or
"BLINKENLIGHTS". The old names will still work just fine so the
transition should be smooth. We issue a warning that the long names are
deprecated.
Document in offlineimap.conf that we don't accept lists of fallback UIs,
but only one UI option (this was already the case before this commit but
still wrongly documented).
The list of accepted ui names is:
ttyui (default), basic, quiet, machineui, blinkenlights
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Multithreading speeds up account syncing a lot and the offlineimap
defaults are very conservative. Let's make it use 2 IMAP connections by
default to gain some of the benefits that offlineimap offers.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than inserting our own home-grown header, everytime we save a
message to an IMAP server, we check if we suport the UIDPLUS extension
which provides us with an APPENDUID reply. Use that to find the new UID
if possible, but keep the old way if we don't have that extension.
If a folder is read-only, return the uid that we have passed in per API
description in folder.Base.py
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The working horse of the savemessage() function, imaplib.append() was
hidden away in an assert statement. Pull the real functions out of the
asserts and simply assert on the return values. This looks less
convoluted and makes this easier to understand in my opinion.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
savemessage was too long and complex. Factor out the date guessing part
of the function and put it into a function of its own. The logic of the
date guessing is the same, however, we do not use the
imaplib.Time2InternalDate() function as it is buggy
(http://bugs.python.org/issue11024) and returns localized patches. So we
create INTERNALDATE ourselves and pass it to append() as a string.
This commit fixes a bug that international users used to pass an invalid
date to the IMAP server, which the server will either ignore or complain
about.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
savemessage_getnewheader was an undocmented, cryptic and overengineered
function. It generates a new unique value that can be used as a mail
header to be inserted. For this it used LOTS of randomness sources: hash
of the mail content, hash of the folder name, hash of the repository
name, the current time, a random() value, and the offlineimap version string.
All we need is something random. So reduce this to hash of content
appended by a random integer. Sufficient and somewhat faster to calculate.
Rename the function to actually describe accurately what it does or
would you have guessed that savemessage_getnewheader() did nothing more
than returning ('X-OfflineIMAP', <randomstring> )? Rename to
generate_randomheader() to make it clearer what this is all about.
Also document the function, describing what it does, and what it returns.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
There is a clumsy workaround for Darwin that chunks reads into 8kb
blocks to avoid huge memory allocations. First, this fix should not only
be required but on FreeBSD2.6 too (see
http://bugs.python.org/issue3531). Second, decent python versions (I
checked 2.6) already chunk in the SSL case anyway, so there is no need to do
that again. Remove that level of indirection.
http://evanjones.ca/python-memory.html claims that this problem has been
fixed since python 2.5, so we might consider removing the workaround
completely even for the non-SSL case.
Increase the chunk size on Mac from 8kb to 64kb. Even Macs should be
able to take that amount of memory usage nowadays.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
A typo was preventing profiling mode to really enable singlethreading
mode. Fixing the unfortunate typo of mine makes it work.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It is currently very hard to find out what folderfilter actually does
and makes it hard to debug for a user. With this patch if the user has
enabled "-d imap" (even better would perhaps be a different debug type
for this kind of thing?), we see a message
"Filtering out folder 'foo' due to folderfilter"
in the logs.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This enables us to make use of self.ui in all repositories without
having to import and use getglobalui() in all types of repositories and
all places.
Note that this patch only makes this available, it does not yet make use
of it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It was not clear how the reference value is used. Improve this by exposing
the internal path build system for the 'reference' option so that users
know what value is expected.
Also, explain how it impacts both nametrans and folderfilter options.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When e.g. specifying an invalid 'reference' value for an IMAP server to
a root folder that does not exist, we would previously have crashed with
a nonsensical and non-intuitive error message (trying to address an
element of a NoneType).
This will also raise an Exception (which should be ok, given that this
is really a misconfiguration on the user side), but it will explain to
the user WHY it exited.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The debug output for dequote, optionsplit is very verbose, outputing
what the functions are called with and what they return. Those functions
are now very old mature and rather simple, so it suffices to output
their return value rather than cluttering out log with too much
uninteresting garbarge.
This makes log files much more readable.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
A Repository() returns the correctly instanciated dervivate of a
BaseRepository, depending on the parameters passed to it. The returned
instance is eg an ImapRepository(). This makes the code look nicer,
and we have less functions lying around outside of classes (no more
global LoadRepository() function).
This will also enable us to conveniently hand back a
LocalStatusRepository based on SQLITE rather than plain text, if the
user configures this to be the experimental and optional backend
(once it exists).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
