Commit Graph

2443 Commits

Author SHA1 Message Date
Ethan Glasser-Camp
105da1b0c3 Always logout() on imaplib2 objects, even during exceptions
Without this, trying to Ctrl-C out of offlineimap will go into a hang.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-25 18:26:22 +01:00
Nicolas Sebrecht
756eb72494 Merge branch 'master' into next 2011-03-24 19:08:30 +01:00
Nicolas Sebrecht
51b89e6c43 Merge branch 'maint'
Conflicts:
	Changelog.draft.rst
	Makefile
2011-03-24 19:07:34 +01:00
Sebastian Spaeth
65faec834f accounts.py: Print the stacktrace via traceback module
All other instances were converted to format crash output including a
stacktrace, but this one seems to have been left out. Make Exceptions
print their stacktrace here too.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-24 19:03:16 +01:00
Vincent Beffara
bb48b6deaf Get rid of the UsefulIMAP4 classes
The three classes with names starting with UsefulIMAP4 were used for two
purposes, to include the UsefulIMAPMixIn class and to implement various
system-specific kludges. None of these kludges remain, so it is cleaner
to include UsefulIMAPMixIn directly in imaplibutil and forget about them
for good.

Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-23 20:00:35 +01:00
Vincent Beffara
9f03f41b70 Move UsefulIMAPMixIn to imaplibutil.py
Now that we do not need any system-specific hack, the three UsefulIMAP4
classes have become empty and should be removed. This implies importing
UsefulIMAPMixIn directly from the classes defined in imaplibutil.

Prepare this change by moving the code into imaplibutil.py. Functionally
this is a no-op.

Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-23 20:00:30 +01:00
Vincent Beffara
9a277cfd02 Remove a darwin-specific workaround for read()
Because of a buggy realloc() implementation in earlier versions of
Python on Mac OS X, we had to cut reads into manageable chunks by hand;
this is no more needed with Python 2.6, and besides it causes problems
with imaplib2, which we now use. Revert the special case to use the
system's read() instead, which is now safe.

Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-23 20:00:23 +01:00
Sebastian Spaeth
3f77afeb8a offlineimap.conf: Clarify password options via netrc
Document that only one user name per host name can be given via netrc
file.  Reformat the enumeration text.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-22 18:30:40 +01:00
Nicolas Sebrecht
37d0fe8b01 v6.3.2.1
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 20:46:18 +01:00
Nicolas Sebrecht
fe0e17e45f v6.3.3-rc1
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 18:09:31 +01:00
Nicolas Sebrecht
3eba44f636 Merge branch 'ss/uidmaps' into next 2011-03-16 18:05:35 +01:00
Sebastian Spaeth
fd28c5a2d3 folder/IMAP: savemessage() should just save flags if uid already exists
As the LocalStatus and UIDMap backend already did: If the uid already
exists for savemessage(), only modify the flags and don't append a new
message.

We don't invoke savemessage() on messages that already exist in our sync
logic, so this has no change on our current behavior. But it makes
backends befave more consistent with each other.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 18:05:16 +01:00
Sebastian Spaeth
dc3ad723c9 Give some love to UIDMaps
- Some documentation improvements, this is a severely underdocumented
  class. This still needs some further improvements though.

- Don't use apply(Baseclass) (which is going away in Python 3), use
  IMAPFolder.__init__(self, *args, **kwargs).

- Don't call ValueError, string. It is ValueError(string)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 18:05:16 +01:00
Nicolas Sebrecht
3b45782cc4 Merge branch 'master' into next 2011-03-15 19:28:02 +01:00
Nicolas Sebrecht
21342ab164 Merge branch 'ns/clean-import-pep8' into next 2011-03-15 18:52:23 +01:00
Nicolas Sebrecht
fa2e9f5a6a update Changelog
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-15 18:49:43 +01:00
Nicolas Sebrecht
180e86007a Merge branch 'maint' 2011-03-15 18:46:26 +01:00
Sebastian Spaeth
1c71e37f8f Sanity checks for SSL cacertfile configuration
We were not able to handle ~/... type of path configurations and we
crashed with mysterious SSL errors when no file was found at the
configured location. Expand '~' and bomb out with usable error messages
in case such a file does not exist. This will still not protect against
corrupt cacert files but it goes a long way towards user friendliness.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-15 18:45:31 +01:00
Nicolas Sebrecht
44eefae043 cleanup import statements and conform to PEP-8
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 21:35:33 +01:00
Nicolas Sebrecht
933d7c4eed Merge branch 'egc/imaplib2' into next
Rebased-For-Conflicts-Resolution-By: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-03-14 19:54:07 +01:00
Ethan Glasser-Camp
5ea95002f5 Create new connections with a timeout
imaplib2 does not use socket, so does not know about the
defaulttimeout we set based on the config. Instead, we explicitly pass
the default timeout.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
a139d9deed Throw away broken connections
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
197030da1a Remove obsolete read(), readline(), _read_upto() methods
For read(), the imaplib2 version seems to work perfectly well. The
others aren't used any more, either by imaplib2, nor by us, so we may
as well get rid of them.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
1bf4bee5e6 Update to match semantics of new imaplib2
The biggest change here is that imapobj.untagged_responses is no
longer a dictionary, but a list. To access it, I use the semi-private
_get_untagged_response method.

* offlineimap/folder/IMAP.py (IMAPFolder.quickchanged,
  IMAPFolder.cachemessagelist): imaplib2 now explicitly removes its
  EXISTS response on select(), so instead we use the return values from
  select() to get the number of messages.

* offlineimap/imapserver.py (UsefulIMAPMixIn.select): imaplib2 now
  stores untagged_responses for different mailboxes, which confuses us
  because it seems like our mailboxes are "still" in read-only mode when
  we just re-opened them.  Additionally, we have to return the value
  from imaplib2's select() so that the above thing works.

* offlineimap/imapserver.py (UsefulIMAPMixIn._mesg): imaplib2 now
  calls _mesg with the name of a thread, so we display this
  information in debug output. This requires a corresponding change to
  imaplibutil.new_mesg.

* offlineimap/imaplibutil.py: We override IMAP4_SSL.open, whose
  default arguments have changed, so update the default arguments. We
  also subclass imaplib.IMAP4 in a few different places, which now
  relies on having a read_fd file descriptor to poll on.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
f9413226b8 Import imaplib2 instead of imaplib
imaplib2 has slightly different semantics than standard imaplib, so
this patch will break the build, but I thought it was helpful to have it as
a separate commit.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
0a76f0a23d Import newest version of imaplib2
This change does not do anything yet with imaplib2, merely makes it
available for future commits.

This file is identical to the one at
http://sydney.edu.au/engineering/it/~piers/python/imaplib2 .

imaplib2, written by the same guy who wrote imaplib, is very different
from imaplib itself. Calling it a modified version from the standard
distribution is misleading. It's more like a complete rewrite. As
such, it's not really possible to summarize what was changed.

The largest thing is that imaplib2 is "threaded". Instead of doing
blocking writes/reads on the socket during/after every command,
imaplib2 forks off threads to read and write to the socket based on
input and output buffers. This opens the door to asynchronous
commands (every command is potentially asynchronous, according to the
docs), and in particular IDLE, which is by definition an asynchronous
command.

The author writes: "imaplib2 can be substituted for imaplib in
existing clients with no changes in the code", but that's pretty
misleading. It might be true for certain simple users of imaplib, but
for us it's completely false. Among other things, how untagged
responses are stored in-memory is different -- instead of a hash
table, it's a list. I'm guessing this is to preserve order of
responses.

I think there are other miscellaneous improvements, like I think
imaplib2 is IPv6 safe out-of-the-box, but I haven't conducted an
extremely thorough examination of the differences :)

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
e2354fd37c Remove some unneeded imports
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:52 +01:00
Sebastian Spaeth
2946a1ea5d Convert rfc822 module to email module
The rfc822 module has been deprecated since python 2.3, and conversion to
the email module is straightforward, so let us do that. rfc822 is
completely gone in python3.

This also fixes a bug that led to offlineimap abortion (but that code path
is apparently usually not exercised so I did not notice:
rfc822|email.utils.parsedate return a tuple which has no named attributes,
but we were using them later in that function. So pass the tuple into a
struct_time() to get named attributes.

While reading the docs, I noticed that email.parsedate returns invalid
daylight savings information (is_dst attribute), and we are using it
anyway. Oh well, the imap server might think the mails are off by an hour
at worst.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-13 20:30:35 +01:00
Nicolas Sebrecht
30d95ad0ac Merge branch 'ns/uibase-fix-destlist' into next 2011-03-12 13:48:52 +01:00
Nicolas Sebrecht
6887bd995d Merge branch 'ss/remove-stringio-in-threadutil' into next 2011-03-11 20:34:56 +01:00
Sebastian Spaeth
10024731e6 threadutil: Don't require the StringIO module
The only reason we used it here was to do a
traceback.print_exc(StringIO()) to get a string of our traceback. But we
can simply use traceback.format_exc() which exists since python 2.4.

One less module (and it is in the way to python 3 compatability too)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-11 20:34:33 +01:00
Nicolas Sebrecht
90949a4bfc UIBase: fix regression while deletingflags
In commit 7a2a0225 [Don't pass list to ui.adding/deletingflags] we changed the
list logic for a per folder logic but forgot to remove one instance of
"destlist" which isn't valid anymore.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-11 20:20:12 +01:00
Sebastian Spaeth
6c6fdfc769 folder: Implement helper functions uidexists() and getmessageuidlist()
More convenient way to test if a certain uid exists and getting a list
of all uids. Also, the SQL backend will have efficient overrides for
these methods.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-11 18:59:20 +01:00
Sebastian Spaeth
4d352a528a folder: Implement helper function getmessagecount()
Rather than always having to call len(getmessagelist.keys()) as was done
before. No functional change, just nicer looking code. Also the SQLite
backend or other backends could implement more efficient implementations.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-10 20:30:19 +01:00
Nicolas Sebrecht
5048d16913 Merge branch 'ss/declutter-tty-output' into next
Conflicts:
	Changelog.draft.rst
2011-03-10 19:26:26 +01:00
Nicolas Sebrecht
b1a570d393 Merge branch 'ns/fix-documentation-build' into next 2011-03-10 19:24:17 +01:00
Sebastian Spaeth
efcce01d64 Declutter TTY output
Previously we would output:

Folder sync sspaeth.de[INBOX.INBOX201004]:
 Syncing INBOX.INBOX201004: IMAP -> Maildir
Folder sync sspaeth.de[INBOX.INBOX201006]:
 Syncing INBOX.INBOX201006: IMAP -> Maildir
Folder sync sspaeth.de[INBOX.INBOX201009]:
 Syncing INBOX.INBOX201009: IMAP -> Maildir

which is very repetitive and cluttered. By naming the folder sync
threads just according to the account and not the folder, the output
looks much nicer:

Folder sync [sspaeth.de]:
 Syncing INBOX.INBOX201004: IMAP -> Maildir
 Syncing INBOX.INBOX201006: IMAP -> Maildir
 Syncing INBOX.INBOX201009: IMAP -> Maildir

If syncing multiple accounts in parallel, we will still get headers
indicating the account:

Folder sync [sspaeth.de]:
 Syncing INBOX: IMAP -> Maildir
 Syncing INBOX.INBOX201006: IMAP -> Maildir
Folder sync [gmail]:
 Syncing INBOX: IMAP -> Maildir

This is a small fix that makes the output much nicer in my opinion.

Also don't output the thread name if we are in the MainThread, e.g. when
we output the initial offlineimap banner.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-10 19:04:57 +01:00
Nicolas Sebrecht
b18bde70c9 doc: rst2xxx: fix ignoring output while determining the command name
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 22:27:35 +01:00
Nicolas Sebrecht
7b8d7501d1 Merge branch 'master' into next
Conflicts:
	Changelog.draft.rst
	offlineimap/imapserver.py
2011-03-07 21:55:43 +01:00
Haojun Bao
b94bf79258 fix hang because of infinite loop reading EOF
Read() should return empty string when EOF happen, instead of looping
forever. This is the right semantics of read(), and a wrapped version
should not change it.

If you read the read(2) system call manpage, it tells you that when EOF
is seen, return value is 0; it does not say

	``loop forever when EOF happen''.

After the EOF detection is patched you can see the
following exception:

  WARNING: ERROR attempting to copy message 344 for account Gmail:Traceback (most recent call last):
    File "/usr/lib/pymodules/python2.6/offlineimap/folder/Base.py", line 282, in copymessageto
      message = self.getmessage(uid)
    File "/usr/lib/pymodules/python2.6/offlineimap/folder/IMAP.py", line 216, in getmessage
      initialresult = imapobj.uid('fetch', '%d' % uid, '(BODY.PEEK[])')
    File "/usr/lib/python2.6/imaplib.py", line 753, in uid
      typ, dat = self._simple_command(name, command, *args)
    File "/usr/lib/python2.6/imaplib.py", line 1060, in _simple_command
      return self._command_complete(name, self._command(name, *args))
    File "/usr/lib/python2.6/imaplib.py", line 890, in _command_complete
      raise self.abort('command: %s => %s' % (name, val))
  abort: command: UID => socket error: EOF

Signed-off-by: Bao Haojun <baohaojun@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:54:28 +01:00
Sebastian Spaeth
7a2a02254e Don't pass list to ui.adding/deletingflags
We only have one "dstfolder" at a time when deleting/adding flags, so no
need to pass in a list of those to the ui functions that output the log
info.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:11:10 +01:00
Sebastian Spaeth
387fbf3aaa ui: clean up importment statements
They were not PEP-8 formatted, and some imports were simply
unnecessary. Removed those.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:11:10 +01:00
Sebastian Spaeth
ab1df868c2 String representation of a Folder is its name
This enables us to just use the folder instance in the ui output and get
a name rather than having to call getname() all the time.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:11:10 +01:00
Sebastian Spaeth
3eee821382 Simplify the syncing strategy a bit
The previous syncing strategy was doing more than we needed to and was a
bit underdocumented. This is an attempt to clean it up.

1) Do away with the previous different code paths depending on
whether there is a LocalStatus file or not (the isnewfolder() test). We
always use the same strategy now, which makes the strategy easier to
understand. This strategy is simply:

a) Sync remote to local folder first
b) Sync local to remote

Where each sync implies a 4 pass strategy which does basically the same
as before (explained below).

2) Don't delete messages on LOCAL which don't exist on REMOTE right at
the beginning anymore. This prevented us e.g. from keeping local
messages rather than redownloading everything once LocalStatus got
corrupted or deleted. This surprised many who put in an existing local
maildir and expected it to be synced to the remote place. Instead, the
local maildir was deleted. This is a data loss that actually occured to
people!

3) No need to separately sync the statusfolder, we update that one
simultanously with the destfolders...

3) Simplified the sync function API by only taking one destdir rather
than a list of destdirs, we never used more anyway. This makes the code
easier to read.

4) Added plenty of code comments while I was going through to make sure
the strategy is easy to understand.
-----------------------------------------

 Pass1: Transfer new local messages
        Upload msg with negative/no UIDs to dstfolder. dstfolder should
        assign that message a new UID. Update statusfolder.

 Pass2: Copy existing messages
        Copy messages in self, but not statusfolder to dstfolder if not
        already in dstfolder. Update statusfolder.

 Pass3: Remove deleted messages
        Get all UIDS in statusfolder but not self. These are messages
        that we have locally deleted. Delete those from dstfolder and
        statusfolder.

 Pass4: Synchronize flag changes
        Compare flags in self with those in statusfolder. If msg has a
        valid UID and exists on dstfolder (has not e.g.  been deleted
        there), sync the flag change to dstfolder and statusfolder.

The user visible implications of this change should be unnoticable
except in one situation:
 Blowing away LocalStatus will not require you to redownload ALL of
 your mails if you still have the local Maildir. It will simply recreate
 LocalStatus.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:11:10 +01:00
Nicolas Sebrecht
976c071dd7 Merge branch 'ss/deprecate-current-ui-names' into next 2011-03-06 18:33:29 +01:00
Sebastian Spaeth
fc03475b9e Remove thread Lock() when saving UIDvalidity
Removing this lock makes the function not threadsafe, but then it is
only ever called from one thread, the main account syncer. Also, it
doesn't make it worse than most of the other functions in that class
which are also not threadsafe.

Removing this makes the code simpler, and removes the need to import the
threading module.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-06 18:33:11 +01:00
Nicolas Sebrecht
607eba178a Merge branch 'ss/maxconnections-to-2' into next 2011-03-06 18:31:06 +01:00
Sebastian Spaeth
4e28c7c93f Allow to use nicer UI names
The previous ui names were pretty unwieldy. Is it TTYUI.TTY or
TTY.TTYUI? Do I have to use capitals and where?

Simplify the names by making them case insensitive and by dropping
everything before the dot.

So "Curses.Blinkenlights" can now be invoked as "blinkenlights" or
"BLINKENLIGHTS". The old names will still work just fine so the
transition should be smooth. We issue a warning that the long names are
deprecated.

Document in offlineimap.conf that we don't accept lists of fallback UIs,
but only one UI option (this was already the case before this commit but
still wrongly documented).

The list of accepted ui names is:
  ttyui (default), basic, quiet, machineui, blinkenlights

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-06 15:33:29 +01:00
Sebastian Spaeth
c44e94d64a Set maxconnections default to 2
Multithreading speeds up account syncing a lot and the offlineimap
defaults are very conservative. Let's make it use 2 IMAP connections by
default to gain some of the benefits that offlineimap offers.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-06 15:27:38 +01:00
Sebastian Spaeth
3b8e1f91cd Implement APPENDUID support
Rather than inserting our own home-grown header, everytime we save a
message to an IMAP server, we check if we suport the UIDPLUS extension
which provides us with an APPENDUID reply. Use that to find the new UID
if possible, but keep the old way if we don't have that extension.

If a folder is read-only, return the uid that we have passed in per API
description in folder.Base.py

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00