Commit Graph

705 Commits

Author SHA1 Message Date
Sebastian Spaeth
b3a383d151 accounts: handle OfflineImapError severity FOLDER
Throw an OfflineImapError when SELECTing a folder is unsuccessful and
bail out with a FOLDER serverity. In accounts.py catch all
OfflineImapErrors and either just log the error and skip the folder or
bubble it up if it's severe.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 13:56:04 +02:00
Sebastian Spaeth
dd82f213f0 accounts: fix broken warn statement
Use two %s in the message for both string parameters.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 12:48:59 +02:00
Sebastian Spaeth
f4081985dc Prettify and use new uidexists() helper function
Make the folder classes use uidexists() more. Add some code
documentation while going through.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:50:22 +02:00
Sebastian Spaeth
0318c6ad34 Create LocalStatus or LocalStatusSQLite folders
Depending on the configuration we use the plain text or the new
experimental sqlite backend for the LocalStatus cache. Make plain text
the default status backend but allow people to configure
status_backend=sqlite in their [Account ...] section.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:35:03 +02:00
Sebastian Spaeth
0af9ef70a7 Factor out SQL retries
Test if sqlite is multithreading-safe and bail out if not. sqlite
versions since at least 2008 are.
But, as it still causes errors when 2
threads try to write to the same connection simultanously (We get a
"cannot start transaction within a transaction" error), we protect
writes with a per class, ie per-connection lock. Factor out the retrying
to write when the database is locked.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
af25c2779f repository.LocalStatus: Remove code duplication
Make getfolders() invoke getfolder() for each folder rather than
duplicating code. Also add a forgetfolders() implementation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
e1e9c8e831 Use self.doautosave rather than self.dofsync
doautosave was a useless variable before (it was *always* 1). So we
remove the self.dofsync variable and store in doautosave whether we
should fsync as often as possible (which really hurts performance).

The sqlite backend could (at one point) use the doautosave variable to
determine if it should autocommit after each modification.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
bd3766eca5 Make getnicename work again for classes that derive from object()
Python's new style classes derive from object and str(class().__class__)
will return a slightly different format. class().__class.__name__ will
still work for both old and new style classes, so use that instead.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
2fc12e875a Experimental LocalStatus stored in SQLite database
Based on patches by Stewart Smith, updated by Rob Browning.

plus:
- Inherit LocalStatusSQLFolder from LocalStatusFolder
  This lets us remove all functions that are available via our ancestors
  classes and are not needed.

- Don't fail if pysql import fails. Fail rather at runtime when needed.

- When creating the db file, create a metadata table which contains the
  format version info, so we can upgrade nicely to other formats.

- Create an upgrade_db() function which allows us to upgrade from any
  previous file format to the current one (including plain text)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
535f4592fc Replace 2 regexes with a single one
No functional changes

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Nicolas Sebrecht
c6259fbb86 Merge branch 'master' into next
Conflicts:
	Changelog.draft.rst
2011-05-05 21:16:02 +02:00
Sebastian Spaeth
f2ad4bc230 Fix typo to force singlethreading in debug mode
A typo prevented us from enforcing singlethreading mode when selecting debugging.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 21:12:10 +02:00
Sebastian Spaeth
deab62fbd8 Fix the broken thread debugging
Using threading._VERBOSE=1 is broken since python 2.6 till at least
python 3.2, (http://bugs.python.org/issue4188) so we can't use it for
our thread debugging.

Remove the usage of threading._VERBOSE, and implement a "light thread
debug log" that for now outputs information when a new thread is being
registered and when it is being unregistered. I am sure we will be able
to add more thread debugging information over the time.

Besides '-d thread' this will re-enable the usage of -d 'all' for the
most verbose debugging of all categories.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 21:07:24 +02:00
Sebastian Spaeth
5c7d7ee445 Output more detailed error on corrupt LocalStatus
When our LocalStatus cache is corrupt, ie e.g. it contains lines not in
the form number:number, we would previously just raise a ValueError
stating things like "too many values". In case we encounter clearly
corrupt LocalStatus cache entries, clearly raise an exception stating
the filename and the line, so that people can attempt to repair it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 20:09:57 +02:00
Sebastian Spaeth
f3774343db If we loop, exit the account synchronization after 3 failed attempts
This should get rid of intermittent network failures, but lets us bail
out on permanent errors.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 20:00:07 +02:00
Sebastian Spaeth
4608d14836 Handle OfflineImapError of severity REPO and CRIT
By aborting the account syncing, the looping and logging an error
message. We will introduce a ui.error() rather than a ui.warn() function
which saves all Exceptions in a Queue and outputs them at the end of the
program.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 20:00:07 +02:00
Sebastian Spaeth
0f45d89e34 Throw OfflineImapError on DNS error
In case we misconfigured a server name or are otherwise offline, a
socket.gaierror will be raised when attempting to connect. We catch that
case and raise an OfflineImapError with severity ERROR.REPO, meaning we
should stop syncing this account and continue with the next one.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 20:00:07 +02:00
Sebastian Spaeth
bc004a911a Add OfflineImapError class
This Exception can be thrown whenever a sync error occurs. It is used
for outputting sensible error messages to the user and it denotes a
"severity", it can tell offlineimap if we need to abort a message, a
folder, a repo sync, or whether we should indeed abort immediately.

The exception is not yet used in the code.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 20:00:07 +02:00
Sebastian Spaeth
3446b592f3 accounts: remove duplicated code paths
Currently, account.syncrunner() has 2 separate duplicated code paths
depending on whether we want to autorefresh after some waiting perios
or not. Unify those code paths by setting "looping = False" in case
self.refeshperiod == 0 after the first run. Behavior is identical to
before.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 19:52:24 +02:00
Sebastian Spaeth
f3249e0856 True 1-way sync (backup)
This commit enables true 1-way syncing between repositories. This has
often been demanded for backup purposes when you do not want to cause
accidental modifications of your backup that would be propagated to the
other side.

This has been implemented by allowing to configure a Repository as
'readonly' to forbid any modification on it.

'readonly' applies to all the type of repositories.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 20:41:36 +02:00
Sebastian Spaeth
3d4dc11a8b Introduce an *empty* debug type
This debug type will always be enabled whenever any debugging is enables
and it outputs debug messages that cannot be categorized among any of
imap, maildir (e.g. things that concern the sync logic).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 20:41:36 +02:00
Sebastian Spaeth
6add201436 Improve the developer API documentation
Improve the code documentation (still much more to do) and also add some
more meat to the structure of the developer documentation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 20:16:45 +02:00
Nicolas Sebrecht
e9a7afda6d Merge branch 'ss/corrupted-uidmap-file' into next
Conflicts:
	Changelog.draft.rst
2011-05-02 19:09:52 +02:00
Sebastian Spaeth
2ed1c357a0 More detailed error output on corrupt UID mapping files
This function will need much more "robustifying", but the very least we
can do is to print the file name and line that are giving trouble.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 19:08:59 +02:00
Nicolas Sebrecht
cda932351c Merge branch 'ss/folder-name-whitspaces' into next
Conflicts:
	Changelog.draft.rst
2011-04-28 18:29:14 +02:00
Dan Christensen
9bd7a21f16 Don't strip whitespace in the -f option
Allow leading and trailing spaces in folder names specified on the
command line.

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Dan Christensen <jdc@uwo.ca>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-28 18:28:23 +02:00
Sebastian Spaeth
163f1eefc4 Drop all connections when a sync failed
in accounts.sync() we would holdordropconnections() after each sync. But
depending on the repository configuration that might imply that
offlineimap tries to keep the same connections. But when a sync failed,
e.g. after a user had his computer suspended, it might be that our
connections that we have are worthless. So definitely drop them after a
failed sync.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 23:04:24 +02:00
Nicolas Sebrecht
0bc1c9583e Merge branch 'ss/folder-cleanup-getmessage' into next
Conflicts:
	offlineimap/folder/IMAP.py
2011-04-27 22:52:17 +02:00
Sebastian Spaeth
1ff628bd5c folder/IMAP: cleanup getmessage()
Add some comments how the data structures actually look like.
Describe the function properly, and make sure we only hold on to the
data connection as quickly as possible.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 22:49:18 +02:00
Sebastian Spaeth
c8f80bc6d2 We had been setting this variable twice
This was in the code for a very long time, it seems.
Remove one instance, no functional changes.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 22:35:39 +02:00
Sebastian Spaeth
e01968973d Use "%s" % account rather than account.name
uncritical patch, but we can make the code a bit shorter so why not.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:10 +02:00
Sebastian Spaeth
3e5d6f5450 Make str() of Repository() be its name
So we can simply hand a repository instance to error messages rather than
having to call Repository().getname() all the time.

This is not used yet.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:10 +02:00
Sebastian Spaeth
074ea14cf1 Make str() of Account() be its name
So we can simply hand an account instance to error messages rather than
having to call Account().getname() all the time.

This is not used yet.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:10 +02:00
Sebastian Spaeth
bb8546e846 Whitespace fixes
Just clean up lines which end in whitespaces.
Also adapt the copyright to the current year while touching the file.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:09 +02:00
Nicolas Sebrecht
b2be0c3697 Merge branch 'ss/sync-remove-neguid-pass' into next 2011-04-25 13:01:59 +02:00
Sebastian Spaeth
e20d8b9679 Remove upload neguid pass from sync logic
In order to optimize performance, we fold the 1st and 2nd pass of our
sync strategy into one. They were essentially doing the same thing:
uploading a message to the other side. The only difference was that in
one case we have a negative UID locally, and in the other case, we have
a positive one already.

This saves some time, as we don't have to run through that function on
IMAP servers anyway (they always have positive UIDs), and 2nd were we
stalling further copying until phase 1 was finished. So uploading a
single new message would prevent us from starting to copy existing
regular messages.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-25 13:01:53 +02:00
Sebastian Spaeth
36eb37b47d IMAP: reduce quickchanged() checks
For each folder we were making a second IMAP request asking for the
latest UID and compared that with the highest UID in our
statusfolder. This catched the case that 1 mail has been deleted by
someone else and another one has arrived since we checked, so that the
total number of mails appears to not have changed.

We don't capture anymore this case in the quickchanged() case.

It improves my performance from 8 to about 7.5 seconds per check (with lots of
variation) and we would benefit even more in the IMAP<->IMAP case as we do one
additional IMAP lookup per folder on each side then.

Do cleanups on whitespaces while in this file.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-25 12:58:47 +02:00
Nicolas Sebrecht
3b09695236 v6.3.3
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-21 18:34:24 +02:00
Nicolas Sebrecht
bf1c9e3233 Merge branch 'ss/maildir-quickchanged-cleanup' into next 2011-04-15 19:52:35 +02:00
Ethan Glasser-Camp
9e734006f6 Fix IMAP4_Tunnel to work with imaplib2
* IMAP4_Tunnel constructor should support base-class arguments, in
  order to support the timeout argument.

* IMAP4_Tunnel needs to store the member IMAP4.host, which is normally
  done in IMAP4.open().

* Update IMAP4_Tunnel.read() and IMAP4_Tunnel.send().  We turn on
  nonblocking mode for these sockets, so we can return immediately
  with whatever data is available.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-13 18:41:53 +02:00
Sebastian Spaeth
fdf22400b1 imaplib2: Bump from 2.20 to 2.22
This contains a fixed Time2InternalDate function and a more robust
socket connection, trying twice and raising an error only when that
fails (I believe). The actual code changes are rather minor.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-11 19:10:45 +02:00
Sebastian Spaeth
e37441cd19 folder/Maildir: Make use of helper functions
quickchanged() was iterating a lot, make use of some of the helper
functions that had been introduced recently and document the function a
bit better. No functional change.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-11 18:57:25 +02:00
Sebastian Spaeth
d762175af4 Make -f option with with folder names with spaces.
Previously ALL spaces had been stripped off. Now, only strip spaces
around the comma, so -f "INBOX, Deleted Mails" will work. You will still
need to quote or escape spaces so the shell hand the list as one command
line argument to offlineimap.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-11 18:29:44 +02:00
Sebastian Spaeth
58220fd8e7 Replace calls to getmessagelist() to alternatives
getmessagelist() is slow for the mapped UID case, so replace some of its
occurences with calls that are optimized for this case, ie
getmessagecount() and uidexists().

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-and-tested-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 17:53:14 +02:00
Sebastian Spaeth
ca012d3a81 Implement more efficient functions for the MappedUID case
We are calling getmessagelist() internally a lot, e.g. just to check if
a UID exists (from uidexist()). This is a very expensive operation in
the UIDMapped case, as we reconstruct the whole messagelist dict every
single time, involving lots of copying etc.

So we provide more efficient implementations for the uidexists()
getmessageuidlist() and getmessagecount() functions that are fast in the
UIDMapped case. This should solve the performance regression that was
recently observed in the Mapped UID case.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-and-tested-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 17:52:22 +02:00
Nicolas Sebrecht
0f85e592a3 Merge branch 'df/fix-debug-msg' into next 2011-03-28 16:50:18 +02:00
David Favro
954655b7ec Fixed bug: wrong number of arguments to debug() [IMAP.py].
Signed-off-by: David Favro <offlineimap@meta-dynamic.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 16:49:21 +02:00
Nicolas Sebrecht
71d0828cd6 Merge branch 'master' into next 2011-03-25 18:42:46 +01:00
Sebastian Spaeth
09515f8f90 Increase compatability with Gmail
When uploading a new message to Gmail we need to find out the UID it
assigned it, but Gmail does not advertize the UIDPLUS extension (in all
cases) and it fails to find the email that we just uploaded when
searching for it. This prevented us effectively from uploading to
gmail.

See analysis in
http://lists.alioth.debian.org/pipermail/offlineimap-project/2011-March/001449.html
for details on what is going wrong.

This patch increases compatability with Gmail by checking for APPENDUID
responses to an APPEND action even if the server did not claim to
support it. This restores the capability to upload messages to the
*broken* Gmail IMAP implementation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-03-25 18:42:16 +01:00
Ethan Glasser-Camp
105da1b0c3 Always logout() on imaplib2 objects, even during exceptions
Without this, trying to Ctrl-C out of offlineimap will go into a hang.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-25 18:26:22 +01:00
Nicolas Sebrecht
756eb72494 Merge branch 'master' into next 2011-03-24 19:08:30 +01:00
Nicolas Sebrecht
51b89e6c43 Merge branch 'maint'
Conflicts:
	Changelog.draft.rst
	Makefile
2011-03-24 19:07:34 +01:00
Sebastian Spaeth
65faec834f accounts.py: Print the stacktrace via traceback module
All other instances were converted to format crash output including a
stacktrace, but this one seems to have been left out. Make Exceptions
print their stacktrace here too.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-24 19:03:16 +01:00
Vincent Beffara
bb48b6deaf Get rid of the UsefulIMAP4 classes
The three classes with names starting with UsefulIMAP4 were used for two
purposes, to include the UsefulIMAPMixIn class and to implement various
system-specific kludges. None of these kludges remain, so it is cleaner
to include UsefulIMAPMixIn directly in imaplibutil and forget about them
for good.

Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-23 20:00:35 +01:00
Vincent Beffara
9f03f41b70 Move UsefulIMAPMixIn to imaplibutil.py
Now that we do not need any system-specific hack, the three UsefulIMAP4
classes have become empty and should be removed. This implies importing
UsefulIMAPMixIn directly from the classes defined in imaplibutil.

Prepare this change by moving the code into imaplibutil.py. Functionally
this is a no-op.

Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-23 20:00:30 +01:00
Vincent Beffara
9a277cfd02 Remove a darwin-specific workaround for read()
Because of a buggy realloc() implementation in earlier versions of
Python on Mac OS X, we had to cut reads into manageable chunks by hand;
this is no more needed with Python 2.6, and besides it causes problems
with imaplib2, which we now use. Revert the special case to use the
system's read() instead, which is now safe.

Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-23 20:00:23 +01:00
Nicolas Sebrecht
37d0fe8b01 v6.3.2.1
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 20:46:18 +01:00
Nicolas Sebrecht
3eba44f636 Merge branch 'ss/uidmaps' into next 2011-03-16 18:05:35 +01:00
Sebastian Spaeth
fd28c5a2d3 folder/IMAP: savemessage() should just save flags if uid already exists
As the LocalStatus and UIDMap backend already did: If the uid already
exists for savemessage(), only modify the flags and don't append a new
message.

We don't invoke savemessage() on messages that already exist in our sync
logic, so this has no change on our current behavior. But it makes
backends befave more consistent with each other.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 18:05:16 +01:00
Sebastian Spaeth
dc3ad723c9 Give some love to UIDMaps
- Some documentation improvements, this is a severely underdocumented
  class. This still needs some further improvements though.

- Don't use apply(Baseclass) (which is going away in Python 3), use
  IMAPFolder.__init__(self, *args, **kwargs).

- Don't call ValueError, string. It is ValueError(string)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 18:05:16 +01:00
Nicolas Sebrecht
3b45782cc4 Merge branch 'master' into next 2011-03-15 19:28:02 +01:00
Nicolas Sebrecht
180e86007a Merge branch 'maint' 2011-03-15 18:46:26 +01:00
Sebastian Spaeth
1c71e37f8f Sanity checks for SSL cacertfile configuration
We were not able to handle ~/... type of path configurations and we
crashed with mysterious SSL errors when no file was found at the
configured location. Expand '~' and bomb out with usable error messages
in case such a file does not exist. This will still not protect against
corrupt cacert files but it goes a long way towards user friendliness.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-15 18:45:31 +01:00
Nicolas Sebrecht
44eefae043 cleanup import statements and conform to PEP-8
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 21:35:33 +01:00
Ethan Glasser-Camp
5ea95002f5 Create new connections with a timeout
imaplib2 does not use socket, so does not know about the
defaulttimeout we set based on the config. Instead, we explicitly pass
the default timeout.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
a139d9deed Throw away broken connections
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
197030da1a Remove obsolete read(), readline(), _read_upto() methods
For read(), the imaplib2 version seems to work perfectly well. The
others aren't used any more, either by imaplib2, nor by us, so we may
as well get rid of them.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
1bf4bee5e6 Update to match semantics of new imaplib2
The biggest change here is that imapobj.untagged_responses is no
longer a dictionary, but a list. To access it, I use the semi-private
_get_untagged_response method.

* offlineimap/folder/IMAP.py (IMAPFolder.quickchanged,
  IMAPFolder.cachemessagelist): imaplib2 now explicitly removes its
  EXISTS response on select(), so instead we use the return values from
  select() to get the number of messages.

* offlineimap/imapserver.py (UsefulIMAPMixIn.select): imaplib2 now
  stores untagged_responses for different mailboxes, which confuses us
  because it seems like our mailboxes are "still" in read-only mode when
  we just re-opened them.  Additionally, we have to return the value
  from imaplib2's select() so that the above thing works.

* offlineimap/imapserver.py (UsefulIMAPMixIn._mesg): imaplib2 now
  calls _mesg with the name of a thread, so we display this
  information in debug output. This requires a corresponding change to
  imaplibutil.new_mesg.

* offlineimap/imaplibutil.py: We override IMAP4_SSL.open, whose
  default arguments have changed, so update the default arguments. We
  also subclass imaplib.IMAP4 in a few different places, which now
  relies on having a read_fd file descriptor to poll on.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
f9413226b8 Import imaplib2 instead of imaplib
imaplib2 has slightly different semantics than standard imaplib, so
this patch will break the build, but I thought it was helpful to have it as
a separate commit.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
0a76f0a23d Import newest version of imaplib2
This change does not do anything yet with imaplib2, merely makes it
available for future commits.

This file is identical to the one at
http://sydney.edu.au/engineering/it/~piers/python/imaplib2 .

imaplib2, written by the same guy who wrote imaplib, is very different
from imaplib itself. Calling it a modified version from the standard
distribution is misleading. It's more like a complete rewrite. As
such, it's not really possible to summarize what was changed.

The largest thing is that imaplib2 is "threaded". Instead of doing
blocking writes/reads on the socket during/after every command,
imaplib2 forks off threads to read and write to the socket based on
input and output buffers. This opens the door to asynchronous
commands (every command is potentially asynchronous, according to the
docs), and in particular IDLE, which is by definition an asynchronous
command.

The author writes: "imaplib2 can be substituted for imaplib in
existing clients with no changes in the code", but that's pretty
misleading. It might be true for certain simple users of imaplib, but
for us it's completely false. Among other things, how untagged
responses are stored in-memory is different -- instead of a hash
table, it's a list. I'm guessing this is to preserve order of
responses.

I think there are other miscellaneous improvements, like I think
imaplib2 is IPv6 safe out-of-the-box, but I haven't conducted an
extremely thorough examination of the differences :)

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
e2354fd37c Remove some unneeded imports
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:52 +01:00
Sebastian Spaeth
2946a1ea5d Convert rfc822 module to email module
The rfc822 module has been deprecated since python 2.3, and conversion to
the email module is straightforward, so let us do that. rfc822 is
completely gone in python3.

This also fixes a bug that led to offlineimap abortion (but that code path
is apparently usually not exercised so I did not notice:
rfc822|email.utils.parsedate return a tuple which has no named attributes,
but we were using them later in that function. So pass the tuple into a
struct_time() to get named attributes.

While reading the docs, I noticed that email.parsedate returns invalid
daylight savings information (is_dst attribute), and we are using it
anyway. Oh well, the imap server might think the mails are off by an hour
at worst.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-13 20:30:35 +01:00
Nicolas Sebrecht
30d95ad0ac Merge branch 'ns/uibase-fix-destlist' into next 2011-03-12 13:48:52 +01:00
Nicolas Sebrecht
6887bd995d Merge branch 'ss/remove-stringio-in-threadutil' into next 2011-03-11 20:34:56 +01:00
Sebastian Spaeth
10024731e6 threadutil: Don't require the StringIO module
The only reason we used it here was to do a
traceback.print_exc(StringIO()) to get a string of our traceback. But we
can simply use traceback.format_exc() which exists since python 2.4.

One less module (and it is in the way to python 3 compatability too)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-11 20:34:33 +01:00
Nicolas Sebrecht
90949a4bfc UIBase: fix regression while deletingflags
In commit 7a2a0225 [Don't pass list to ui.adding/deletingflags] we changed the
list logic for a per folder logic but forgot to remove one instance of
"destlist" which isn't valid anymore.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-11 20:20:12 +01:00
Sebastian Spaeth
6c6fdfc769 folder: Implement helper functions uidexists() and getmessageuidlist()
More convenient way to test if a certain uid exists and getting a list
of all uids. Also, the SQL backend will have efficient overrides for
these methods.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-11 18:59:20 +01:00
Sebastian Spaeth
4d352a528a folder: Implement helper function getmessagecount()
Rather than always having to call len(getmessagelist.keys()) as was done
before. No functional change, just nicer looking code. Also the SQLite
backend or other backends could implement more efficient implementations.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-10 20:30:19 +01:00
Nicolas Sebrecht
5048d16913 Merge branch 'ss/declutter-tty-output' into next
Conflicts:
	Changelog.draft.rst
2011-03-10 19:26:26 +01:00
Sebastian Spaeth
efcce01d64 Declutter TTY output
Previously we would output:

Folder sync sspaeth.de[INBOX.INBOX201004]:
 Syncing INBOX.INBOX201004: IMAP -> Maildir
Folder sync sspaeth.de[INBOX.INBOX201006]:
 Syncing INBOX.INBOX201006: IMAP -> Maildir
Folder sync sspaeth.de[INBOX.INBOX201009]:
 Syncing INBOX.INBOX201009: IMAP -> Maildir

which is very repetitive and cluttered. By naming the folder sync
threads just according to the account and not the folder, the output
looks much nicer:

Folder sync [sspaeth.de]:
 Syncing INBOX.INBOX201004: IMAP -> Maildir
 Syncing INBOX.INBOX201006: IMAP -> Maildir
 Syncing INBOX.INBOX201009: IMAP -> Maildir

If syncing multiple accounts in parallel, we will still get headers
indicating the account:

Folder sync [sspaeth.de]:
 Syncing INBOX: IMAP -> Maildir
 Syncing INBOX.INBOX201006: IMAP -> Maildir
Folder sync [gmail]:
 Syncing INBOX: IMAP -> Maildir

This is a small fix that makes the output much nicer in my opinion.

Also don't output the thread name if we are in the MainThread, e.g. when
we output the initial offlineimap banner.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-10 19:04:57 +01:00
Nicolas Sebrecht
7b8d7501d1 Merge branch 'master' into next
Conflicts:
	Changelog.draft.rst
	offlineimap/imapserver.py
2011-03-07 21:55:43 +01:00
Haojun Bao
b94bf79258 fix hang because of infinite loop reading EOF
Read() should return empty string when EOF happen, instead of looping
forever. This is the right semantics of read(), and a wrapped version
should not change it.

If you read the read(2) system call manpage, it tells you that when EOF
is seen, return value is 0; it does not say

	``loop forever when EOF happen''.

After the EOF detection is patched you can see the
following exception:

  WARNING: ERROR attempting to copy message 344 for account Gmail:Traceback (most recent call last):
    File "/usr/lib/pymodules/python2.6/offlineimap/folder/Base.py", line 282, in copymessageto
      message = self.getmessage(uid)
    File "/usr/lib/pymodules/python2.6/offlineimap/folder/IMAP.py", line 216, in getmessage
      initialresult = imapobj.uid('fetch', '%d' % uid, '(BODY.PEEK[])')
    File "/usr/lib/python2.6/imaplib.py", line 753, in uid
      typ, dat = self._simple_command(name, command, *args)
    File "/usr/lib/python2.6/imaplib.py", line 1060, in _simple_command
      return self._command_complete(name, self._command(name, *args))
    File "/usr/lib/python2.6/imaplib.py", line 890, in _command_complete
      raise self.abort('command: %s => %s' % (name, val))
  abort: command: UID => socket error: EOF

Signed-off-by: Bao Haojun <baohaojun@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:54:28 +01:00
Sebastian Spaeth
7a2a02254e Don't pass list to ui.adding/deletingflags
We only have one "dstfolder" at a time when deleting/adding flags, so no
need to pass in a list of those to the ui functions that output the log
info.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:11:10 +01:00
Sebastian Spaeth
387fbf3aaa ui: clean up importment statements
They were not PEP-8 formatted, and some imports were simply
unnecessary. Removed those.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:11:10 +01:00
Sebastian Spaeth
ab1df868c2 String representation of a Folder is its name
This enables us to just use the folder instance in the ui output and get
a name rather than having to call getname() all the time.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:11:10 +01:00
Sebastian Spaeth
3eee821382 Simplify the syncing strategy a bit
The previous syncing strategy was doing more than we needed to and was a
bit underdocumented. This is an attempt to clean it up.

1) Do away with the previous different code paths depending on
whether there is a LocalStatus file or not (the isnewfolder() test). We
always use the same strategy now, which makes the strategy easier to
understand. This strategy is simply:

a) Sync remote to local folder first
b) Sync local to remote

Where each sync implies a 4 pass strategy which does basically the same
as before (explained below).

2) Don't delete messages on LOCAL which don't exist on REMOTE right at
the beginning anymore. This prevented us e.g. from keeping local
messages rather than redownloading everything once LocalStatus got
corrupted or deleted. This surprised many who put in an existing local
maildir and expected it to be synced to the remote place. Instead, the
local maildir was deleted. This is a data loss that actually occured to
people!

3) No need to separately sync the statusfolder, we update that one
simultanously with the destfolders...

3) Simplified the sync function API by only taking one destdir rather
than a list of destdirs, we never used more anyway. This makes the code
easier to read.

4) Added plenty of code comments while I was going through to make sure
the strategy is easy to understand.
-----------------------------------------

 Pass1: Transfer new local messages
        Upload msg with negative/no UIDs to dstfolder. dstfolder should
        assign that message a new UID. Update statusfolder.

 Pass2: Copy existing messages
        Copy messages in self, but not statusfolder to dstfolder if not
        already in dstfolder. Update statusfolder.

 Pass3: Remove deleted messages
        Get all UIDS in statusfolder but not self. These are messages
        that we have locally deleted. Delete those from dstfolder and
        statusfolder.

 Pass4: Synchronize flag changes
        Compare flags in self with those in statusfolder. If msg has a
        valid UID and exists on dstfolder (has not e.g.  been deleted
        there), sync the flag change to dstfolder and statusfolder.

The user visible implications of this change should be unnoticable
except in one situation:
 Blowing away LocalStatus will not require you to redownload ALL of
 your mails if you still have the local Maildir. It will simply recreate
 LocalStatus.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:11:10 +01:00
Nicolas Sebrecht
976c071dd7 Merge branch 'ss/deprecate-current-ui-names' into next 2011-03-06 18:33:29 +01:00
Sebastian Spaeth
fc03475b9e Remove thread Lock() when saving UIDvalidity
Removing this lock makes the function not threadsafe, but then it is
only ever called from one thread, the main account syncer. Also, it
doesn't make it worse than most of the other functions in that class
which are also not threadsafe.

Removing this makes the code simpler, and removes the need to import the
threading module.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-06 18:33:11 +01:00
Nicolas Sebrecht
607eba178a Merge branch 'ss/maxconnections-to-2' into next 2011-03-06 18:31:06 +01:00
Sebastian Spaeth
4e28c7c93f Allow to use nicer UI names
The previous ui names were pretty unwieldy. Is it TTYUI.TTY or
TTY.TTYUI? Do I have to use capitals and where?

Simplify the names by making them case insensitive and by dropping
everything before the dot.

So "Curses.Blinkenlights" can now be invoked as "blinkenlights" or
"BLINKENLIGHTS". The old names will still work just fine so the
transition should be smooth. We issue a warning that the long names are
deprecated.

Document in offlineimap.conf that we don't accept lists of fallback UIs,
but only one UI option (this was already the case before this commit but
still wrongly documented).

The list of accepted ui names is:
  ttyui (default), basic, quiet, machineui, blinkenlights

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-06 15:33:29 +01:00
Sebastian Spaeth
c44e94d64a Set maxconnections default to 2
Multithreading speeds up account syncing a lot and the offlineimap
defaults are very conservative. Let's make it use 2 IMAP connections by
default to gain some of the benefits that offlineimap offers.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-06 15:27:38 +01:00
Sebastian Spaeth
3b8e1f91cd Implement APPENDUID support
Rather than inserting our own home-grown header, everytime we save a
message to an IMAP server, we check if we suport the UIDPLUS extension
which provides us with an APPENDUID reply. Use that to find the new UID
if possible, but keep the old way if we don't have that extension.

If a folder is read-only, return the uid that we have passed in per API
description in folder.Base.py

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
96db608e4d Remove convoluted assert statements
The working horse of the savemessage() function, imaplib.append() was
hidden away in an assert statement. Pull the real functions out of the
asserts and simply assert on the return values. This looks less
convoluted and makes this easier to understand in my opinion.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
d22c762385 Factor out the date guessing/retrieving
savemessage was too long and complex. Factor out the date guessing part
of the function and put it into a function of its own. The logic of the
date guessing is the same, however, we do not use the
imaplib.Time2InternalDate() function as it is buggy
(http://bugs.python.org/issue11024) and returns localized patches. So we
create INTERNALDATE ourselves and pass it to append() as a string.

This commit fixes a bug that international users used to pass an invalid
date to the IMAP server, which the server will either ignore or complain
about.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
419f27418e Simplify & document savemessage_getnewheader
savemessage_getnewheader was an undocmented, cryptic and overengineered
function. It generates a new unique value that can be used as a mail
header to be inserted. For this it used LOTS of randomness sources: hash
of the mail content, hash of the folder name, hash of the repository
name, the current time, a random() value, and the offlineimap version string.
All we need is something random. So reduce this to hash of content
appended by a random integer. Sufficient and somewhat faster to calculate.

Rename the function to actually describe accurately what it does or
would you have guessed that savemessage_getnewheader() did nothing more
than returning ('X-OfflineIMAP', <randomstring> )? Rename to
generate_randomheader() to make it clearer what this is all about.

Also document the function, describing what it does, and what it returns.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:09 +01:00
Nicolas Sebrecht
a271f90271 Merge branch 'ss/remove-uneeded-darwin-workaround' into next 2011-03-05 16:37:35 +01:00
Sebastian Spaeth
ab3900e479 Remove unneeded workaround for Darwin
There is a clumsy workaround for Darwin that chunks reads into 8kb
blocks to avoid huge memory allocations. First, this fix should not only
be required but on FreeBSD2.6 too (see
http://bugs.python.org/issue3531). Second, decent python versions (I
checked 2.6) already chunk in the SSL case anyway, so there is no need to do
that again. Remove that level of indirection.

http://evanjones.ca/python-memory.html claims that this problem has been
fixed since python 2.5, so we might consider removing the workaround
completely even for the non-SSL case.

Increase the chunk size on Mac from 8kb to 64kb. Even Macs should be
able to take that amount of memory usage nowadays.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 16:37:00 +01:00
Nicolas Sebrecht
4e0aaa07c0 Merge branch 'master' into next 2011-03-03 19:09:30 +01:00
Sebastian Spaeth
2ab51e6855 Make profiling mode really enforce singlethreading
A typo was preventing profiling mode to really enable singlethreading
mode. Fixing the unfortunate typo of mine makes it work.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-03 19:08:49 +01:00
Nicolas Sebrecht
c9ac254c8c Merge branch 'ss/debug-folderfilter' into next 2011-03-03 19:07:35 +01:00