Commit Graph

636 Commits

Author SHA1 Message Date
Sebastian Spaeth
f3249e0856 True 1-way sync (backup)
This commit enables true 1-way syncing between repositories. This has
often been demanded for backup purposes when you do not want to cause
accidental modifications of your backup that would be propagated to the
other side.

This has been implemented by allowing to configure a Repository as
'readonly' to forbid any modification on it.

'readonly' applies to all the type of repositories.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 20:41:36 +02:00
Sebastian Spaeth
3d4dc11a8b Introduce an *empty* debug type
This debug type will always be enabled whenever any debugging is enables
and it outputs debug messages that cannot be categorized among any of
imap, maildir (e.g. things that concern the sync logic).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 20:41:36 +02:00
Sebastian Spaeth
6add201436 Improve the developer API documentation
Improve the code documentation (still much more to do) and also add some
more meat to the structure of the developer documentation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 20:16:45 +02:00
Nicolas Sebrecht
e9a7afda6d Merge branch 'ss/corrupted-uidmap-file' into next
Conflicts:
	Changelog.draft.rst
2011-05-02 19:09:52 +02:00
Sebastian Spaeth
2ed1c357a0 More detailed error output on corrupt UID mapping files
This function will need much more "robustifying", but the very least we
can do is to print the file name and line that are giving trouble.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 19:08:59 +02:00
Nicolas Sebrecht
cda932351c Merge branch 'ss/folder-name-whitspaces' into next
Conflicts:
	Changelog.draft.rst
2011-04-28 18:29:14 +02:00
Dan Christensen
9bd7a21f16 Don't strip whitespace in the -f option
Allow leading and trailing spaces in folder names specified on the
command line.

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Dan Christensen <jdc@uwo.ca>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-28 18:28:23 +02:00
Sebastian Spaeth
163f1eefc4 Drop all connections when a sync failed
in accounts.sync() we would holdordropconnections() after each sync. But
depending on the repository configuration that might imply that
offlineimap tries to keep the same connections. But when a sync failed,
e.g. after a user had his computer suspended, it might be that our
connections that we have are worthless. So definitely drop them after a
failed sync.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 23:04:24 +02:00
Nicolas Sebrecht
0bc1c9583e Merge branch 'ss/folder-cleanup-getmessage' into next
Conflicts:
	offlineimap/folder/IMAP.py
2011-04-27 22:52:17 +02:00
Sebastian Spaeth
1ff628bd5c folder/IMAP: cleanup getmessage()
Add some comments how the data structures actually look like.
Describe the function properly, and make sure we only hold on to the
data connection as quickly as possible.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 22:49:18 +02:00
Sebastian Spaeth
c8f80bc6d2 We had been setting this variable twice
This was in the code for a very long time, it seems.
Remove one instance, no functional changes.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 22:35:39 +02:00
Sebastian Spaeth
e01968973d Use "%s" % account rather than account.name
uncritical patch, but we can make the code a bit shorter so why not.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:10 +02:00
Sebastian Spaeth
3e5d6f5450 Make str() of Repository() be its name
So we can simply hand a repository instance to error messages rather than
having to call Repository().getname() all the time.

This is not used yet.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:10 +02:00
Sebastian Spaeth
074ea14cf1 Make str() of Account() be its name
So we can simply hand an account instance to error messages rather than
having to call Account().getname() all the time.

This is not used yet.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:10 +02:00
Sebastian Spaeth
bb8546e846 Whitespace fixes
Just clean up lines which end in whitespaces.
Also adapt the copyright to the current year while touching the file.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:09 +02:00
Nicolas Sebrecht
b2be0c3697 Merge branch 'ss/sync-remove-neguid-pass' into next 2011-04-25 13:01:59 +02:00
Sebastian Spaeth
e20d8b9679 Remove upload neguid pass from sync logic
In order to optimize performance, we fold the 1st and 2nd pass of our
sync strategy into one. They were essentially doing the same thing:
uploading a message to the other side. The only difference was that in
one case we have a negative UID locally, and in the other case, we have
a positive one already.

This saves some time, as we don't have to run through that function on
IMAP servers anyway (they always have positive UIDs), and 2nd were we
stalling further copying until phase 1 was finished. So uploading a
single new message would prevent us from starting to copy existing
regular messages.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-25 13:01:53 +02:00
Sebastian Spaeth
36eb37b47d IMAP: reduce quickchanged() checks
For each folder we were making a second IMAP request asking for the
latest UID and compared that with the highest UID in our
statusfolder. This catched the case that 1 mail has been deleted by
someone else and another one has arrived since we checked, so that the
total number of mails appears to not have changed.

We don't capture anymore this case in the quickchanged() case.

It improves my performance from 8 to about 7.5 seconds per check (with lots of
variation) and we would benefit even more in the IMAP<->IMAP case as we do one
additional IMAP lookup per folder on each side then.

Do cleanups on whitespaces while in this file.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-25 12:58:47 +02:00
Nicolas Sebrecht
3b09695236 v6.3.3
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-21 18:34:24 +02:00
Nicolas Sebrecht
bf1c9e3233 Merge branch 'ss/maildir-quickchanged-cleanup' into next 2011-04-15 19:52:35 +02:00
Ethan Glasser-Camp
9e734006f6 Fix IMAP4_Tunnel to work with imaplib2
* IMAP4_Tunnel constructor should support base-class arguments, in
  order to support the timeout argument.

* IMAP4_Tunnel needs to store the member IMAP4.host, which is normally
  done in IMAP4.open().

* Update IMAP4_Tunnel.read() and IMAP4_Tunnel.send().  We turn on
  nonblocking mode for these sockets, so we can return immediately
  with whatever data is available.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-13 18:41:53 +02:00
Sebastian Spaeth
fdf22400b1 imaplib2: Bump from 2.20 to 2.22
This contains a fixed Time2InternalDate function and a more robust
socket connection, trying twice and raising an error only when that
fails (I believe). The actual code changes are rather minor.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-11 19:10:45 +02:00
Sebastian Spaeth
e37441cd19 folder/Maildir: Make use of helper functions
quickchanged() was iterating a lot, make use of some of the helper
functions that had been introduced recently and document the function a
bit better. No functional change.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-11 18:57:25 +02:00
Sebastian Spaeth
d762175af4 Make -f option with with folder names with spaces.
Previously ALL spaces had been stripped off. Now, only strip spaces
around the comma, so -f "INBOX, Deleted Mails" will work. You will still
need to quote or escape spaces so the shell hand the list as one command
line argument to offlineimap.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-11 18:29:44 +02:00
Sebastian Spaeth
58220fd8e7 Replace calls to getmessagelist() to alternatives
getmessagelist() is slow for the mapped UID case, so replace some of its
occurences with calls that are optimized for this case, ie
getmessagecount() and uidexists().

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-and-tested-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 17:53:14 +02:00
Sebastian Spaeth
ca012d3a81 Implement more efficient functions for the MappedUID case
We are calling getmessagelist() internally a lot, e.g. just to check if
a UID exists (from uidexist()). This is a very expensive operation in
the UIDMapped case, as we reconstruct the whole messagelist dict every
single time, involving lots of copying etc.

So we provide more efficient implementations for the uidexists()
getmessageuidlist() and getmessagecount() functions that are fast in the
UIDMapped case. This should solve the performance regression that was
recently observed in the Mapped UID case.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-and-tested-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 17:52:22 +02:00
Nicolas Sebrecht
0f85e592a3 Merge branch 'df/fix-debug-msg' into next 2011-03-28 16:50:18 +02:00
David Favro
954655b7ec Fixed bug: wrong number of arguments to debug() [IMAP.py].
Signed-off-by: David Favro <offlineimap@meta-dynamic.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 16:49:21 +02:00
Nicolas Sebrecht
71d0828cd6 Merge branch 'master' into next 2011-03-25 18:42:46 +01:00
Sebastian Spaeth
09515f8f90 Increase compatability with Gmail
When uploading a new message to Gmail we need to find out the UID it
assigned it, but Gmail does not advertize the UIDPLUS extension (in all
cases) and it fails to find the email that we just uploaded when
searching for it. This prevented us effectively from uploading to
gmail.

See analysis in
http://lists.alioth.debian.org/pipermail/offlineimap-project/2011-March/001449.html
for details on what is going wrong.

This patch increases compatability with Gmail by checking for APPENDUID
responses to an APPEND action even if the server did not claim to
support it. This restores the capability to upload messages to the
*broken* Gmail IMAP implementation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-03-25 18:42:16 +01:00
Ethan Glasser-Camp
105da1b0c3 Always logout() on imaplib2 objects, even during exceptions
Without this, trying to Ctrl-C out of offlineimap will go into a hang.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-25 18:26:22 +01:00
Nicolas Sebrecht
756eb72494 Merge branch 'master' into next 2011-03-24 19:08:30 +01:00
Nicolas Sebrecht
51b89e6c43 Merge branch 'maint'
Conflicts:
	Changelog.draft.rst
	Makefile
2011-03-24 19:07:34 +01:00
Sebastian Spaeth
65faec834f accounts.py: Print the stacktrace via traceback module
All other instances were converted to format crash output including a
stacktrace, but this one seems to have been left out. Make Exceptions
print their stacktrace here too.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-24 19:03:16 +01:00
Vincent Beffara
bb48b6deaf Get rid of the UsefulIMAP4 classes
The three classes with names starting with UsefulIMAP4 were used for two
purposes, to include the UsefulIMAPMixIn class and to implement various
system-specific kludges. None of these kludges remain, so it is cleaner
to include UsefulIMAPMixIn directly in imaplibutil and forget about them
for good.

Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-23 20:00:35 +01:00
Vincent Beffara
9f03f41b70 Move UsefulIMAPMixIn to imaplibutil.py
Now that we do not need any system-specific hack, the three UsefulIMAP4
classes have become empty and should be removed. This implies importing
UsefulIMAPMixIn directly from the classes defined in imaplibutil.

Prepare this change by moving the code into imaplibutil.py. Functionally
this is a no-op.

Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-23 20:00:30 +01:00
Vincent Beffara
9a277cfd02 Remove a darwin-specific workaround for read()
Because of a buggy realloc() implementation in earlier versions of
Python on Mac OS X, we had to cut reads into manageable chunks by hand;
this is no more needed with Python 2.6, and besides it causes problems
with imaplib2, which we now use. Revert the special case to use the
system's read() instead, which is now safe.

Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-23 20:00:23 +01:00
Nicolas Sebrecht
37d0fe8b01 v6.3.2.1
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 20:46:18 +01:00
Nicolas Sebrecht
3eba44f636 Merge branch 'ss/uidmaps' into next 2011-03-16 18:05:35 +01:00
Sebastian Spaeth
fd28c5a2d3 folder/IMAP: savemessage() should just save flags if uid already exists
As the LocalStatus and UIDMap backend already did: If the uid already
exists for savemessage(), only modify the flags and don't append a new
message.

We don't invoke savemessage() on messages that already exist in our sync
logic, so this has no change on our current behavior. But it makes
backends befave more consistent with each other.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 18:05:16 +01:00
Sebastian Spaeth
dc3ad723c9 Give some love to UIDMaps
- Some documentation improvements, this is a severely underdocumented
  class. This still needs some further improvements though.

- Don't use apply(Baseclass) (which is going away in Python 3), use
  IMAPFolder.__init__(self, *args, **kwargs).

- Don't call ValueError, string. It is ValueError(string)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 18:05:16 +01:00
Nicolas Sebrecht
3b45782cc4 Merge branch 'master' into next 2011-03-15 19:28:02 +01:00
Nicolas Sebrecht
180e86007a Merge branch 'maint' 2011-03-15 18:46:26 +01:00
Sebastian Spaeth
1c71e37f8f Sanity checks for SSL cacertfile configuration
We were not able to handle ~/... type of path configurations and we
crashed with mysterious SSL errors when no file was found at the
configured location. Expand '~' and bomb out with usable error messages
in case such a file does not exist. This will still not protect against
corrupt cacert files but it goes a long way towards user friendliness.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-15 18:45:31 +01:00
Nicolas Sebrecht
44eefae043 cleanup import statements and conform to PEP-8
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 21:35:33 +01:00
Ethan Glasser-Camp
5ea95002f5 Create new connections with a timeout
imaplib2 does not use socket, so does not know about the
defaulttimeout we set based on the config. Instead, we explicitly pass
the default timeout.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
a139d9deed Throw away broken connections
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
197030da1a Remove obsolete read(), readline(), _read_upto() methods
For read(), the imaplib2 version seems to work perfectly well. The
others aren't used any more, either by imaplib2, nor by us, so we may
as well get rid of them.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
1bf4bee5e6 Update to match semantics of new imaplib2
The biggest change here is that imapobj.untagged_responses is no
longer a dictionary, but a list. To access it, I use the semi-private
_get_untagged_response method.

* offlineimap/folder/IMAP.py (IMAPFolder.quickchanged,
  IMAPFolder.cachemessagelist): imaplib2 now explicitly removes its
  EXISTS response on select(), so instead we use the return values from
  select() to get the number of messages.

* offlineimap/imapserver.py (UsefulIMAPMixIn.select): imaplib2 now
  stores untagged_responses for different mailboxes, which confuses us
  because it seems like our mailboxes are "still" in read-only mode when
  we just re-opened them.  Additionally, we have to return the value
  from imaplib2's select() so that the above thing works.

* offlineimap/imapserver.py (UsefulIMAPMixIn._mesg): imaplib2 now
  calls _mesg with the name of a thread, so we display this
  information in debug output. This requires a corresponding change to
  imaplibutil.new_mesg.

* offlineimap/imaplibutil.py: We override IMAP4_SSL.open, whose
  default arguments have changed, so update the default arguments. We
  also subclass imaplib.IMAP4 in a few different places, which now
  relies on having a read_fd file descriptor to poll on.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
f9413226b8 Import imaplib2 instead of imaplib
imaplib2 has slightly different semantics than standard imaplib, so
this patch will break the build, but I thought it was helpful to have it as
a separate commit.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00