Commit Graph

97 Commits

Author SHA1 Message Date
Sebastian Spaeth
e20d8b9679 Remove upload neguid pass from sync logic
In order to optimize performance, we fold the 1st and 2nd pass of our
sync strategy into one. They were essentially doing the same thing:
uploading a message to the other side. The only difference was that in
one case we have a negative UID locally, and in the other case, we have
a positive one already.

This saves some time, as we don't have to run through that function on
IMAP servers anyway (they always have positive UIDs), and 2nd were we
stalling further copying until phase 1 was finished. So uploading a
single new message would prevent us from starting to copy existing
regular messages.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-25 13:01:53 +02:00
Sebastian Spaeth
e37441cd19 folder/Maildir: Make use of helper functions
quickchanged() was iterating a lot, make use of some of the helper
functions that had been introduced recently and document the function a
bit better. No functional change.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-11 18:57:25 +02:00
Sebastian Spaeth
58220fd8e7 Replace calls to getmessagelist() to alternatives
getmessagelist() is slow for the mapped UID case, so replace some of its
occurences with calls that are optimized for this case, ie
getmessagecount() and uidexists().

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-and-tested-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 17:53:14 +02:00
Sebastian Spaeth
ca012d3a81 Implement more efficient functions for the MappedUID case
We are calling getmessagelist() internally a lot, e.g. just to check if
a UID exists (from uidexist()). This is a very expensive operation in
the UIDMapped case, as we reconstruct the whole messagelist dict every
single time, involving lots of copying etc.

So we provide more efficient implementations for the uidexists()
getmessageuidlist() and getmessagecount() functions that are fast in the
UIDMapped case. This should solve the performance regression that was
recently observed in the Mapped UID case.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-and-tested-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 17:52:22 +02:00
David Favro
954655b7ec Fixed bug: wrong number of arguments to debug() [IMAP.py].
Signed-off-by: David Favro <offlineimap@meta-dynamic.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-28 16:49:21 +02:00
Sebastian Spaeth
09515f8f90 Increase compatability with Gmail
When uploading a new message to Gmail we need to find out the UID it
assigned it, but Gmail does not advertize the UIDPLUS extension (in all
cases) and it fails to find the email that we just uploaded when
searching for it. This prevented us effectively from uploading to
gmail.

See analysis in
http://lists.alioth.debian.org/pipermail/offlineimap-project/2011-March/001449.html
for details on what is going wrong.

This patch increases compatability with Gmail by checking for APPENDUID
responses to an APPEND action even if the server did not claim to
support it. This restores the capability to upload messages to the
*broken* Gmail IMAP implementation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-03-25 18:42:16 +01:00
Nicolas Sebrecht
3eba44f636 Merge branch 'ss/uidmaps' into next 2011-03-16 18:05:35 +01:00
Sebastian Spaeth
fd28c5a2d3 folder/IMAP: savemessage() should just save flags if uid already exists
As the LocalStatus and UIDMap backend already did: If the uid already
exists for savemessage(), only modify the flags and don't append a new
message.

We don't invoke savemessage() on messages that already exist in our sync
logic, so this has no change on our current behavior. But it makes
backends befave more consistent with each other.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 18:05:16 +01:00
Sebastian Spaeth
dc3ad723c9 Give some love to UIDMaps
- Some documentation improvements, this is a severely underdocumented
  class. This still needs some further improvements though.

- Don't use apply(Baseclass) (which is going away in Python 3), use
  IMAPFolder.__init__(self, *args, **kwargs).

- Don't call ValueError, string. It is ValueError(string)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-16 18:05:16 +01:00
Nicolas Sebrecht
44eefae043 cleanup import statements and conform to PEP-8
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 21:35:33 +01:00
Ethan Glasser-Camp
1bf4bee5e6 Update to match semantics of new imaplib2
The biggest change here is that imapobj.untagged_responses is no
longer a dictionary, but a list. To access it, I use the semi-private
_get_untagged_response method.

* offlineimap/folder/IMAP.py (IMAPFolder.quickchanged,
  IMAPFolder.cachemessagelist): imaplib2 now explicitly removes its
  EXISTS response on select(), so instead we use the return values from
  select() to get the number of messages.

* offlineimap/imapserver.py (UsefulIMAPMixIn.select): imaplib2 now
  stores untagged_responses for different mailboxes, which confuses us
  because it seems like our mailboxes are "still" in read-only mode when
  we just re-opened them.  Additionally, we have to return the value
  from imaplib2's select() so that the above thing works.

* offlineimap/imapserver.py (UsefulIMAPMixIn._mesg): imaplib2 now
  calls _mesg with the name of a thread, so we display this
  information in debug output. This requires a corresponding change to
  imaplibutil.new_mesg.

* offlineimap/imaplibutil.py: We override IMAP4_SSL.open, whose
  default arguments have changed, so update the default arguments. We
  also subclass imaplib.IMAP4 in a few different places, which now
  relies on having a read_fd file descriptor to poll on.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
f9413226b8 Import imaplib2 instead of imaplib
imaplib2 has slightly different semantics than standard imaplib, so
this patch will break the build, but I thought it was helpful to have it as
a separate commit.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:56 +01:00
Ethan Glasser-Camp
e2354fd37c Remove some unneeded imports
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-14 19:51:52 +01:00
Sebastian Spaeth
2946a1ea5d Convert rfc822 module to email module
The rfc822 module has been deprecated since python 2.3, and conversion to
the email module is straightforward, so let us do that. rfc822 is
completely gone in python3.

This also fixes a bug that led to offlineimap abortion (but that code path
is apparently usually not exercised so I did not notice:
rfc822|email.utils.parsedate return a tuple which has no named attributes,
but we were using them later in that function. So pass the tuple into a
struct_time() to get named attributes.

While reading the docs, I noticed that email.parsedate returns invalid
daylight savings information (is_dst attribute), and we are using it
anyway. Oh well, the imap server might think the mails are off by an hour
at worst.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-13 20:30:35 +01:00
Sebastian Spaeth
6c6fdfc769 folder: Implement helper functions uidexists() and getmessageuidlist()
More convenient way to test if a certain uid exists and getting a list
of all uids. Also, the SQL backend will have efficient overrides for
these methods.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-11 18:59:20 +01:00
Sebastian Spaeth
4d352a528a folder: Implement helper function getmessagecount()
Rather than always having to call len(getmessagelist.keys()) as was done
before. No functional change, just nicer looking code. Also the SQLite
backend or other backends could implement more efficient implementations.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-10 20:30:19 +01:00
Sebastian Spaeth
7a2a02254e Don't pass list to ui.adding/deletingflags
We only have one "dstfolder" at a time when deleting/adding flags, so no
need to pass in a list of those to the ui functions that output the log
info.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:11:10 +01:00
Sebastian Spaeth
ab1df868c2 String representation of a Folder is its name
This enables us to just use the folder instance in the ui output and get
a name rather than having to call getname() all the time.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:11:10 +01:00
Sebastian Spaeth
3eee821382 Simplify the syncing strategy a bit
The previous syncing strategy was doing more than we needed to and was a
bit underdocumented. This is an attempt to clean it up.

1) Do away with the previous different code paths depending on
whether there is a LocalStatus file or not (the isnewfolder() test). We
always use the same strategy now, which makes the strategy easier to
understand. This strategy is simply:

a) Sync remote to local folder first
b) Sync local to remote

Where each sync implies a 4 pass strategy which does basically the same
as before (explained below).

2) Don't delete messages on LOCAL which don't exist on REMOTE right at
the beginning anymore. This prevented us e.g. from keeping local
messages rather than redownloading everything once LocalStatus got
corrupted or deleted. This surprised many who put in an existing local
maildir and expected it to be synced to the remote place. Instead, the
local maildir was deleted. This is a data loss that actually occured to
people!

3) No need to separately sync the statusfolder, we update that one
simultanously with the destfolders...

3) Simplified the sync function API by only taking one destdir rather
than a list of destdirs, we never used more anyway. This makes the code
easier to read.

4) Added plenty of code comments while I was going through to make sure
the strategy is easy to understand.
-----------------------------------------

 Pass1: Transfer new local messages
        Upload msg with negative/no UIDs to dstfolder. dstfolder should
        assign that message a new UID. Update statusfolder.

 Pass2: Copy existing messages
        Copy messages in self, but not statusfolder to dstfolder if not
        already in dstfolder. Update statusfolder.

 Pass3: Remove deleted messages
        Get all UIDS in statusfolder but not self. These are messages
        that we have locally deleted. Delete those from dstfolder and
        statusfolder.

 Pass4: Synchronize flag changes
        Compare flags in self with those in statusfolder. If msg has a
        valid UID and exists on dstfolder (has not e.g.  been deleted
        there), sync the flag change to dstfolder and statusfolder.

The user visible implications of this change should be unnoticable
except in one situation:
 Blowing away LocalStatus will not require you to redownload ALL of
 your mails if you still have the local Maildir. It will simply recreate
 LocalStatus.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-07 21:11:10 +01:00
Sebastian Spaeth
fc03475b9e Remove thread Lock() when saving UIDvalidity
Removing this lock makes the function not threadsafe, but then it is
only ever called from one thread, the main account syncer. Also, it
doesn't make it worse than most of the other functions in that class
which are also not threadsafe.

Removing this makes the code simpler, and removes the need to import the
threading module.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-06 18:33:11 +01:00
Sebastian Spaeth
3b8e1f91cd Implement APPENDUID support
Rather than inserting our own home-grown header, everytime we save a
message to an IMAP server, we check if we suport the UIDPLUS extension
which provides us with an APPENDUID reply. Use that to find the new UID
if possible, but keep the old way if we don't have that extension.

If a folder is read-only, return the uid that we have passed in per API
description in folder.Base.py

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
96db608e4d Remove convoluted assert statements
The working horse of the savemessage() function, imaplib.append() was
hidden away in an assert statement. Pull the real functions out of the
asserts and simply assert on the return values. This looks less
convoluted and makes this easier to understand in my opinion.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
d22c762385 Factor out the date guessing/retrieving
savemessage was too long and complex. Factor out the date guessing part
of the function and put it into a function of its own. The logic of the
date guessing is the same, however, we do not use the
imaplib.Time2InternalDate() function as it is buggy
(http://bugs.python.org/issue11024) and returns localized patches. So we
create INTERNALDATE ourselves and pass it to append() as a string.

This commit fixes a bug that international users used to pass an invalid
date to the IMAP server, which the server will either ignore or complain
about.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:10 +01:00
Sebastian Spaeth
419f27418e Simplify & document savemessage_getnewheader
savemessage_getnewheader was an undocmented, cryptic and overengineered
function. It generates a new unique value that can be used as a mail
header to be inserted. For this it used LOTS of randomness sources: hash
of the mail content, hash of the folder name, hash of the repository
name, the current time, a random() value, and the offlineimap version string.
All we need is something random. So reduce this to hash of content
appended by a random integer. Sufficient and somewhat faster to calculate.

Rename the function to actually describe accurately what it does or
would you have guessed that savemessage_getnewheader() did nothing more
than returning ('X-OfflineIMAP', <randomstring> )? Rename to
generate_randomheader() to make it clearer what this is all about.

Also document the function, describing what it does, and what it returns.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-03-05 18:02:09 +01:00
Sebastian Spaeth
6398bf7795 Don't require the string module
There is no need for using the string module if all we want is to split
a string at the white space. All pythons since at least 2.4 can do that.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-02-21 13:20:21 +01:00
Sebastian Spaeth1
074cd11418 Use self.ui rather than UIBase.getglobalui()
We have vonverted all places in folder/* to have self.ui available,
rather than having to use UIBase.getglobalui() all the
time. Unfortunately, we did not convert the users in folder/Base.py.
This patch does it belatedly. This fixes
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=613483

Signed-off-by: Sebastian Spaeth1 <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-02-17 19:03:29 +01:00
Sebastian
997dc1f510 Remove unneeded imports of UIBase
Code was broken, as these backends import UIBase (which had been moved).
However, they don't use it, so we can just delete the import.
Sorry, I failed to find those earlier somehow.

Signed-off-by: Sebastian <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-02-09 19:29:01 +01:00
Nicolas Sebrecht
84b7c686ec Merge branch 'master' into next
Conflicts:
	Changelog.draft.rst
2011-01-28 19:48:35 +01:00
Edward Z. Yang
e506442996 Better trace information when an exception is caught.
Signed-off-by: Edward Z. Yang <ezyang@mit.edu>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-28 19:35:21 +01:00
Nicolas Sebrecht
9afb432053 folder/IMAP.py: accord __version__ with import
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-27 19:45:14 +01:00
Sebastian Spaeth
b47cc1d276 import getglobalui as we use it
This branch is currently broken, as we were using getglobalui() but it
was not imported. This fixes the missing import and makes offlineimap
run successfully.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-21 20:10:18 +01:00
Nicolas Sebrecht
a9c63078a2 Merge branch 'master' into next 2011-01-20 20:12:02 +01:00
Knut Anders Hatlen
9ad1810e43 understand multiple EXISTS replies from servers like Zimbra
Lars Thalmann <web@larsthalmann.com> writes:

> It seems some servers (Zimbra?) respond to imap SELECT requests with
> multiple EXISTS lines:
>
> ? SELECT INBOX
> * 500 EXISTS
> * 0 RECENT
> * 1000 EXISTS
> * 0 RECENT
> * 1500 EXISTS
> * 0 RECENT
> * 1567 EXISTS
> * 0 RECENT

Signed-off-by: Knut Anders Hatlen <kahatlen@gmail.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-20 20:06:46 +01:00
Sebastian Spaeth
760698253b LocalStatus: Don't ignore Exceptions on os.fsync
Earlier we would ignore *ALL* Exceptions that could occur during the
fsyncing of our LocalStatus database. Ignoring all Exceptions is not the
right thing here though. A recent commit improved the situation by
raising at least KeyboardInterrupt Exceptions, but that is still not
optimal.

os.fsync() is available on Unix, and Windows starting in python
2.2.3. so it should always work. If it doesn't, something is wrong.

It has been suggested to only catch EnvironmentError (ie SystemError and
OSError) here, but even those should be thrown. Something *is* wrong if
this fails and we should not ignore it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-14 19:25:11 +01:00
Sebastian Spaeth
1a1e68d8be Catch KeyboardInterrupt exceptions explicitely
Previously we did not catch KeyboardInterrupts explicitly as all of the
code was executed in forked child threads which would never receive
Ctrl-c exceptions. With the upcoming single threaded modus, this code
can be run in the main thread however, so we need to take care of
KeyboardInterrupts explicitly.

This was done wherever we would catch *ALL* exceptions universally and
print out an error message.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-14 19:25:11 +01:00
Nicolas Sebrecht
3446aa9368 Merge branch 'ss/folder/base_clean-threadutil-import' into next
Conflicts:
	offlineimap/folder/Base.py
2011-01-10 18:58:05 +01:00
Sebastian Spaeth
a6b2b8014f folder/Base.py: No need to import InstanceLimitedThread explicitely
We already import threadutil, so no need to also import
threadutil.InstanceLimitedThread separately.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-10 18:56:39 +01:00
Nicolas Sebrecht
ee8a00cce2 Merge branch 'ss/ui' into next
Conflicts:
	offlineimap/folder/IMAP.py

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-05 19:31:08 +01:00
Sebastian Spaeth
de618d84f3 Save ui as self.ui in folder/Base class
Rather than regetting the ui with UIBase.getglobalui() all the time, we get it once in the base class and let all derivative classes just make use of self.ui rather than refetching the ui all the time, this makes for a bit less code and shorter lines.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-01-05 19:24:00 +01:00
Sebastian Spaeth
0b5b38d298 Define version constants etc in __init__.py
Move central constant definitions into __init__.py.  This does away
with version.py which contained nothing else and __init__.py is where
things like __VERSION__ are usually defined.

This commit also changes code to use offlineimap.__version__ rather
than offlineimap.version.__version__ as was before. Cleaned up some
duplicate or unneeded imports while touching those, formatting import
statements per PEP8 (one import per row).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2010-12-22 19:45:01 +01:00
Johannes Stezenbach
573d7ed52a Change X-OfflineIMAP header format
Change the X-OfflineIMAP header to work around possible Exchange MAPI
table overflow problem described in
http://article.gmane.org/gmane.mail.imap.offlineimap.general/1699
(It is unknown whether this problem still exits in current
Exchange versions, but let's assume the worst.)

The X-OfflineIMAP header is neccessary with some IMAP servers to
reliably determine the UID of a new messages uploaded to the server
by using the "UID SEARCH HEADER name string" command.  Since this
command compares header name and value it is sufficient to have
a unique header value and a non-unique header name.

Note that a message can have more than one X-OfflineIMAP header if
the message was copied between IMAP folders multiple times.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2010-12-09 19:52:48 +01:00
Michal Schmidt
caa7d8a128 much faster deleting of messages from LocalStatus
After tens of thousands of messages on the IMAP server were deleted it
takes offlineimap extremely long time (several hours of high CPU usage)
to delete them locally. It spends almost all the time modifying
LocalStatus. It processes the messages one by one, rewriting the
folder's status file in LocalStatus after each message.

It is much more efficient to save the status file only once, after
removing all the messages from the messagelist.

Deleting lots of messages now takes seconds instead of hours.

This should solve Debian bug #518093:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=518093

Signed-off-by: Michal Schmidt <mschmidt@redhat.com>
2010-03-31 21:32:51 -05:00
Mike Dawson
30344587d9 Patch for error handling / separation of accounts etc.
Dear All,
I have made the attached patch to try and make offlineimap a bit more
stable in challenging situations.  It's extremely useful in slow
connection environments - but sometimes if one account had the wrong
password or the connection went down then unfortunately the whole
program would crash.

I have tested this on our connection and tried throwing at it just about
every situation - connection, up down, up, down again, change password,
error whilst copying one message, etc.  I have been running this patch
for the last 5 days or so syncing 6 accounts at the moment...  It seems
to work and stay alive nicely (even if your connection does not)...

Hope that this can go in for the next release... Please let me know if
anyone notices any problems with this...

Regards,

-Mike

-- Attached file included as plaintext by Ecartis --
-- File: submit

From 1d6777cab23637eb830031c7cab0ae9b8589afd6 Mon Sep 17 00:00:00 2001
From: mike <mike@mikelaptop.(none)>
Date: Mon, 24 Aug 2009 19:37:59 +0430
Subject: [PATCH] This patch attempts to introduce a little more error handling - e.g.
 if one account has an error because of a changed password or something
 that should not affect the other accounts.

Specifically:
If one sync run has an issue this is in a try-except clause - if it
has an auto refresh period the thread will sleep and try again - this
could be quite useful in the event of the connection going down for a
little while, changed password etc.

If one folder cannot be created an error message will be displayed through
the UI and the program will continue (e.g. permission denied to create a folder)

If one message does not want to copy for whatever resaon an error message
will be displayed through the UI and at least the other messages will
be copied

If one folder run has an exception then the others will still run
2009-08-28 00:01:10 -05:00
John Goerzen
1148df7685 Merge branch 'maxage'
Conflicts:
	offlineimap/folder/IMAP.py
2009-08-16 09:29:00 -05:00
Mike Dawson
cde94e5047 Patch for maxage and maxsize options
Dear All,
Attached is the patch that I have developed to provide maxage and
maxsize options.  You can thus sync only the last x days of messages and
exclude large messages.  All details in the attached git file.

Regards,

-Mike

-- Attached file included as plaintext by Ecartis --
-- File: submit

From 04fead2b46a79675a5b29de6f2b4088b9c9448e5 Mon Sep 17 00:00:00 2001
From: mike <mike@mikelaptop.(none)>
Date: Sun, 16 Aug 2009 17:00:49 +0430
Subject: [PATCH] Patch to provide maxage and maxsize account options to exclude old/large messages

This is designed to make offlineimap even better for low bandwidth connections.

maxage allows you to specify a number of days and only messages within
that range will be considered by offlineimap for the sync.  This can be
useful if you would like to start using offlineimap with a large
existing account and do not want to import large archives of mail.

maxsize allows you to specify the maximum size of a message to consider so
that you can exclude messages with large attachments etc.

In both cases the cachemessagelist function of the folder was modified to ignore
messages that do not meet the criteria.  If the criteria are not specified
then the existing code will be executed the same as before.  If a message
does not meet the criteria it will be as though this message does not exist
- offlineimap will completely ignore it.  It will not have flags updated,
it will not be deleted, it will not be considered at all.

When operating against an IMAP repository a server side search function
is used.  This of course requires support for server side search.

I have tested this with either option, no options etc. against IMAP, Maildir
and Gmail.  I have run variations of this patch here for the last 3 weeks or
so syncing about 4 accounts normally.
2009-08-16 09:27:49 -05:00
John Goerzen
2433dc9f06 Merge branches 'master' and 'master' of ssh://jpgarch@complete.org/~jpgarch/git/offlineimap 2009-08-12 15:25:26 -05:00
John Goerzen
1f8024a70e [imaplib2 removal] Revert "Implementation of IMAP IDLE"
This reverts commit 3847d0ba9d.
2009-08-12 14:49:58 -05:00
John Goerzen
e66fd5f51a Don't request INTERNALDATE when we don't need it. Patch from Marc MERLIN. 2009-07-16 20:57:07 -05:00
John Goerzen
141a23849b Reduce memory usage when scanning Maildirs
Patch from Michal Vitecek
2009-07-07 00:52:00 -05:00
Peter Colberg
3299b2240c Applied patch from Peter Colberg to remove usage of hard linking
Closes: #535160

Bug#535160: offlineimap fails on encfs (FUSE) due to hard linking

Package: offlineimap
Version: 6.0.3
Severity: normal
Tags: patch

Hi,

if the localrepository is stored on an encfs, offlineimap fails as follows:

    OfflineIMAP 6.0.3
    Copyright (C) 2002 - 2008 John Goerzen <jgoerzen@complete.org>
    This software comes with ABSOLUTELY NO WARRANTY; see the file
    COPYING for details.  This is free software, and you are welcome
    to distribute it under the conditions laid out in COPYING.
    ***** Processing account pleione
    Copying folder structure from IMAP to Maildir
    Establishing connection to tunnel:eval `keychain --quiet --eval` && ssh -q mymailhost /usr/sbin/dovecot --exec-mail imap.
    Syncing Drafts: IMAP -> Maildir
    Syncing INBOX: IMAP -> Maildir
    ----------8<----------
    Syncing net: IMAP -> Maildir
    Copy message 360 IMAP[net] -> Maildir[net], LocalStatus[net]
    Syncing personal: IMAP -> Maildir
    Thread 'Copy message 360 from net' terminated with exception:
    Traceback (most recent call last):
      File "/var/lib/python-support/python2.4/offlineimap/threadutil.py", line 149, in run
	Thread.run(self)
      File "threading.py", line 422, in run
	self.__target(*self.__args, **self.__kwargs)
      File "/var/lib/python-support/python2.4/offlineimap/folder/Base.py", line 282, in copymessageto
	newuid = object.savemessage(uid, message, flags, rtime)
      File "/var/lib/python-support/python2.4/offlineimap/folder/Maildir.py", line 198, in savemessage
	os.path.join(tmpdir, messagename))
    OSError: [Errno 1] Operation not permitted

    Last 4 debug messages logged for Copy message 360 from net prior to exception:
    ----------8<----------

As encfs does not support hard linking, the rename of the temporary
message file fails.

This is the only occurrence of hard linking in offlineimap and can be
fixed by using an atomic `os.rename` instead of `os.link`+`os.unlink`.
I have been using the attached patch for more than 1.5 years and have
never again had any problems with offlineimap on encfs.

Regards,
Peter
2009-06-30 00:03:37 -05:00