We we using the variable 'severity' in a few places to throw
OfflineImapErrorrs of severity REPO. Somehow, that variable is now not
accessible in all places that refer to it, so we move where it is
defined to before all the 'if' checks which might make use of it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If we do not use a SSL connection anyway and if the server supports it,
authenticate automatically with STARTTLS.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In an IMAP run where we did not have to sync anything, I spend nearly a
fulls second in imaputil.debug() without even having debug output
enabled. imapsplit is mainly called by flagsplit() which will also do
debug output, so we get TONS of nearly duplicate debug output in the log
which makes it really hard to analyze.
Cut down the debug logging in imapsplit, we should be debug logging
stuff at a slightly higher level than here anyway.
This one-line change sped up my folder sync (without having to sync
anything) by 0.5 seconds even when debugging is disabled.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously, we instanciated an MappedImapFolder, and would cleverly (too
cleverly?) invoke methods on it casting it to an IMAPFolder by calling
methods such as: self._mb.cachemessages(self) where self._MB is the class
IMAPFolder and self and instance of MappedImapFolder. If
e.g. cachemessages() invokes a method uidexists() which exists for
MappedImapFolder, but not directly in IMAPFolder, I am not sure if
Python would at some point attempt to use the method of the wrong class.
Also, this leads to some twisted thinking as our class would in same
cases act as an IMAPFolder and in some cases as an MappedImapFOlder and
it is not always clear if we mean REMOTE UID or LOCAL UID.
This commit simplifies the class, by a)doing away with the complex Mixin
construct and directly inheriting from IMAPFOlder (so we get all the
IMAPFOlder methods that we can inherit). We instantiate self._mb as a
new instance of IMAPFolder which represents the local IMAP using local
UIDs, separating the MappedIMAPFolder construct logically from the
IMAPFolder somewhat.
In the long run, I would like to remove self._mb completely and simply
override any method that needs overriding, but let us take small and
understandable baby steps here.
Reported-and-tested-by: Vincent Beffara <vbeffara@gmail.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
sudir->subdir in a debug statement. Thanks ccxCZ on IRC for the heads
up.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
During a sync run, someone might remove or move IMAP messages. As we
only cache the list of UIDs in the beginning, we might be requesting
UIDs that don't exist anymore. Protect folder.IMAP.getmessage() against
the response that we get when we ask for unknown UIDs.
Also, if the server responds with anything else than "OK", (eg. Gmail
seems to be saying frequently ['NO', 'Dave I can't let you do that now']
:-) so we should also be throwing OfflineImapErrors here rather than
AssertionErrors.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
I discovered that we do not run ui.terminate in all circumstances, so
make sure that we call with properly at the end of each run (whether in
threaded or single-thread mode).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Commit 1754bf4110 introduced a blunder:
- for dirname in os.listdir(toppath) + ['.']:
+ for dirname in os.listdir(toppath) + [toppath]:
...
- if self.getsep() == '/' and dirname != '.':
+ if self.getsep() == '/' and dirname:
This change was plainly wrong and would never have worked, so this
commit reverts above bit. While touching the function, some minor code
documentation, cleanup and limiting line length to 80 chars.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Folder.savemessage() is supposed to return the new UID that a backend
assigned, and it BaseFolder.copymessageto() fails if we don't return a
non-negative number in the savemessage() there.
For some reason, the UIDMappedFolder was not returning anything in
savemessage, despite clearly stating in the code docs that it is
supposed to return a UID. Not sure how long this has already been the
case. This patch fixes the UIDMappedFolder to behave as it should.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
During a sync run, someone might remove or move IMAP messages. As we
only cache the list of UIDs in the beginning, we might be requesting
UIDs that don't exist anymore. Protect folder.IMAP.getmessage() against
the response that we get when we ask for unknown UIDs.
Also, if the server responds with anything else than "OK", (eg. Gmail
seems to be saying frequently ['NO', 'Dave I can't let you do that now']
:-) so we should also be throwing OfflineImapErrors here rather than
AssertionErrors.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Fix a gssapi issue where threads beyond the first would not
be able to authenticate against the imap server. This is
done by using the connection lock around the gssapi
authentication code and resetting (and releasing) the
kerberos state after success so that subsequent connections
may make use of kerberos.
Signed-off-by: Scott Henson <sjh@foolishpride.org>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We currently don't care about expiration dates of the servers SSL
certificate. This patch adds a check that fails Cert verification when
it is past its due date. There is no way or option to override this
check.
Unfortunately we only seem to be able to get SSL certificate data when
we passed in a CA cert file? How do we get that date when we don't have
a ca cert file?
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously we were attempting to save out mails according to
http://www.qmail.org/man/man5/maildir.html in 4 steps:
1 Create a unique filename
2 Do stat(tmp/<filename>). If it found a file, wait 2 sec and go back to 1.
3 Create and write the message to the tmp/<filename>.
4 Link from tmp/* to new/*
(we did step 2 up to 15 times) But as stated by
http://wiki1.dovecot.org/MailboxFormat/Maildir (see section 'Issues with
the specification'), this is a pointless approach, e.g. there are race
issues between stating that the filename does not exist and the actual
moving (when it might exist).
So, we can simplify the steps as suggested in the dovecot wiki and
tighten up our safety at the same time.
One improvement that we do is to open the file, guaranteeing that it did
not exist before in an atomic manner, thus our simplified approach is
really more secure than what we had before.
Also, we throw an OfflineImapError at MESSAGE level when the supposedly
unique filename already exists, so that we can skip this message and
still continue with other messages.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
MaildirFolder.messagelist[*]['filename'] was storing the absolute file
paths for all stored emails. While this is convenient, it wastes much
space, as the folder prefix is always the same and it is known to the
MaildirFolder. Just 40 chars in a folder with 100k mails waste >4MB of
space. Adapt the few locations where we need the full path to construct
it dynamically.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We use getfullname() very often (thousands to millions), yet we
dynamically calculate the very same value over and over. Optimize this
by caching the value once and be done with it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In some cases we had offlineimap trying to delete emails that shouldn't
be deleted. E.g. https://bugzilla.redhat.com/show_bug.cgi?id=708898.
It turns out that imaplib2 does not like FETCH responses that are
interrupted by other unsolicited server responses, e.g.
* OK Searched 43% of the mailbox, ETA 0:12\r\n
Bump imaplib2 to a version that can cope with these (legal) responses by
the IMAP server.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
we do:
for msgid in imapdata:
maxmsgid = max(long(msgid), maxmsgid)
and then basically immediately:
maxmsgid = long(imapdata[0])
throwing away the first assignment although the first method of
assigning is the correct one. The second had been forgotten to be
removed when we introduced the above iteration. This bug would fix a
regression with those broken ZIMBRA servers that send multiple EXISTS
replies.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
All months names are 3-letter abbreviated, but accidentally June and
July slipped through. Thanks to the heads up by
Philipp Kern <pkern@debian.org>.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously we hard-coded the imap server name in the case of Gmail
repositories, but often we need a different host name. So, allow people
to specify the hostname via the regular "remotehosteval" and
"remotehost" settings, and only falling back to imap.gmail.com when
nothing has been specified.
Cache the hostname, so we don't evaluate the whole thing each time we
query the host name.
Make the remotehosteval processing more robust, by catching any
Exceptions that occur, and throw a OfflineImapError, that explains where
exactly the error had occured. You can test this, e.g. by setting
remotehosteval to 1/"n" or some other invalid expression.
The whole IMAP.gethost() function has been documented code wise while
going through.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We currently do not allow nametrans rules such as
nametrans = lambda foldername: re.sub('^INBOX$', '', foldername)
because we crash with a traceback when running:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=499755
The underlying reason is that we cannot create the "top level" root
directory of the Maildir in the function makefolders(), it will bail
out. John Goerzen intentionally prevented offlineimap from creating the
top-level dir, so that a misconfiguration could not arbitrarily create
folders on the file system. I believe that it should be perfectly
possible to automatically create the root dirctory of the maildir. We
still protect against folder creations at arbitrary places in the file
system though.
This patch cleans up makefolders(), adds documentation, allows to
automatically create rootfolders if needed (using absolute paths) and
adds some robustness in case the folders already exist that we want to
create (rather than simply crapping out).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=499755 shows the
cryptic output we have when, e.g. trying to put somethin in our mailDir
root via the nametrans options. This commit adds at least some hint as
to what went wrong using an "assert" message, although the correct thing
is to allow the creation of a maildir in the root folder.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were "crashing" with tracebacks when we could not connect to a host,
(e.g. because no service was on the port) and we were getting mysterious
SSL tracebacks when someone tried to connect via SSL to a non-ssl port.
In these cases, we will now throw an nice error message. On python<2.6
where no ssl module exists, we simply won't throw those errors.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
commit 0318c6a [Create LocalStatus or LocalStatusSQLite folders] changes import
of LocalStatus but doesn't preserve magicline.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Raise OfflineImapError with severity REPO explaining that the connection failed.
Before, no valuable information was given to the user.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Without this patch, we try to NOOP on a bad connection and crash messily.
Signed-off-by: Tom Lawton <tlawton@gmx.de>
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This commit was originally by James Bunton <jamesbunton@fastmail.fm>.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If the server doesn't support IDLE, we fall back to the standard
noop() keepalive.
This commit was originally by James Bunton <jamesbunton@fastmail.fm>.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This is the commit that enables IDLE support. In order to do this, we
hijack the keepalive method. Instead of just sending NOOPs, it now
sends IDLE and responds accordingly, thanks to the IdleThread class.
This code was originally by James Bunton <jamesbunton@fastmail.fm>.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This encapsulates the logic for sending a keepalive/IDLE call,
including starting a sync if needed.
This code was originally by James Bunton <jamesbunton@fastmail.fm>. I
modified the idle() method to put the select() call after
acquireconnection().
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Mark this option as experimental and document its shortcomings in
MANUAL.rst.
This code was originally by James Bunton <jamesbunton@fastmail.fm>.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This is a regression introduced by commit d5493fe894
[threadutil: explicitly import get_ident from thread].
The threadid attribute was wrongly removed from the ExitNotifyThread class.
Restore it.
Tested-by: Mark Foxwell <fastfret79@archlinux.org.uk>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We are missing the import of 'os' python module since commit d839be3c61
(Sat Apr 16 20:33:35 2005 +0100) which was when John switched from SVN to Git.
Happily, it help us today: we still had no feedback for this missing import,
6 years later. So, we can remove the os.exit() call safely.
That beeing said, we still don't know if the above sys.exit() was ever touched.
My guess is that it never was. Keep this (hopefully) commented statement to
ensure the thread terminate and not play too much with the Murphy's law. :-)
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The semaphorewait()/waitforthread() logic is usefull for IMAP starting
connections. We actually use it in imapserver only.
This patch removes the over-engineered factorized methods. It tend to simplify
the code by cleaning out a chain of two direct calls with no other processes.
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The python module thread is the low-level module we should avoid to use in favor
of threading. We still need it to support old python because Thread.ident
doesn't exist before python 2.6:
http://docs.python.org/library/threading.html#threading.Thread.ident
Make it clear we should avoid it.
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
- conform to PEP8
- explicitly define symbols instead of 'import *'
- remove unused import
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were outputting full message bodies to the debug log (often stderr),
and then again (as they go over the imaplib2 wire, imaplib logs
everything too). Not only is quite a privacy issue when sending in debug
logs but it can also freeze a console for quite some time. Plus it
bloats debug logs A LOT.
Only output the first and last 100 bytes of each message body to the
debug log (we still get the full body from imaplib2 logging). This
limits privacy issues when handing the log to someone else, but usually
still contains all the interesting bits that we want to see in a log.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The syntax was not right, and deleting messages from the LocalStatus
failed. (We passed in the full list of uids and we need to pass in one
uid at a time (as a tuple). Deleting messages works now.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The SigListener class was used to queue folders that we need to sync and
to receive "resync" and "abort" signals. It was undocumented and weird
and we had to pass "siglisteners" through the whole program.
Simply do away with it, and make 2 functions in the Account() class:
set_abort_event and get_abort_event which can be used to set and check
for such signals. This way we do not need to pass siglisteners all over
the place. Tested Blinkenlights and TTYUI uis to make sure that SIGUSR1
and SIGUSR2 actually still work.
Document those signals in MANUAL.rst. They were completly undocumented.
This simplifies the code and interdependencies by passing less stuff
around. Removes an undocumented and weirdly named class.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Throw an OfflineImapError when SELECTing a folder is unsuccessful and
bail out with a FOLDER serverity. In accounts.py catch all
OfflineImapErrors and either just log the error and skip the folder or
bubble it up if it's severe.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Use two %s in the message for both string parameters.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make the folder classes use uidexists() more. Add some code
documentation while going through.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Depending on the configuration we use the plain text or the new
experimental sqlite backend for the LocalStatus cache. Make plain text
the default status backend but allow people to configure
status_backend=sqlite in their [Account ...] section.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Test if sqlite is multithreading-safe and bail out if not. sqlite
versions since at least 2008 are.
But, as it still causes errors when 2
threads try to write to the same connection simultanously (We get a
"cannot start transaction within a transaction" error), we protect
writes with a per class, ie per-connection lock. Factor out the retrying
to write when the database is locked.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make getfolders() invoke getfolder() for each folder rather than
duplicating code. Also add a forgetfolders() implementation.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
doautosave was a useless variable before (it was *always* 1). So we
remove the self.dofsync variable and store in doautosave whether we
should fsync as often as possible (which really hurts performance).
The sqlite backend could (at one point) use the doautosave variable to
determine if it should autocommit after each modification.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Python's new style classes derive from object and str(class().__class__)
will return a slightly different format. class().__class.__name__ will
still work for both old and new style classes, so use that instead.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Based on patches by Stewart Smith, updated by Rob Browning.
plus:
- Inherit LocalStatusSQLFolder from LocalStatusFolder
This lets us remove all functions that are available via our ancestors
classes and are not needed.
- Don't fail if pysql import fails. Fail rather at runtime when needed.
- When creating the db file, create a metadata table which contains the
format version info, so we can upgrade nicely to other formats.
- Create an upgrade_db() function which allows us to upgrade from any
previous file format to the current one (including plain text)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
A typo prevented us from enforcing singlethreading mode when selecting debugging.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Using threading._VERBOSE=1 is broken since python 2.6 till at least
python 3.2, (http://bugs.python.org/issue4188) so we can't use it for
our thread debugging.
Remove the usage of threading._VERBOSE, and implement a "light thread
debug log" that for now outputs information when a new thread is being
registered and when it is being unregistered. I am sure we will be able
to add more thread debugging information over the time.
Besides '-d thread' this will re-enable the usage of -d 'all' for the
most verbose debugging of all categories.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When our LocalStatus cache is corrupt, ie e.g. it contains lines not in
the form number:number, we would previously just raise a ValueError
stating things like "too many values". In case we encounter clearly
corrupt LocalStatus cache entries, clearly raise an exception stating
the filename and the line, so that people can attempt to repair it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This should get rid of intermittent network failures, but lets us bail
out on permanent errors.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
By aborting the account syncing, the looping and logging an error
message. We will introduce a ui.error() rather than a ui.warn() function
which saves all Exceptions in a Queue and outputs them at the end of the
program.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In case we misconfigured a server name or are otherwise offline, a
socket.gaierror will be raised when attempting to connect. We catch that
case and raise an OfflineImapError with severity ERROR.REPO, meaning we
should stop syncing this account and continue with the next one.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This Exception can be thrown whenever a sync error occurs. It is used
for outputting sensible error messages to the user and it denotes a
"severity", it can tell offlineimap if we need to abort a message, a
folder, a repo sync, or whether we should indeed abort immediately.
The exception is not yet used in the code.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Currently, account.syncrunner() has 2 separate duplicated code paths
depending on whether we want to autorefresh after some waiting perios
or not. Unify those code paths by setting "looping = False" in case
self.refeshperiod == 0 after the first run. Behavior is identical to
before.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This commit enables true 1-way syncing between repositories. This has
often been demanded for backup purposes when you do not want to cause
accidental modifications of your backup that would be propagated to the
other side.
This has been implemented by allowing to configure a Repository as
'readonly' to forbid any modification on it.
'readonly' applies to all the type of repositories.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This debug type will always be enabled whenever any debugging is enables
and it outputs debug messages that cannot be categorized among any of
imap, maildir (e.g. things that concern the sync logic).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Improve the code documentation (still much more to do) and also add some
more meat to the structure of the developer documentation.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This function will need much more "robustifying", but the very least we
can do is to print the file name and line that are giving trouble.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Allow leading and trailing spaces in folder names specified on the
command line.
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Dan Christensen <jdc@uwo.ca>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
in accounts.sync() we would holdordropconnections() after each sync. But
depending on the repository configuration that might imply that
offlineimap tries to keep the same connections. But when a sync failed,
e.g. after a user had his computer suspended, it might be that our
connections that we have are worthless. So definitely drop them after a
failed sync.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Add some comments how the data structures actually look like.
Describe the function properly, and make sure we only hold on to the
data connection as quickly as possible.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This was in the code for a very long time, it seems.
Remove one instance, no functional changes.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
uncritical patch, but we can make the code a bit shorter so why not.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
So we can simply hand a repository instance to error messages rather than
having to call Repository().getname() all the time.
This is not used yet.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
So we can simply hand an account instance to error messages rather than
having to call Account().getname() all the time.
This is not used yet.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Just clean up lines which end in whitespaces.
Also adapt the copyright to the current year while touching the file.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In order to optimize performance, we fold the 1st and 2nd pass of our
sync strategy into one. They were essentially doing the same thing:
uploading a message to the other side. The only difference was that in
one case we have a negative UID locally, and in the other case, we have
a positive one already.
This saves some time, as we don't have to run through that function on
IMAP servers anyway (they always have positive UIDs), and 2nd were we
stalling further copying until phase 1 was finished. So uploading a
single new message would prevent us from starting to copy existing
regular messages.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
For each folder we were making a second IMAP request asking for the
latest UID and compared that with the highest UID in our
statusfolder. This catched the case that 1 mail has been deleted by
someone else and another one has arrived since we checked, so that the
total number of mails appears to not have changed.
We don't capture anymore this case in the quickchanged() case.
It improves my performance from 8 to about 7.5 seconds per check (with lots of
variation) and we would benefit even more in the IMAP<->IMAP case as we do one
additional IMAP lookup per folder on each side then.
Do cleanups on whitespaces while in this file.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
* IMAP4_Tunnel constructor should support base-class arguments, in
order to support the timeout argument.
* IMAP4_Tunnel needs to store the member IMAP4.host, which is normally
done in IMAP4.open().
* Update IMAP4_Tunnel.read() and IMAP4_Tunnel.send(). We turn on
nonblocking mode for these sockets, so we can return immediately
with whatever data is available.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This contains a fixed Time2InternalDate function and a more robust
socket connection, trying twice and raising an error only when that
fails (I believe). The actual code changes are rather minor.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
quickchanged() was iterating a lot, make use of some of the helper
functions that had been introduced recently and document the function a
bit better. No functional change.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously ALL spaces had been stripped off. Now, only strip spaces
around the comma, so -f "INBOX, Deleted Mails" will work. You will still
need to quote or escape spaces so the shell hand the list as one command
line argument to offlineimap.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
getmessagelist() is slow for the mapped UID case, so replace some of its
occurences with calls that are optimized for this case, ie
getmessagecount() and uidexists().
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-and-tested-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We are calling getmessagelist() internally a lot, e.g. just to check if
a UID exists (from uidexist()). This is a very expensive operation in
the UIDMapped case, as we reconstruct the whole messagelist dict every
single time, involving lots of copying etc.
So we provide more efficient implementations for the uidexists()
getmessageuidlist() and getmessagecount() functions that are fast in the
UIDMapped case. This should solve the performance regression that was
recently observed in the Mapped UID case.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Reviewed-and-tested-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Signed-off-by: David Favro <offlineimap@meta-dynamic.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When uploading a new message to Gmail we need to find out the UID it
assigned it, but Gmail does not advertize the UIDPLUS extension (in all
cases) and it fails to find the email that we just uploaded when
searching for it. This prevented us effectively from uploading to
gmail.
See analysis in
http://lists.alioth.debian.org/pipermail/offlineimap-project/2011-March/001449.html
for details on what is going wrong.
This patch increases compatability with Gmail by checking for APPENDUID
responses to an APPEND action even if the server did not claim to
support it. This restores the capability to upload messages to the
*broken* Gmail IMAP implementation.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Without this, trying to Ctrl-C out of offlineimap will go into a hang.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
All other instances were converted to format crash output including a
stacktrace, but this one seems to have been left out. Make Exceptions
print their stacktrace here too.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The three classes with names starting with UsefulIMAP4 were used for two
purposes, to include the UsefulIMAPMixIn class and to implement various
system-specific kludges. None of these kludges remain, so it is cleaner
to include UsefulIMAPMixIn directly in imaplibutil and forget about them
for good.
Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Now that we do not need any system-specific hack, the three UsefulIMAP4
classes have become empty and should be removed. This implies importing
UsefulIMAPMixIn directly from the classes defined in imaplibutil.
Prepare this change by moving the code into imaplibutil.py. Functionally
this is a no-op.
Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Because of a buggy realloc() implementation in earlier versions of
Python on Mac OS X, we had to cut reads into manageable chunks by hand;
this is no more needed with Python 2.6, and besides it causes problems
with imaplib2, which we now use. Revert the special case to use the
system's read() instead, which is now safe.
Signed-off-by: Vincent Beffara <vbeffara@ens-lyon.fr>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
As the LocalStatus and UIDMap backend already did: If the uid already
exists for savemessage(), only modify the flags and don't append a new
message.
We don't invoke savemessage() on messages that already exist in our sync
logic, so this has no change on our current behavior. But it makes
backends befave more consistent with each other.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
- Some documentation improvements, this is a severely underdocumented
class. This still needs some further improvements though.
- Don't use apply(Baseclass) (which is going away in Python 3), use
IMAPFolder.__init__(self, *args, **kwargs).
- Don't call ValueError, string. It is ValueError(string)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were not able to handle ~/... type of path configurations and we
crashed with mysterious SSL errors when no file was found at the
configured location. Expand '~' and bomb out with usable error messages
in case such a file does not exist. This will still not protect against
corrupt cacert files but it goes a long way towards user friendliness.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
imaplib2 does not use socket, so does not know about the
defaulttimeout we set based on the config. Instead, we explicitly pass
the default timeout.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
For read(), the imaplib2 version seems to work perfectly well. The
others aren't used any more, either by imaplib2, nor by us, so we may
as well get rid of them.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The biggest change here is that imapobj.untagged_responses is no
longer a dictionary, but a list. To access it, I use the semi-private
_get_untagged_response method.
* offlineimap/folder/IMAP.py (IMAPFolder.quickchanged,
IMAPFolder.cachemessagelist): imaplib2 now explicitly removes its
EXISTS response on select(), so instead we use the return values from
select() to get the number of messages.
* offlineimap/imapserver.py (UsefulIMAPMixIn.select): imaplib2 now
stores untagged_responses for different mailboxes, which confuses us
because it seems like our mailboxes are "still" in read-only mode when
we just re-opened them. Additionally, we have to return the value
from imaplib2's select() so that the above thing works.
* offlineimap/imapserver.py (UsefulIMAPMixIn._mesg): imaplib2 now
calls _mesg with the name of a thread, so we display this
information in debug output. This requires a corresponding change to
imaplibutil.new_mesg.
* offlineimap/imaplibutil.py: We override IMAP4_SSL.open, whose
default arguments have changed, so update the default arguments. We
also subclass imaplib.IMAP4 in a few different places, which now
relies on having a read_fd file descriptor to poll on.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
imaplib2 has slightly different semantics than standard imaplib, so
this patch will break the build, but I thought it was helpful to have it as
a separate commit.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This change does not do anything yet with imaplib2, merely makes it
available for future commits.
This file is identical to the one at
http://sydney.edu.au/engineering/it/~piers/python/imaplib2 .
imaplib2, written by the same guy who wrote imaplib, is very different
from imaplib itself. Calling it a modified version from the standard
distribution is misleading. It's more like a complete rewrite. As
such, it's not really possible to summarize what was changed.
The largest thing is that imaplib2 is "threaded". Instead of doing
blocking writes/reads on the socket during/after every command,
imaplib2 forks off threads to read and write to the socket based on
input and output buffers. This opens the door to asynchronous
commands (every command is potentially asynchronous, according to the
docs), and in particular IDLE, which is by definition an asynchronous
command.
The author writes: "imaplib2 can be substituted for imaplib in
existing clients with no changes in the code", but that's pretty
misleading. It might be true for certain simple users of imaplib, but
for us it's completely false. Among other things, how untagged
responses are stored in-memory is different -- instead of a hash
table, it's a list. I'm guessing this is to preserve order of
responses.
I think there are other miscellaneous improvements, like I think
imaplib2 is IPv6 safe out-of-the-box, but I haven't conducted an
extremely thorough examination of the differences :)
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The rfc822 module has been deprecated since python 2.3, and conversion to
the email module is straightforward, so let us do that. rfc822 is
completely gone in python3.
This also fixes a bug that led to offlineimap abortion (but that code path
is apparently usually not exercised so I did not notice:
rfc822|email.utils.parsedate return a tuple which has no named attributes,
but we were using them later in that function. So pass the tuple into a
struct_time() to get named attributes.
While reading the docs, I noticed that email.parsedate returns invalid
daylight savings information (is_dst attribute), and we are using it
anyway. Oh well, the imap server might think the mails are off by an hour
at worst.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The only reason we used it here was to do a
traceback.print_exc(StringIO()) to get a string of our traceback. But we
can simply use traceback.format_exc() which exists since python 2.4.
One less module (and it is in the way to python 3 compatability too)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In commit 7a2a0225 [Don't pass list to ui.adding/deletingflags] we changed the
list logic for a per folder logic but forgot to remove one instance of
"destlist" which isn't valid anymore.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
More convenient way to test if a certain uid exists and getting a list
of all uids. Also, the SQL backend will have efficient overrides for
these methods.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than always having to call len(getmessagelist.keys()) as was done
before. No functional change, just nicer looking code. Also the SQLite
backend or other backends could implement more efficient implementations.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously we would output:
Folder sync sspaeth.de[INBOX.INBOX201004]:
Syncing INBOX.INBOX201004: IMAP -> Maildir
Folder sync sspaeth.de[INBOX.INBOX201006]:
Syncing INBOX.INBOX201006: IMAP -> Maildir
Folder sync sspaeth.de[INBOX.INBOX201009]:
Syncing INBOX.INBOX201009: IMAP -> Maildir
which is very repetitive and cluttered. By naming the folder sync
threads just according to the account and not the folder, the output
looks much nicer:
Folder sync [sspaeth.de]:
Syncing INBOX.INBOX201004: IMAP -> Maildir
Syncing INBOX.INBOX201006: IMAP -> Maildir
Syncing INBOX.INBOX201009: IMAP -> Maildir
If syncing multiple accounts in parallel, we will still get headers
indicating the account:
Folder sync [sspaeth.de]:
Syncing INBOX: IMAP -> Maildir
Syncing INBOX.INBOX201006: IMAP -> Maildir
Folder sync [gmail]:
Syncing INBOX: IMAP -> Maildir
This is a small fix that makes the output much nicer in my opinion.
Also don't output the thread name if we are in the MainThread, e.g. when
we output the initial offlineimap banner.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Read() should return empty string when EOF happen, instead of looping
forever. This is the right semantics of read(), and a wrapped version
should not change it.
If you read the read(2) system call manpage, it tells you that when EOF
is seen, return value is 0; it does not say
``loop forever when EOF happen''.
After the EOF detection is patched you can see the
following exception:
WARNING: ERROR attempting to copy message 344 for account Gmail:Traceback (most recent call last):
File "/usr/lib/pymodules/python2.6/offlineimap/folder/Base.py", line 282, in copymessageto
message = self.getmessage(uid)
File "/usr/lib/pymodules/python2.6/offlineimap/folder/IMAP.py", line 216, in getmessage
initialresult = imapobj.uid('fetch', '%d' % uid, '(BODY.PEEK[])')
File "/usr/lib/python2.6/imaplib.py", line 753, in uid
typ, dat = self._simple_command(name, command, *args)
File "/usr/lib/python2.6/imaplib.py", line 1060, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "/usr/lib/python2.6/imaplib.py", line 890, in _command_complete
raise self.abort('command: %s => %s' % (name, val))
abort: command: UID => socket error: EOF
Signed-off-by: Bao Haojun <baohaojun@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We only have one "dstfolder" at a time when deleting/adding flags, so no
need to pass in a list of those to the ui functions that output the log
info.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
They were not PEP-8 formatted, and some imports were simply
unnecessary. Removed those.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This enables us to just use the folder instance in the ui output and get
a name rather than having to call getname() all the time.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The previous syncing strategy was doing more than we needed to and was a
bit underdocumented. This is an attempt to clean it up.
1) Do away with the previous different code paths depending on
whether there is a LocalStatus file or not (the isnewfolder() test). We
always use the same strategy now, which makes the strategy easier to
understand. This strategy is simply:
a) Sync remote to local folder first
b) Sync local to remote
Where each sync implies a 4 pass strategy which does basically the same
as before (explained below).
2) Don't delete messages on LOCAL which don't exist on REMOTE right at
the beginning anymore. This prevented us e.g. from keeping local
messages rather than redownloading everything once LocalStatus got
corrupted or deleted. This surprised many who put in an existing local
maildir and expected it to be synced to the remote place. Instead, the
local maildir was deleted. This is a data loss that actually occured to
people!
3) No need to separately sync the statusfolder, we update that one
simultanously with the destfolders...
3) Simplified the sync function API by only taking one destdir rather
than a list of destdirs, we never used more anyway. This makes the code
easier to read.
4) Added plenty of code comments while I was going through to make sure
the strategy is easy to understand.
-----------------------------------------
Pass1: Transfer new local messages
Upload msg with negative/no UIDs to dstfolder. dstfolder should
assign that message a new UID. Update statusfolder.
Pass2: Copy existing messages
Copy messages in self, but not statusfolder to dstfolder if not
already in dstfolder. Update statusfolder.
Pass3: Remove deleted messages
Get all UIDS in statusfolder but not self. These are messages
that we have locally deleted. Delete those from dstfolder and
statusfolder.
Pass4: Synchronize flag changes
Compare flags in self with those in statusfolder. If msg has a
valid UID and exists on dstfolder (has not e.g. been deleted
there), sync the flag change to dstfolder and statusfolder.
The user visible implications of this change should be unnoticable
except in one situation:
Blowing away LocalStatus will not require you to redownload ALL of
your mails if you still have the local Maildir. It will simply recreate
LocalStatus.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Removing this lock makes the function not threadsafe, but then it is
only ever called from one thread, the main account syncer. Also, it
doesn't make it worse than most of the other functions in that class
which are also not threadsafe.
Removing this makes the code simpler, and removes the need to import the
threading module.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The previous ui names were pretty unwieldy. Is it TTYUI.TTY or
TTY.TTYUI? Do I have to use capitals and where?
Simplify the names by making them case insensitive and by dropping
everything before the dot.
So "Curses.Blinkenlights" can now be invoked as "blinkenlights" or
"BLINKENLIGHTS". The old names will still work just fine so the
transition should be smooth. We issue a warning that the long names are
deprecated.
Document in offlineimap.conf that we don't accept lists of fallback UIs,
but only one UI option (this was already the case before this commit but
still wrongly documented).
The list of accepted ui names is:
ttyui (default), basic, quiet, machineui, blinkenlights
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Multithreading speeds up account syncing a lot and the offlineimap
defaults are very conservative. Let's make it use 2 IMAP connections by
default to gain some of the benefits that offlineimap offers.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than inserting our own home-grown header, everytime we save a
message to an IMAP server, we check if we suport the UIDPLUS extension
which provides us with an APPENDUID reply. Use that to find the new UID
if possible, but keep the old way if we don't have that extension.
If a folder is read-only, return the uid that we have passed in per API
description in folder.Base.py
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The working horse of the savemessage() function, imaplib.append() was
hidden away in an assert statement. Pull the real functions out of the
asserts and simply assert on the return values. This looks less
convoluted and makes this easier to understand in my opinion.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
savemessage was too long and complex. Factor out the date guessing part
of the function and put it into a function of its own. The logic of the
date guessing is the same, however, we do not use the
imaplib.Time2InternalDate() function as it is buggy
(http://bugs.python.org/issue11024) and returns localized patches. So we
create INTERNALDATE ourselves and pass it to append() as a string.
This commit fixes a bug that international users used to pass an invalid
date to the IMAP server, which the server will either ignore or complain
about.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
savemessage_getnewheader was an undocmented, cryptic and overengineered
function. It generates a new unique value that can be used as a mail
header to be inserted. For this it used LOTS of randomness sources: hash
of the mail content, hash of the folder name, hash of the repository
name, the current time, a random() value, and the offlineimap version string.
All we need is something random. So reduce this to hash of content
appended by a random integer. Sufficient and somewhat faster to calculate.
Rename the function to actually describe accurately what it does or
would you have guessed that savemessage_getnewheader() did nothing more
than returning ('X-OfflineIMAP', <randomstring> )? Rename to
generate_randomheader() to make it clearer what this is all about.
Also document the function, describing what it does, and what it returns.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
There is a clumsy workaround for Darwin that chunks reads into 8kb
blocks to avoid huge memory allocations. First, this fix should not only
be required but on FreeBSD2.6 too (see
http://bugs.python.org/issue3531). Second, decent python versions (I
checked 2.6) already chunk in the SSL case anyway, so there is no need to do
that again. Remove that level of indirection.
http://evanjones.ca/python-memory.html claims that this problem has been
fixed since python 2.5, so we might consider removing the workaround
completely even for the non-SSL case.
Increase the chunk size on Mac from 8kb to 64kb. Even Macs should be
able to take that amount of memory usage nowadays.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
A typo was preventing profiling mode to really enable singlethreading
mode. Fixing the unfortunate typo of mine makes it work.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It is currently very hard to find out what folderfilter actually does
and makes it hard to debug for a user. With this patch if the user has
enabled "-d imap" (even better would perhaps be a different debug type
for this kind of thing?), we see a message
"Filtering out folder 'foo' due to folderfilter"
in the logs.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This enables us to make use of self.ui in all repositories without
having to import and use getglobalui() in all types of repositories and
all places.
Note that this patch only makes this available, it does not yet make use
of it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When e.g. specifying an invalid 'reference' value for an IMAP server to
a root folder that does not exist, we would previously have crashed with
a nonsensical and non-intuitive error message (trying to address an
element of a NoneType).
This will also raise an Exception (which should be ok, given that this
is really a misconfiguration on the user side), but it will explain to
the user WHY it exited.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The debug output for dequote, optionsplit is very verbose, outputing
what the functions are called with and what they return. Those functions
are now very old mature and rather simple, so it suffices to output
their return value rather than cluttering out log with too much
uninteresting garbarge.
This makes log files much more readable.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
A Repository() returns the correctly instanciated dervivate of a
BaseRepository, depending on the parameters passed to it. The returned
instance is eg an ImapRepository(). This makes the code look nicer,
and we have less functions lying around outside of classes (no more
global LoadRepository() function).
This will also enable us to conveniently hand back a
LocalStatusRepository based on SQLITE rather than plain text, if the
user configures this to be the experimental and optional backend
(once it exists).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This seems to prevent mysterious hangs with SSL imap servers (especially
gmail?) and does not harm in any case. So let us enable keep-alive
messages for ssl connections.
Our thread pool should be made more robust against closed SSL
connections (which do not always seem to raise Exceptions), and not
deadlock while waiting for resources or data that will never arrive.
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Gábor Melis <mega@retes.hu>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Commit e506442996 changed getglobalui() back to UIBase.getglobalui()
although the import had changed earlier, causing a regression.
Fix this by using the correct and current way of calling the ui.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Signed-off-by: Thomas Jost <schnouki@schnouki.net>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
There is no need for using the string module if all we want is to split
a string at the white space. All pythons since at least 2.4 can do that.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We have vonverted all places in folder/* to have self.ui available,
rather than having to use UIBase.getglobalui() all the
time. Unfortunately, we did not convert the users in folder/Base.py.
This patch does it belatedly. This fixes
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=613483
Signed-off-by: Sebastian Spaeth1 <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
I accidentally added "self" as first parameter to the signal handler
method. Of course it is not called with a class instance, the signal
handler always only receives 2 parameters: the signal number and the
stack frame. So just removing the self fixes things.
Proposed-by: Christian Holme <cholme@gmx.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Code was broken, as these backends import UIBase (which had been moved).
However, they don't use it, so we can just delete the import.
Sorry, I failed to find those earlier somehow.
Signed-off-by: Sebastian <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
AccountSynchronizationMixin was never used on its own and it is a very
confusing class until you understand what it is used for. (It
complemented the Account() class with a few methods to make Account()
syncable.
But we use the SyncableAccount class anyway, so merge the former Mixin'
methods directly in there.
This does away with a class that is not directly used, and was a case of
over-object-orientation which confuses more than it helps.
Touched up code documentation while going through the file.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
"mailboxes" is defined global and set to an empty list, but never used
from anywhere within offlineimap. So let us just delete it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Signed-off-by: Edward Z. Yang <ezyang@mit.edu>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make --version much less verbose and only print out the version number
to facilitate easy parsing by scripts. We don't really need the verbose
copyright output, it is show with --help anyway.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Currently the Curses code is broken. Importing offlineimap.ui.Curses
will not succeed due to cyclic imports (threadutils imports ui, but ui
wants threadutils.MultiLock). So Curses cannot be chosen.
Incidentally, the only part in the code that uses "MultiLock" is the
Curses UI, to prevent concurrent access from several threads to the
ui-internal thread list and to IO resources such as the
screen. Fortunately for these purposes we don't need a MultiLock, so we
can do away with that implementation completely. A simple RLock that
allows us to have a thread "own" a lock and makes other threads wanting
access to the resource wait until the owning thread is finished.
The MultiLock implementation looked a bit weird, so simplifying code
here is a good thing, it might well be that we fix some "hangs" that
have been reported (and that would only ever occur with the Curses UI).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This fixes some leftover of 0b5b38d298.
Signed-off-by: Thomas Schwinge <thomas@schwinge.name>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This branch is currently broken, as we were using getglobalui() but it
was not imported. This fixes the missing import and makes offlineimap
run successfully.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Improve documentation about what CustomConfigHelperMixin does,
it was not very clear without a close look to the code before.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The current code path checked the CA cert host name, even if we did not
specify a CA cert file to use. Make the host name check dependent on a
CA cert file.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
All invocations of -d will imply the single-threaded mode unless one of
the debug options is "thread" (in which case it would be senseless to
force single-threaded mode).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously, we would spawn child threads for account synchronization
even if we had single-threading enabled. This prevented us from catching
the true location of exceptions, for example. Now, in single-threaded
mode, we perform the account synchronization truely in the main thread
which will ease our debugging.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Earlier we would ignore *ALL* Exceptions that could occur during the
fsyncing of our LocalStatus database. Ignoring all Exceptions is not the
right thing here though. A recent commit improved the situation by
raising at least KeyboardInterrupt Exceptions, but that is still not
optimal.
os.fsync() is available on Unix, and Windows starting in python
2.2.3. so it should always work. If it doesn't, something is wrong.
It has been suggested to only catch EnvironmentError (ie SystemError and
OSError) here, but even those should be thrown. Something *is* wrong if
this fails and we should not ignore it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously we did not catch KeyboardInterrupts explicitly as all of the
code was executed in forked child threads which would never receive
Ctrl-c exceptions. With the upcoming single threaded modus, this code
can be run in the main thread however, so we need to take care of
KeyboardInterrupts explicitly.
This was done wherever we would catch *ALL* exceptions universally and
print out an error message.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously we did not catch KeyboardInterrupts explicitly as all of the
code was executed in forked child threads which would never receive
Ctrl-c exceptions. With the upcoming single threaded modus, this code
can be run in the main thread however, so we need to take care of
KeyboardInterrupts explicitly. As this is pretty highlevel code, we also
protect against receiving a SystemExit exception which is raised e.g. in
the ui.terminate() code by calling sys.exit().
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make imports conform to PEP8 (one import per line), remove redundant or
unneeded imports.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We already import threadutil, so no need to also import
threadutil.InstanceLimitedThread separately.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The ui.detector class was not really needed and leads to the illusion
that we provide GUI plugins. For the sake of code maintainability we
don't :-).
Rather than having GUI names equivalent to the classes they are in
(which leads to weird names like TTY.TTYUI), this patch allows to give
each GUI an arbitrary string name. GUI names remain still unchanged in
this patch, the default UI when none was configured is TTY.TTYUI.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than regetting the ui with UIBase.getglobalui() all the time, we get it once in the base class and let all derivative classes just make use of self.ui rather than refetching the ui all the time, this makes for a bit less code and shorter lines.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The latter is much shorter and looks nicer. UIBase was a very weird
name and with this patch, we don't need to use (or see) it from higher
level code anymore.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The latter is shorter and looks nicer. UIBase was a very weird class
name for something that is "user visible". We don't need to use (or
see) it from higher level code for most of the code now.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Move central constant definitions into __init__.py. This does away
with version.py which contained nothing else and __init__.py is where
things like __VERSION__ are usually defined.
This commit also changes code to use offlineimap.__version__ rather
than offlineimap.version.__version__ as was before. Cleaned up some
duplicate or unneeded imports while touching those, formatting import
statements per PEP8 (one import per row).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously, we did not check at all the authenticy and validity of
the SSL server we connected to. This is bad as it allows
man-in-the-middle attacks etc. This patch remedies the situation
somewhat.
If we specify a sslcacertfile= setting in the Repository section,
validate the server cert (on python>=2.6 or abort with python<=2.5).
As before, no certificate check is performed without that option.
In the future, the hostname check should be made optional and also
a mutt-lick "accept this certificate forever" thing should be
implemented.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Move them into the correct classes, overriding the open() function.
This is what we intent to do anyway, so do it in a clean way.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Move the command line option handling from the more manual "getopts"
module to the full fledged "OptionParser". It has the advantage of
e.g. automatically creating the Help text that is shown with the
"-h/--help" option rather than having to hardcode the text and
format the line breaking manually.
Also, do away with the version check when invoking OfflineImap so we
do not have to change the version number in three places when bumping
it.
Rename startup() to run() which sounds more in line what other modules
call their run functions (e.g. Thread.run()).
Signed-Off-By: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
On Mon, Dec 13, 2010 at 03:03:20PM -0600, Sebastian Spaeth wrote:
>
> > ...From where is the "Repository" object used?
>
> Please delete the Respository clas (repository.py) for now. I am sorry
> it sneaked it (I blame my horrible jet leg on it). I would like to
> introduce it in a later topic, but not at this time.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Commit 9239a2d326 broke getting the password from the UI. This
unbreaks the change and adds some extended documentation and cleanups in
the functino en-passent.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previoiusly, we would just bomb out with a KeyError("Foo") if a user
configured a repository Type=Foo. Or in case he tried to sync from a
Maildir to a Maildir. Still abort with an Exception now, but with one
that explains what actually had happened.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
On 12/13/2010 12:25 PM, Nicolas Sebrecht wrote:
> The release number information is hard coded. Here is a quick fix to update it
> to the last release.
>
> We may introduce more elaborated stuff to define the release dynamically.
FYI: not sure if you all saw it or changed it, but there were three
places I always changed for each new release:
offlineimap.py
bin/offlineimap
offlineimap/version.py
in particular, the value in version.py must match the value in the
offlineimap(.py) file that someone uses to start the thing up.
Reported-by: John Goerzen <jgoerzen@complete.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Instead of repeating a mapping table twice, once for each direction, it is only
stored once. This is safer and faster.
As the patch stands, it makes no custom flags available. The behaviour
is exactly as before.
Patch-written-by: Dan Christensen <jdc@uwo.ca>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
the cProfile/profile modules are great for performance debugging. The
pure-python profile module has much more overhead though and the
cProfile module is recommended if it exists. This changes to import to
first try the cProfile module and then fall back to the profile
module. The cProfile/profiles modules are API compatible for all that
its worth...
If that does not exist we continue to complain as before.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than calling a function in a module, invoke offlineimap by
calling an OfflineImap object.
This removes code lying outside of objects; I prefer to keep code
within an object and provides us with a nicer Object encapsulation.
It will also ease the testing of Object functionality in unittests
when they are introduced.
Previously we would import and start Offlineimap like this:
from offlineimap import init
init.startup('6.2.0')
now we do:
from offlineimap import OfflineImap
offlineimap = OfflineImap()
offlineimap.startup('6.2.0')
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This is very excessive and a bit annoying. Output that information
only if the next line concerns a different account/thread than the
previous one. This quiets down the UI quite a bit without losing
information.
While modifying this line, use the newer Thread.name and not the as
per python doc's old syntax getName()
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Only every 10 seconds. Also fix up the documentation of that function
while at it. The Curses ui actually implements user abort it
seems. Not sure if we could do the same in the UIBase, but that is for
another time.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than poll our exitthread Queue in a non-blocking fashion and
always sleep for 1 second inbetween, simply call it in a blocking
fashion which will return immediately when a thread has exited. This
is somewhat faster as we don't do unnecessary sleeps after a thread
exited.
Do note that we need to specify some timeout value here (the 60 chosen
is pretty arbitary, but what the value exactly is, is not that
important, it could be any positive value) in order to make the
Queue.get() call work with SIGINT (cf
http://bugs.python.org/issue1360).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Straightforward reformatting of the Error output to look nicer on the eyes.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
All other uis (especially BaseUI) define as warn(self, msg, minor = 0)
just MachineUI required minor without a default. This leads the
Machine UI to error out with an exception if we pass it
ui.warn("string") which is the common thing in our code base. This
patch is therefore small but critical in fixing this UI.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Change the X-OfflineIMAP header to work around possible Exchange MAPI
table overflow problem described in
http://article.gmane.org/gmane.mail.imap.offlineimap.general/1699
(It is unknown whether this problem still exits in current
Exchange versions, but let's assume the worst.)
The X-OfflineIMAP header is neccessary with some IMAP servers to
reliably determine the UID of a new messages uploaded to the server
by using the "UID SEARCH HEADER name string" command. Since this
command compares header name and value it is sufficient to have
a unique header value and a non-unique header name.
Note that a message can have more than one X-OfflineIMAP header if
the message was copied between IMAP folders multiple times.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Python 2.4 doesn't allow try...except...finally clauses, see PEP
341. Also, yield statements inside try...finally is not allowed. The
commit changes the logic to no longer use those syntactical features.
Signed-off-by: Nicolas Sebrecht <ni.s@laposte.net>
Hello John,
i fixed some tiny bugs in offlineimap, mainly just for myself. They are
more dirty fixes than real bugfixes since I'm missing the deeper insight
into the code.
Especially the first one for Curses.py is very dirty and breaks the
scaling of the interface when the terminal size changes, but at least
the terminal is in proper state after exiting offlineimap.
In the order of appearance in the patchfile:
1. 'fixes' terminal breakage on quit of curses interface in python 2.6
to 2.6.5 (fixed since 2.6.6 http://bugs.python.org/issue7567)
2. fixes netrc password authentication
3. fixes user name querying from netrc
The patch is made for git revision 6b1cb5e036
Thanks a lot for the great application!
Best regards,
buergi
After tens of thousands of messages on the IMAP server were deleted it
takes offlineimap extremely long time (several hours of high CPU usage)
to delete them locally. It spends almost all the time modifying
LocalStatus. It processes the messages one by one, rewriting the
folder's status file in LocalStatus after each message.
It is much more efficient to save the status file only once, after
removing all the messages from the messagelist.
Deleting lots of messages now takes seconds instead of hours.
This should solve Debian bug #518093:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=518093
Signed-off-by: Michal Schmidt <mschmidt@redhat.com>
Dear All,
I have made the attached patch to try and make offlineimap a bit more
stable in challenging situations. It's extremely useful in slow
connection environments - but sometimes if one account had the wrong
password or the connection went down then unfortunately the whole
program would crash.
I have tested this on our connection and tried throwing at it just about
every situation - connection, up down, up, down again, change password,
error whilst copying one message, etc. I have been running this patch
for the last 5 days or so syncing 6 accounts at the moment... It seems
to work and stay alive nicely (even if your connection does not)...
Hope that this can go in for the next release... Please let me know if
anyone notices any problems with this...
Regards,
-Mike
-- Attached file included as plaintext by Ecartis --
-- File: submit
From 1d6777cab23637eb830031c7cab0ae9b8589afd6 Mon Sep 17 00:00:00 2001
From: mike <mike@mikelaptop.(none)>
Date: Mon, 24 Aug 2009 19:37:59 +0430
Subject: [PATCH] This patch attempts to introduce a little more error handling - e.g.
if one account has an error because of a changed password or something
that should not affect the other accounts.
Specifically:
If one sync run has an issue this is in a try-except clause - if it
has an auto refresh period the thread will sleep and try again - this
could be quite useful in the event of the connection going down for a
little while, changed password etc.
If one folder cannot be created an error message will be displayed through
the UI and the program will continue (e.g. permission denied to create a folder)
If one message does not want to copy for whatever resaon an error message
will be displayed through the UI and at least the other messages will
be copied
If one folder run has an exception then the others will still run
Dear All,
Attached is the patch that I have developed to provide maxage and
maxsize options. You can thus sync only the last x days of messages and
exclude large messages. All details in the attached git file.
Regards,
-Mike
-- Attached file included as plaintext by Ecartis --
-- File: submit
From 04fead2b46a79675a5b29de6f2b4088b9c9448e5 Mon Sep 17 00:00:00 2001
From: mike <mike@mikelaptop.(none)>
Date: Sun, 16 Aug 2009 17:00:49 +0430
Subject: [PATCH] Patch to provide maxage and maxsize account options to exclude old/large messages
This is designed to make offlineimap even better for low bandwidth connections.
maxage allows you to specify a number of days and only messages within
that range will be considered by offlineimap for the sync. This can be
useful if you would like to start using offlineimap with a large
existing account and do not want to import large archives of mail.
maxsize allows you to specify the maximum size of a message to consider so
that you can exclude messages with large attachments etc.
In both cases the cachemessagelist function of the folder was modified to ignore
messages that do not meet the criteria. If the criteria are not specified
then the existing code will be executed the same as before. If a message
does not meet the criteria it will be as though this message does not exist
- offlineimap will completely ignore it. It will not have flags updated,
it will not be deleted, it will not be considered at all.
When operating against an IMAP repository a server side search function
is used. This of course requires support for server side search.
I have tested this with either option, no options etc. against IMAP, Maildir
and Gmail. I have run variations of this patch here for the last 3 weeks or
so syncing about 4 accounts normally.
This reverts commit 220db8a77d.
From: Daniel Jacobowitz
Date: July 17, 2009
The documentation may have been unclear, but quick never worked the
way this patch changes it. My installed offlineimap does one sync
every $autorefresh minutes, and the $quick'th one is a full sync. All
the others are quick. Now it's going to do a full sync every
$autorefresh minutes and a ludicrous number of $quick syncs.
-- Daniel Jacobowitz CodeSourcery
Hello John,
The attached patch makes quick synchronization work again (according to
the documentation), so there are <quick> quick synchronizations
performed between full synchronizations.
Thanks,
Michal
* fixes rh bz #510036
Signed-off-by: Christoph Höger <choeger@cs.tu-berlin.de>
[ per jgoerzen: rh bz #510036 is Kerberos issue. also Closes: #535794 in Debian ]
Closes: #535160
Bug#535160: offlineimap fails on encfs (FUSE) due to hard linking
Package: offlineimap
Version: 6.0.3
Severity: normal
Tags: patch
Hi,
if the localrepository is stored on an encfs, offlineimap fails as follows:
OfflineIMAP 6.0.3
Copyright (C) 2002 - 2008 John Goerzen <jgoerzen@complete.org>
This software comes with ABSOLUTELY NO WARRANTY; see the file
COPYING for details. This is free software, and you are welcome
to distribute it under the conditions laid out in COPYING.
***** Processing account pleione
Copying folder structure from IMAP to Maildir
Establishing connection to tunnel:eval `keychain --quiet --eval` && ssh -q mymailhost /usr/sbin/dovecot --exec-mail imap.
Syncing Drafts: IMAP -> Maildir
Syncing INBOX: IMAP -> Maildir
----------8<----------
Syncing net: IMAP -> Maildir
Copy message 360 IMAP[net] -> Maildir[net], LocalStatus[net]
Syncing personal: IMAP -> Maildir
Thread 'Copy message 360 from net' terminated with exception:
Traceback (most recent call last):
File "/var/lib/python-support/python2.4/offlineimap/threadutil.py", line 149, in run
Thread.run(self)
File "threading.py", line 422, in run
self.__target(*self.__args, **self.__kwargs)
File "/var/lib/python-support/python2.4/offlineimap/folder/Base.py", line 282, in copymessageto
newuid = object.savemessage(uid, message, flags, rtime)
File "/var/lib/python-support/python2.4/offlineimap/folder/Maildir.py", line 198, in savemessage
os.path.join(tmpdir, messagename))
OSError: [Errno 1] Operation not permitted
Last 4 debug messages logged for Copy message 360 from net prior to exception:
----------8<----------
As encfs does not support hard linking, the rename of the temporary
message file fails.
This is the only occurrence of hard linking in offlineimap and can be
fixed by using an atomic `os.rename` instead of `os.link`+`os.unlink`.
I have been using the attached patch for more than 1.5 years and have
never again had any problems with offlineimap on encfs.
Regards,
Peter
On Tue, Apr 21, 11:19:00 -0500, John Goerzen wrote:
> I think the one loss of functionality we have here is that it doesn't
> check if a given account has already been listed before adding to the
> list. Should be a simple tweak. If you could tweak that and test, I'd
> apply a new patch.
Good catch. I attach an updated patch which I've tested and it appears
to work fine, including not syncing two accounts twice.
Thanks,
--
Jonny Lamb, UK
jonny@debian.org
From 7f348ee116bba64f7330e28d4e7b2c015910a890 Mon Sep 17 00:00:00 2001
From: Jonny Lamb <jonny@debian.org>
Date: Tue, 05 May 2009 20:45:17 +0100
Subject: [PATCH] Respect order of general.accounts config setting.
This makes the order of account synchronisation the same as the order of
the general.accounts setting by using a list instead of a dict, which
was actually pointless as the value of each dict item was never even
looked at.
Signed-off-by: Jonny Lamb <jonny@debian.org>
Patch from Eric Dorland
Closes: #470875
From: Wouter Verhelst
Subject: kerberos authentication works only the first time
Date: Fri, 14 Mar 2008 09:28:37 +0100
Package: offlineimap
Version: 5.99.8
Severity: normal
Hi,
I have the "autorefresh" configuration option specified in my
.offlineimaprc, and am now using the kerberos authentication.
However, this kerberos authentication seems to work only the first
time
offlineimap tries to fetch mails. The next time, it fails with this
output:
Thread 'Account sync Test' terminated with exception:
Traceback (most recent call last):
File "/var/lib/python-support/python2.4/offlineimap/threadutil.py",
line 153, in run
Thread.run(self)
File "/usr/lib/python2.4/threading.py", line 422, in run
self.__target(*self.__args, **self.__kwargs)
File "/var/lib/python-support/python2.4/offlineimap/accounts.py",
line 119, in syncrunner
self.sync()
File "/var/lib/python-support/python2.4/offlineimap/accounts.py",
line 148, in sync
remoterepos.syncfoldersto(localrepos, [statusrepos])
File
"/var/lib/python-support/python2.4/offlineimap/repository/Base.py",
line 135, in syncfoldersto
srcfolders = src.getfolders()
File
"/var/lib/python-support/python2.4/offlineimap/repository/IMAP.py",
line 192, in getfolders
listresult = imapobj.list(directory =
self.imapserver.reference)[1]
File "/usr/lib/python2.4/imaplib.py", line 469, in list
typ, dat = self._simple_command(name, directory, pattern)
File "/usr/lib/python2.4/imaplib.py", line 1028, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "/usr/lib/python2.4/imaplib.py", line 787, in _command
raise self.error(
error: command LIST illegal in state NONAUTH
Paul Hinze <paul.t.hinze@gmail.com> on 2008-12-22 at 19:16:
> I did a bit of debugging and asking around in #python and it turns out
> that because Maildir.py uses a conditional import of md5 from either
> hashlib or the md5 module, the md5.new() call is not always correct.
> The attached patch should fix this problem.
Thanks to Daniel Rall for pointing out that the attachment got stripped
off my last message. Patch follows.
Paul
----
From e01fdfbf62c61c31b514e826e6cc00fb98d88808 Mon Sep 17 00:00:00 2001
From: Paul Hinze <paul.t.hinze@gmail.com>
Date: Mon, 22 Dec 2008 18:42:44 -0600
Subject: [PATCH] Fixing md5 import when hashlib is available
Thanks to habnabit and Rhamphoryncus in #python for help debugging
Here's the way I'd like to use offlineimap on my laptop:
1. Have a regular cron job running infrequently. The cron job
checks to see
if I'm online, plugged in, and that no other copy of offlineimap is
running. If
all of these conditions are satisfied, it runs offlineimap just once:
"offlineimap -o -u Noninteractive.Quiet"
2. When I start up mutt, I do it by calling a wrapper script that
delays
until cron-started copies of offlineimap have finished, then starts
offlineimap
on its regular, stay-alive and keep checking schedule. When I quit
mutt, the
wrapper script tells offlineimap to stop.
This way I get frequent regular checks while I have mutt running, but
I don't
waste my battery/cpu checking frequently for mail when I'm not
interested in
it.
To make this work, though, it'd be nicer if it were easier to tell
offlineimap,
from the outside, things like "terminate cleanly now" and "when you've
finished
synching, then terminate instead of sleeping and synching again."
OK, to put my money where my mouth is, I attach two patches against
offlineimap
6.0.3.
The first, "cleanup.patch", cleans up a few spots that tend to throw
exceptions
for me as offlineimap is exiting from a KeyboardInterrupt.
The second adds signaling capabilities to offlineimap.
* sending a SIGTERM tells offlineimap to terminate immediately but
cleanly,
just as if "q" had been pressed in the GUI interface
* sending a SIGUSR1 tells every account to do a full sync asap: if
it's
sleeping, then wake up and do the sync now. If it's mid-sync, then
re-synch
any folders whose syncing has already been started or completed, and
continue
to synch the other, queued but not-yet-synched folders.
* sending a SIGHUP tells every account to die as soon as it can (but
not
immediately: only after finishing any synch it's now engaged in)
* sending a SIGUSR2 tells every account to do a full sync asap (as
with
SIGUSR1), then die
It's tricky to mix signals with threads, but I think I've done this
correctly.
I've been using it now for a few weeks without any obvious
problems. But I'm passing it
on so that others can review the code and test it out on their
systems. I developed the
patch when I was running Python 2.5.2, but to my knowledge I don't use
any Python 2.5-specific
code. Now I'm using the patch with Python 2.6.
Although I said "without any obvious problems," let me confess that
I'm
seeing offlineimap regularly choke when I do things like this: start
up
my offlineimap-wrapped copy of mutt, wait a while, put the machine to
sleep (not sure if offlineimap is active in the background or idling),
move to a different spot, wake the machine up again and it acquires a
new network, sometimes a wired network instead of wifi. Offlineimap
doesn't like that so much. I don't yet have any reason to think the
problems here come from my patches. But I'm just acknowledging them,
so
that if others are able to use offlineimap without any difficulty in
situations like I described, then maybe the fault is with my patches.
From Jim Pryor
The first, "cleanup.patch", cleans up a few spots that tend to throw
exceptions for me as offlineimap is exiting from a KeyboardInterrupt.
Frankly, the original code doesn't even make much sense, and
moreover it's not forward compatible with python 2.6, and
furthermore:
DeprecationWarning: socket.ssl() is deprecated.
So, this commit is just a temporary fix.
Signed-off-by: Michael Witten <mfwitten@mit.edu>
module threading should be taken out back and shot.
Condition.wait() is a busywait loop that has negative implications for
battery consumption on laptops.
Queue.get() in blocking mode blocks SIGINT from being delivered.
Argh, argh, argh.
Closes: #493801.
This should improve power-management abilities some more
The catch is that we can't wait any longer for the kathread to
terminate. We were waiting for this in some cases. This is probably
not a big deal.
fixes deb#434074
fixes#66
This patch adds sslclientcert and sslclientkey configuration
options which are passed through to imaplib in order to allow
the use of client certificates for authentication.
Tue Nov 13 14:44:17 CST 2007 Mark Hymers <mhy@debian.org>
Noted thie code:
statusfolder =
statusrepos.getfolder(remotefolder.getvisiblename().\
replace(remoterepos.getsep(),
statusrepos.getsep()))
in accounts.py. Should have been using the sep of the LocalStatus all
along.
refs deb#479798, #68
fixes#20
04/27/2008 09:34 AM - Vincent Beffara
* File darwin.patch added
OK, I finally installed an IMAPS server on my iMac, so I could test a
little further. For some reason, fixing the read() method in
imaplibutil.py did not seem to work (it hung on connecting to the
server) - however, modifying the file imapserver.py similarly to the
non-SSL case fixed the crash.
I also reduced the chunk size to 100k instead of 1M, as 1M seemed to
still trigger the memory error in some cases. Ah, and I added a
platform test, so that the patch does essentially nothing on
non-Darwin machines ...
So, still no guarantee, but the attached patch works for me. Any
comments ?
(If noone here screams in horror at my code, I will include the patch
into the Fink package and see what happens there.)
I created new IMAP folders on my server. In some moment I found offlineimap
crashed. The reason were empty LocalStatus files. From looking at the code I
have no explanation how those files came into existence.
* John Goerzen (jgoerzen@complete.org) wrote:
> tages 359213 help
> thanks
>
> Hi folks,
>
> Unfortunately, I lack both the expertise to add Kerberos
> authentication and a way to test this. A patch would be very welcome
> :-)
Well it was a bit unclear but there's a patch for offlineimap attached
to Guido's message:
http://bugs.debian.org/cgi-bin/bugreport.cgi?msg=10;filename=support-GSSAPI-via-pykerberos.diff;att=1;bug=359213
I've tried it out and it works great! Except that if the Kerberos
layer throws an exception it wouldn't fail over to plain
authentication properly. Attached is a slightly reworked patch that I
tested it in the following situations:
1. python-kerberos uninstalled, with kerberos credentials.
Prompted for password authentication as expected.
2. python-kerberos installed, with kerberos credentials.
Keberos authentication was successful and my mail was fetched
without password prompting. Yeehaw!
3. python-kerberos installed, with no kerberos credentials.
Prompted for password authentication as expected.
I think those are all the interesting situations. So I think this is
ready to go in, with a Suggest on python-kerberos (>=
1.0+mk080218-1).
--
Eric Dorland <eric@kuroneko.ca>
ICQ: #61138586, Jabber: hooty@jabber.com
refs deb#469598
Bug report received in Debian. User is having occasional trouble with Gmail
driver crashing, indicates problem may have started after switching to Gmail.
Gmail driver merged to tree in 81b86fb74 on 2008-01-03 and recently released
in beta tarballs.
BT shows:
File "/var/lib/python-support/python2.4/offlineimap/folder/Gmail.py", line 102, in processmessagesflags
attributehash = imaputil.flags2hash(imaputil.imapsplit(result)[1])
File "/var/lib/python-support/python2.4/offlineimap/imaputil.py", line 88, in imapsplit
for i in range(len(imapstring)):
TypeError: len() of unsized object
imap debug log shows:
imap: 29:58.16 > BEAL75 UID STORE 4887 +FLAGS (\Deleted)
imap: 29:58.47 < BEAL75 OK Success
imap: 29:58.47 matched r'(?P<tag>BEAL\d+) (?P<type>[A-Z]+) (?P<data>.*)' => ('BEAL75', 'OK', 'Success')
imap: imapsplit() called with input: None
imap: imapsplit() got a non-string input; working around.
looking at code for Gmail.py processmessagesflags, comments from Ricardo
indicate it was copied from the same function in folder/IMAP.py.
However, folder/IMAP.py has checks for this, added 2002-07-12 in
5342dacc & 817a10ce. Suspect that Gmail author believed those checks
superfluous for Gmail.
Copied checks from folder/IMAP.py to folder/Gmail.py.
This reverts commit 71a76d9a61.
Zak Smith reported a problem at:
self.processmessagesflags('+', uidlist, flags)
File "/usr/lib/python2.5/site-packages/offlineimap/folder/IMAP.py",
line 372, in processmessagesflags
myrights = imapobj.myrights(self.getfullname())[1][0].split()[1]
IndexError: list index out of range
Conflicts:
offlineimap/folder/IMAP.py
Added WrappedIMAP4_SSL class to help fix up performance of SSL
Standard imaplib.py is really bad with this, since it reads one
character at a time.
Reported by Aaron Kaplan at
http://lists.complete.org/offlineimap@complete.org/2008/01/msg00012.html.gz
He wrote:
I just noticed that the version of offlineimap I've been using
(3.99.17) is well over four years old. How time flies. I haven't
had any problems with it, but out of curiosity I decided to pull in
5.99.2 from the fedora repository. It turns out to take
consistently over twice as long as the old version to sync the same
account. Is this expected?
He tracked it down at
http://lists.complete.org/offlineimap@complete.org/2008/02/msg00012.html.gz
The following changeset is the one responsible for the difference in
speed I was noticing between the imaplib.py that was packaged with
older versions of offlineimap and the one that comes with python:
* /offlineimap/head: changeset 169
More optimizations -- this time fix readline() to not work
character-by-character!
closes#22
from pistore in OfflineIMAP #22:
When an IMAP flag update is performed for multiple messages, some IMAP
servers (e.g. Exchange) return the UID attribute only for some of the
FETCH untagged responses, as shown in the following log:
21:19.04 > DCKF8 UID STORE 66050,50613,52164,40043,40055,25874 +FLAGS
(\Deleted)
21:19.36 < * 35 FETCH (FLAGS (\Seen \Deleted) UID 25874)
21:19.36 < * 321 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 322 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 560 FETCH (FLAGS (\Seen \Deleted))
21:19.57 < * 581 FETCH (FLAGS (\Seen \Deleted) UID 52164)
21:19.62 < * 1022 FETCH (FLAGS (\Seen \Deleted))
21:19.62 < DCKF8 OK STORE completed.
Function IMAPFolder.processmessagesflags is able to manage the servers
which return the UID and the servers which do not return it, but is
not able to deal with the mixed behavior shown above.
The problem is that the fragment of function
IMAPFolder.processmessagesflags that handles the responses with UID
attribute uses variable flags to store the list of flags of the
message in the IMAP format ("flags = attributehashFLAGS?"), while the
fragment that handles the responses without UID expects variable
"flags" to contain the list of modified flags passed to the function
in Maildir format ("self.messagelist[uid]flags?.append(flag)").
As a consequence, the wrong list of flags is used for the messages
without UID, leading to the addition of "strange" flags to the Maildir
messages:
Syncing messages IMAP[INBOX] -> Maildir[.]
Adding flags to 4 messages on Maildir[.]
Adding flags e to 4 messages on Maildir[.]
Adding flags d to 4 messages on Maildir[.]
Adding flags ) to 4 messages on Maildir[.]
Adding flags ( to 4 messages on Maildir[.]
Adding flags l to 4 messages on Maildir[.]
Adding flags n to 4 messages on Maildir[.]
Adding flags t to 4 messages on Maildir[.]
Adding flags \ to 4 messages on Maildir[.]
Adding flags D to 4 messages on Maildir[.]
Deleting flags T to 4 messages on Maildir[.]
Adding flags to 4 messages on LocalStatus[.]
Adding flags e to 4 messages on LocalStatus[.]
Adding flags d to 4 messages on LocalStatus[.]
Adding flags ) to 4 messages on LocalStatus[.]
Adding flags ( to 4 messages on LocalStatus[.]
Adding flags l to 4 messages on LocalStatus[.]
Adding flags n to 4 messages on LocalStatus[.]
Adding flags t to 4 messages on LocalStatus[.]
Adding flags \ to 4 messages on LocalStatus[.]
Adding flags D to 4 messages on LocalStatus[.]
Deleting flags T to 4 messages on LocalStatus[.]
Fix: use a different variable to store IMAP flags when managing
messages corresponding to responses with UID attribute, e.g.:
*** IMAP.py.orig Wed Aug 22 18:23:17 2007
--- IMAP.py Wed Aug 22 18:22:38 2007
*************** class IMAPFolder(BaseFolder):
*** 340,348 ****
if not ('UID' in attributehash and 'FLAGS' in
attributehash):
# Compensate for servers that don't return a UID
attribute.
continue
! flags = attributehash['FLAGS']
uid = long(attributehash['UID'])
! self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(flags)
try:
needupdate.remove(uid)
except ValueError: # Let it slide if it's not
in the list
--- 340,348 ----
if not ('UID' in attributehash and 'FLAGS' in
attributehash):
# Compensate for servers that don't return a UID
attribute.
continue
! lflags = attributehash['FLAGS']
uid = long(attributehash['UID'])
! self.messagelist[uid]['flags'] =
imaputil.flagsimap2maildir(lflags)
try:
needupdate.remove(uid)
except ValueError: # Let it slide if it's not
in the list
02/03/08 14:04:35 changed by js
* attachment flags-fix.patch added.
Delete 02/03/08 14:05:24 changed by js
Unfortunately I have to fetch some of my mail from an Exchange server
(Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.7638.1)
and I can confirm that the analysis of the problem is correct, and the
patch given here fixes the problem.
Looking at the code of the processmessagesflags() method I think it
generally is a bug that the "flags" parameter is reused as a local
variable, since the final "for uid in needupdate:" loop needs the
original value of "flags". This only worked by accident.
I'm attaching a unidiff version of the patch which applies cleanly
against Debian unstable's offlineimap 5.99.4.
New repository/folder classes to support "real deletion" of messages
thorugh Gmail's IMAP interface: to really delete a message in Gmail,
one has to move it to the Trash folder, rather than EXPUNGE it.
This patch maneuvers around python imaplib's mysterious read-only detection
algorithm and correctly calls the UI's deletetoreadonly(), when trying to
delete/expunge in a mailbox without having the necessary rights.
This involves several changes at different places:
- syncfoldersto() takes statusfolder as an argument, and returns the
list of new folders and the list of folders that should be ingnored,
typically those that were deleted. Warns the user about folders that
are present only on one side and are not synced.
- syncfoldersto() is called both ways, and on folder creation
forgetfolders() is used to rebuild the list and take the new creation
into account. Probably not the most efficient, since it involves
talking to the IMAP server again, but it will rarely be used anyway.
- Locally created folders are treated separately in the synchronization,
namely the local messages are uploaded and then the normal sync still
occurs. If the same folder is created on both sides and contains
messages on both sides, a two-way sync occurs.
fixes deb#439384
From: martin f krafft
Subject: race condition in Maildir writing
The offlineimap Maildir code checks for file existence and then
opens a file. That's open to a race condition. It's better to open
the file and fail if it already exists. The following patch does
this. It catches OSError 17 (file exists) and re-raises all others.
I'll leave it up to you to decide whether this is appropriate.
This involves several changes at different places:
- syncfoldersto() takes statusfolder as an argument, and returns the
list of new folders and the list of folders that should be ingnored,
typically those that were deleted. Warns the user about folders that
are present only on one side and are not synced.
- syncfoldersto() is called both ways, and on folder creation
forgetfolders() is used to rebuild the list and take the new creation
into account. Probably not the most efficient, since it involves
talking to the IMAP server again, but it will rarely be used anyway.
- Locally created folders are treated separately in the synchronization,
namely the local messages are uploaded and then the normal sync still
occurs. If the same folder is created on both sides and contains
messages on both sides, a two-way sync occurs.
From: "Mark A. Hershberger"
https://bugs.launchpad.net/ubuntu/+source/offlineimap/+bug/96710
the locked() method isn't implemented for non-interactive UIs, so
exceptions are thrown on cron jobs. Ubuntu's new apport catches these
and ? well, you get the idea.
patch provided.
fixes deb#433732
Date: Sun, 30 Sep 2007 13:54:56 -0400
From: Daniel Jacobowitz <drow@false.org>
To: offlineimap@complete.org
Subject: Assorted patches
Here's the result of a lazy Sunday hacking on offlineimap. Sorry for
not breaking this into multiple patches. They're mostly logically
independent so just ask if that would make a difference.
First, a new -q (quick) option. The quick option means to only update
folders that seem to have had significant changes. For Maildir, any
change to any message UID or flags is significant, because checking
the flags doesn't add a significant cost. For IMAP, only a change to
the total number of messages or a change in the UID of the most recent
message is significant. This should catch everything except for
flags changes.
The difference in bandwidth is astonishing: a quick sync takes 80K
instead of 5.3MB, and 28 seconds instead of 90.
There's a configuration variable that lets you say every tenth sync
should update flags, but let all the intervening ones be lighter.
Second, a fix to the UID validity problems many people have been
reporting with Courier. As discussed in Debian bug #433732, I changed
the UID validity check to use SELECT unless the server complains that
the folder is read-only. This avoids the Courier bug (see the Debian
log for more details). This won't fix existing validity errors, you
need to remove the local status and validity files by hand and resync.
Third, some speedups in Maildir checking. It's still pretty slow
due to a combination of poor performance in os.listdir (never reads
more than 4K of directory entries at a time) and some semaphore that
leads to lots of futex wake operations, but at least this saves
20% or so of the CPU time running offlineimap on a single folder:
Time with quick refresh and md5 in loop: 4.75s user 0.46s system 12%
cpu 41.751 total
Time with quick refresh and md5 out of loop: 4.38s user 0.50s system
14% cpu 34.799 total
Time using string compare to check folder: 4.11s user 0.47s system 13%
cpu 34.788 total
And fourth, some display fixes for Curses.Blinkenlights. I made
warnings more visible, made the new quick sync message cyan, and
made all not explicitly colored messages grey. That last one was
really bugging me. Any time OfflineIMAP printed a warning in
this UI, it had even odds of coming out black on black!
Anyway, I hope these are useful. I'm happy to revise them if you see
a problem.
--
Daniel Jacobowitz
CodeSourcery
patch from Mike Gerber
Two times today I have found my offlineimap to have died with this same
situation. It appears as if certain types of messages (both spam in my
situation), cause offlineimap to choke. When it does it cannot proceed.
This means that when I run offlineimap, it pulls in messages from some
folders, then it hits the folder with the bad message and dies, leaving
undownloaded mail on the server. The only fix to this problem is to find
the problem message on the server and remove it by hand. This isn't such
a huge deal for me, since I run the server, but other people have to
come to me to ask me to delete these messages, and until I do they
cannot download their email.
I have captured the output by running script during one of these
incidents, this has been attached. Additionally, I have also attach the
problematic message.
The patch seems to work for me, might need some Python wizard and better
testing, though.
fixes deb#396443
r[1] is a list. In case it contains more than one 'str' they are concatenated
using '. '. In processmessagesflags the join is tested, for savemessagesflags I
assume that it is also needed.
I have tested this and Dovecot no longer beats offlineimap
to the punch. (-:
I achieved that by keeping the renames in tmp/ until it finally does
one last rename in new/ or cur/.
fsync the Maildir file, its final directory when writing a new message.
fsync the localstatus file and its final directory when writing the
local status cache.
This should reduce duplication in the event of hardware trouble.
fixes#8
see thread at http://lists.complete.org/offlineimap@complete.org/2007/03/threads.html.gz
* Reduced the number of parameters passed to ui.validityproblem() because they were all just method-calls to the folder object, which is already passed as the first parameter (reduction of unnecessary complexity).
* Improved the diagnostic message for an 'UID validity problem' by including the name of the repository in which the folder resides; previously it was not possible to determine from the diagnostic alone on which side the problem was.
* Reduced the number of parameters passed to ui.validityproblem() because they were all just method-calls to the folder object, which is already passed as the first parameter (reduction of unnecessary complexity).
From: Peter Colberg
Hello,
using offlineimap with the preauthtunnel option to start a remote
IMAP daemon via ssh, a zombie process is left behind during the
autorefresh sleep period if not holding the connection open.
Above behaviour is a result of using os.popen2 to spawn the tunnel
process, which makes it impossible waiting for the child process
to terminate when shutting down the tunnel.
The patch included below fixes the issue by employing the Popen
class from the subprocess module, which seems to be the preferred
way to spawn processes and connect to their pipes in any case (at
least since python version 2.4.4, which fixes a memory leak in the
subprocess module).
Regards,
Peter
fixes deb#410730
From: Mark Brown <broonie@sirena.org.uk>
Currently offlineimap will attempt to connect to the first address
returned by addrinfo() for the remote system and will fail if that fails
even if another result would have worked. This is particularly common
when the remote system supports both IPv4 and IPv6 - a laptop may in
some environments have no routable IPv6 connectivity so if the IPv6
address is returned first the connect will fail even though IPv4 would
have worked.
This is actually a bug in imaplib, a copy of which is included in
offlineimap. This patch fixes the problem by looping over all the
results returned by getaddrinfo(). Unfortunately it mangles the error
reporting slightly since I couldn't work out how to raise an appropriate
exception, though given that that that was a Python backtrace there was
work to do there anyway.
Note that I have only tested the SSL case.
It looks like I accidentally recorded the wrong version of Curses.py --
originally this code was there, but I moved it over to UIBase so it would
cover the TTY UI also.
From Ben Kibbey
hello,
Attached is a patch to enable evaluation of account credentials with the
remotehosteval, remoteusereval and remotepasseval configuration options.
I needed this because rather than change all my other programs
configuration settings when I change, say a password, I store them in a
file. So I call a function in pythonfile which parses the credential
file and returns the wanted info. Not really very well tested, but not
complex either. Offlineimap is great, thanks.
From: Ben Kibbey
Subject: Re: Removed restoratime from OfflineIMAP
On Wed, May 03, 2006 at 10:08:35PM -0500, John Goerzen wrote:
> Hi Ben,
>
> Thanks for your restoreatime patch.
>
> However, I have received this bug report:
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=365933
>
> After looking at the problem, here's what's going on.
>
> The person is using IMAP as the local repository as well.
>
> You really need to move the atime save and restore code from accounts.py
> into the repository/Maildir.py. Then, for any new call you add to the
> Maildir repository (that will be called from outside Maildir.py), you
> need to add a corresponding default function to repository/Base.py, and
> also make sure that on folders (such as IMAP) where atime restoration
> makes no sense, no error is generated.
>
> Let me know if that doesn't make sense to you. If you get it fixed, I'd
> be happy to re-apply it to a future version of OfflineIMAP.
>
> -- John Goerzen
>
Attached is a new diff that should work though not really tested
(v4.0.14). In repository/Base.py restore_atime() will call
self.restore_folder_atimes() only if the folder type is Maildir. Let me
know if it has any more problems.
The attached patch adds syncing the INTERNALDATE of IMAP folders with
the mtime of messages in maildir folders.
I want this to happen, because I'm running a dovecot over the maildirs
synced by offlineimap, and that uses the mtime as the INTERNALDATE.
When using mutt to view messages I generally sort based on the received
date, which for IMAP folders is the INTERNALDATE.
Since this is the first real coding I've done in Python the patch may
need to be cleaned up some, but it's working pretty well for me. I've
added new messages to each side, and the received date has been
preserved going both ways.
From: Ben Kibbey <bjk@luxsci.net>
Attached is a patch to restore the atime of Maildir folders after
syncing. It can be enabled via the 'restoreatime' boolean in the
configuration file. I needed this because offlineimap is run after a
fetchmail and my mail checker breaks.
Patch from Nikita V. Youshchenko
From: "Nikita V. Youshchenko"
To: Debian Bug Tracking System
Subject: offlineimap: exception on mail with broken headers (+fix)
Date: Wed, 24 Aug 2005 13:41:08 +0400
Package: offlineimap
Version: 4.0.10
Severity: normal
Tags: patch
Recently I've got an exception (see below) while using offlineimap.
Exception was probably caused by invalid Date header of (likely spam)
message:
Date: Sat, 20 Aug 2005 4294967295:43:18 -0700
From: "Nikita V. Youshchenko"
I trued to use offlineimap and found that while being quite fast on
small folders, it takes up to several minutes (of 100% busy CPU and
almost no network traffic) to sync a folder with 2000+ messages.
While looking into the code, I found why this happens.
In folder/Base.py, in method BaseFolder.syncmessagesto_copy(),
dest.getmessagelist() is called inside a loop, while being a loop
invariant. Similar thing happens in BaseFolder.syncmessagesto_delete()
for self.getmessagelist().
This causes quadratic complexity over folder size.
Moving these calls out of loops make large folder sync fast (several
seconds instead of several minutes for folder with 2000 messages on
700MHz P3).
Applied patch from Joerg Wendland <joergland@debian.org> to use
APPENDUID result from mail servers that provide it. Closes: #198772.
Resolves: [debian.org #198772]
Added a "force" option to imapserver/select to force a reloading of a
folder. Per [complete.org #67], when cachemessagelist() was called on
an object that was cached from a previous run, it would not re-issue
the select(). Closes: [complete.org #67]
Added -l option. Updated documentation for it. Changed _msg to
_display override in UI modules. Renamed "doc" to "docs" target in
Makefile to avoid conflicting with a subdir.
- offlineimap (3.99.17) unstable; urgency=low
- Fixed two potential obscure race conditions in folder/Maildir.py. +
Condition 1 involved the gettimeseq() function. This function
accesses per-module variables but does not have a lock. It may have
been possible for this to have been called in such a way that
timeseq was not properly updated. + Condition 2 involved the call to
gettimeseq(). Since the timeseq is based on the system clock, we now
use the time as reported inside timeseq() rather than outside. This
way, we can be assured that the same value is in use both places.
- Added debug code to savemessage in folder/Maildir.py to try to track
down a mysterious 0-length file bug. -- John Goerzen
<jgoerzen@complete.org> Tue, 6 May 2003 09:21:38 -0500
- Added some significant debug code to folder/IMAP.py when saving a
new message with APPEND. This should make it easier to track down
bugs both in OfflineIMAP and in mail servers that implement this
poorly.
- Fixed adding of X-OfflineIMAP header when the message starts out
with no headers. (This should not generally occur.) This should help
with some "invalid literal for long()" problems.
When sep was /, the new Maildir support code would recursively try to
scan ., resulting in huge paths and an eventual crash. Fixed with a
one-line patch to Maildir.py. Closes: [complete.org #60] Sergei, The
below diff is going into 3.99.16. You can apply it to 3.99.15 and it
should work for you now. Please let me know. (Ignore any patch errors
for debian/changelog). Thanks for the report.
Raise an exception when the status area is locked. This will cause UIs
to go through their normal exception handling code. In particular, for
the Curses.Blinkenlights interface, the Curses module will be stopped
and the error message will be printed on the console. Previously, this
error message would not have been visible. Closes: #185709.
- Slight renaming in offlineimap.conf.minimal to clarify things.
- Documentation updated with information about new features. Closes:
#189771. + Described IMAP-IMAP syncing + Updated minimal example
with new offlineimap.conf.minimal + Updated UID information. Added
link to recent mailing list discussion. + Described KMail syncing,
which now works. + Added link to mailing list archives.
- Now checks that SELECT succeeded when entering a folder.
- Verifies that folders listed on folderincludes actually exist by
trying to enter them. Thus, if they do not exist, they can be
created on the first run.
Fixed line-ending code to deal with files with mixed \n and \r\n
codes. This is a rare case, but now is more onerous because we now
have to find headers.
Due to possibly having one account sleep while another is reading a
password, and other tricky situations, support for nice updating and
cancelling of a sleep in TTY.TTYUI has been removed. However, this is
not going to be a huge problem because the new Curses Blinkenlights
interface has this support, and does it a lot better than TTY.TTYUI
ever could have.
Reworked the canvas. Before, problem was the label and buttons to the
right of the lights would make the window too wide. When the button
got added, the window would get even wider. That was because the
canvas would not shrink. My workaround is to use a separate canvas for
each light. Seems to be OK here....
Fixed up the VerboseUI for new account system. All that really needed
updating with the "Sync immediately" button, to cope with syncing
different accounts at different times. It's better now.
More locking updates. Introduced a new MultiLock to threadutil. This
lock will let a single thread acquire the same lock more than once,
keeping track of how many times this happens, and will release the
actual lock only when the lock's lock count gets back to zero. By
using MultiLock, various functions in Curses.py and Blinkenlights.py
no longer need to pass around to other functions a parameter
indicating whether or not a lock should be obtained. This was a large
cause of complexity and errors, which is now eliminated. Everything
seems to be working properly wrt locking at this point. The
Curses.Blinkenlights interface has achieved basic working
functionality.
Updated the mbnames recorder to bring it back up-to-date with the new
account-centric system. It will now gather reports from account sync
threads, and when it has all that it's supposed to, it'll write out
the file.
Added some temporary debug code to help weed out a few race conditions
with the curses Blinkenlights interface. Think I've finally got it.
I'm leaving the debugging code in for now, though, to help in case
there are future problems.
More progress at debugging. The curses blinkenlights is now working
well, though it still has an occasional tendency to corrupt the light
display with comments from the log. I suspect a locking problem --
need to be more strict with iolock I suspect. Updated various modules
to register the threads' account names, etc.
Changed to a more account-centric behavior. The refresh time is now a
per-account variable. Implemented new account classes. User interfaces
must now be updated to take advantage of this.
Now properly handles folder names that contain parenthesis. Used patch
from Kyler Laird in http://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=173895. Closes: #173895.
Alter handling of messages flagged for deletion -- no longer
automatically delete them if expunge is 0. In Maildir folder, we will
now ignore the T flag entirely, and just pass it back and forth with
IMAP.
Moved password promting into imapserver.py. Passwords are now asked
for on-demand and typos will no longer crash the program (the user
will be re-prompted). Closes: #162672.
When an exception occurs, OfflineIMAP will attempt to print the last
50 debug messages, whether or not debugging was enabled for this
session. This way, even unexpected and non-repeatable errors stand a
chance of getting a more detailed log.
- Moved some code from offlineimap/init.py to new file
offlineimap/syncmaster.py to help dileneate between code that
performs different functions.
- Moved threadexited from offlineimap/init.py to
offlineimap/threadutil.py.
-d now takes a parameter to specify what kind of debugging to do.
imaplib now does debugging through the UI system. UIBase now has a
debugging process.
- Oops, incomplete commit from the last one:
- If a given Maildir folder is new, remove the associated local status
cache file, if any. That way, there will not be any chance of
propogating hordes of deletes and adds based on old status data.
If a given Maildir folder is new, remove the associated local status
cache file, if any. That way, there will not be any chance of
propogating hordes of deletes and adds based on old status data.
Better handling of read-only folders. We will now warn if there is a
change, but not propogate it. New config variable ignore-readonly can
suppress the warnings. This fixes [complete.org #10] and, for Debian,
Closes: #154769. changelog: noted the change IMAP.py: trap
imapobj.readonly more often UIBase.py: new methods to handle the
warnings offlineimap.conf: new ignore-readonly variable.
Backed out check for . in account names for now. Will put it back in
when we have a consensus on what exactly to do. Doubt that anyone has
a foldername that would conflict with Blinkenlights anyway.
This is release 3.1.1. Fixed a typo in the manual: Tk.TKUI -> Tk.TkUI.
Noted the new version and release time in the changelog and in
version.py. Rebuilt the documentation. Feed the cats, watered the
plants, backed up /dev/null.
- Modified imaputil.py and folder/Maildir.py to run faster. Eliminated
many regular expressions; pre-compiled many others.
- Fixed threadutil's exitnotifyloop to always handle threads in the
order they exited, rather than sometimes in the inverse order. This
way, make sure to handle thread's exception messages before a thread
exited.
