I have tested this and Dovecot no longer beats offlineimap
to the punch. (-:
I achieved that by keeping the renames in tmp/ until it finally does
one last rename in new/ or cur/.
fsync the Maildir file, its final directory when writing a new message.
fsync the localstatus file and its final directory when writing the
local status cache.
This should reduce duplication in the event of hardware trouble.
fixes#8
see thread at http://lists.complete.org/offlineimap@complete.org/2007/03/threads.html.gz
* Reduced the number of parameters passed to ui.validityproblem() because they were all just method-calls to the folder object, which is already passed as the first parameter (reduction of unnecessary complexity).
* Improved the diagnostic message for an 'UID validity problem' by including the name of the repository in which the folder resides; previously it was not possible to determine from the diagnostic alone on which side the problem was.
* Reduced the number of parameters passed to ui.validityproblem() because they were all just method-calls to the folder object, which is already passed as the first parameter (reduction of unnecessary complexity).
From: Peter Colberg
Hello,
using offlineimap with the preauthtunnel option to start a remote
IMAP daemon via ssh, a zombie process is left behind during the
autorefresh sleep period if not holding the connection open.
Above behaviour is a result of using os.popen2 to spawn the tunnel
process, which makes it impossible waiting for the child process
to terminate when shutting down the tunnel.
The patch included below fixes the issue by employing the Popen
class from the subprocess module, which seems to be the preferred
way to spawn processes and connect to their pipes in any case (at
least since python version 2.4.4, which fixes a memory leak in the
subprocess module).
Regards,
Peter
fixes deb#410730
From: Mark Brown <broonie@sirena.org.uk>
Currently offlineimap will attempt to connect to the first address
returned by addrinfo() for the remote system and will fail if that fails
even if another result would have worked. This is particularly common
when the remote system supports both IPv4 and IPv6 - a laptop may in
some environments have no routable IPv6 connectivity so if the IPv6
address is returned first the connect will fail even though IPv4 would
have worked.
This is actually a bug in imaplib, a copy of which is included in
offlineimap. This patch fixes the problem by looping over all the
results returned by getaddrinfo(). Unfortunately it mangles the error
reporting slightly since I couldn't work out how to raise an appropriate
exception, though given that that that was a Python backtrace there was
work to do there anyway.
Note that I have only tested the SSL case.
It looks like I accidentally recorded the wrong version of Curses.py --
originally this code was there, but I moved it over to UIBase so it would
cover the TTY UI also.
From Ben Kibbey
hello,
Attached is a patch to enable evaluation of account credentials with the
remotehosteval, remoteusereval and remotepasseval configuration options.
I needed this because rather than change all my other programs
configuration settings when I change, say a password, I store them in a
file. So I call a function in pythonfile which parses the credential
file and returns the wanted info. Not really very well tested, but not
complex either. Offlineimap is great, thanks.
From: Ben Kibbey
Subject: Re: Removed restoratime from OfflineIMAP
On Wed, May 03, 2006 at 10:08:35PM -0500, John Goerzen wrote:
> Hi Ben,
>
> Thanks for your restoreatime patch.
>
> However, I have received this bug report:
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=365933
>
> After looking at the problem, here's what's going on.
>
> The person is using IMAP as the local repository as well.
>
> You really need to move the atime save and restore code from accounts.py
> into the repository/Maildir.py. Then, for any new call you add to the
> Maildir repository (that will be called from outside Maildir.py), you
> need to add a corresponding default function to repository/Base.py, and
> also make sure that on folders (such as IMAP) where atime restoration
> makes no sense, no error is generated.
>
> Let me know if that doesn't make sense to you. If you get it fixed, I'd
> be happy to re-apply it to a future version of OfflineIMAP.
>
> -- John Goerzen
>
Attached is a new diff that should work though not really tested
(v4.0.14). In repository/Base.py restore_atime() will call
self.restore_folder_atimes() only if the folder type is Maildir. Let me
know if it has any more problems.
The attached patch adds syncing the INTERNALDATE of IMAP folders with
the mtime of messages in maildir folders.
I want this to happen, because I'm running a dovecot over the maildirs
synced by offlineimap, and that uses the mtime as the INTERNALDATE.
When using mutt to view messages I generally sort based on the received
date, which for IMAP folders is the INTERNALDATE.
Since this is the first real coding I've done in Python the patch may
need to be cleaned up some, but it's working pretty well for me. I've
added new messages to each side, and the received date has been
preserved going both ways.
From: Ben Kibbey <bjk@luxsci.net>
Attached is a patch to restore the atime of Maildir folders after
syncing. It can be enabled via the 'restoreatime' boolean in the
configuration file. I needed this because offlineimap is run after a
fetchmail and my mail checker breaks.
Patch from Nikita V. Youshchenko
From: "Nikita V. Youshchenko"
To: Debian Bug Tracking System
Subject: offlineimap: exception on mail with broken headers (+fix)
Date: Wed, 24 Aug 2005 13:41:08 +0400
Package: offlineimap
Version: 4.0.10
Severity: normal
Tags: patch
Recently I've got an exception (see below) while using offlineimap.
Exception was probably caused by invalid Date header of (likely spam)
message:
Date: Sat, 20 Aug 2005 4294967295:43:18 -0700
From: "Nikita V. Youshchenko"
I trued to use offlineimap and found that while being quite fast on
small folders, it takes up to several minutes (of 100% busy CPU and
almost no network traffic) to sync a folder with 2000+ messages.
While looking into the code, I found why this happens.
In folder/Base.py, in method BaseFolder.syncmessagesto_copy(),
dest.getmessagelist() is called inside a loop, while being a loop
invariant. Similar thing happens in BaseFolder.syncmessagesto_delete()
for self.getmessagelist().
This causes quadratic complexity over folder size.
Moving these calls out of loops make large folder sync fast (several
seconds instead of several minutes for folder with 2000 messages on
700MHz P3).
Applied patch from Joerg Wendland <joergland@debian.org> to use
APPENDUID result from mail servers that provide it. Closes: #198772.
Resolves: [debian.org #198772]
Added a "force" option to imapserver/select to force a reloading of a
folder. Per [complete.org #67], when cachemessagelist() was called on
an object that was cached from a previous run, it would not re-issue
the select(). Closes: [complete.org #67]
Added -l option. Updated documentation for it. Changed _msg to
_display override in UI modules. Renamed "doc" to "docs" target in
Makefile to avoid conflicting with a subdir.
- offlineimap (3.99.17) unstable; urgency=low
- Fixed two potential obscure race conditions in folder/Maildir.py. +
Condition 1 involved the gettimeseq() function. This function
accesses per-module variables but does not have a lock. It may have
been possible for this to have been called in such a way that
timeseq was not properly updated. + Condition 2 involved the call to
gettimeseq(). Since the timeseq is based on the system clock, we now
use the time as reported inside timeseq() rather than outside. This
way, we can be assured that the same value is in use both places.
- Added debug code to savemessage in folder/Maildir.py to try to track
down a mysterious 0-length file bug. -- John Goerzen
<jgoerzen@complete.org> Tue, 6 May 2003 09:21:38 -0500
- Added some significant debug code to folder/IMAP.py when saving a
new message with APPEND. This should make it easier to track down
bugs both in OfflineIMAP and in mail servers that implement this
poorly.
- Fixed adding of X-OfflineIMAP header when the message starts out
with no headers. (This should not generally occur.) This should help
with some "invalid literal for long()" problems.
When sep was /, the new Maildir support code would recursively try to
scan ., resulting in huge paths and an eventual crash. Fixed with a
one-line patch to Maildir.py. Closes: [complete.org #60] Sergei, The
below diff is going into 3.99.16. You can apply it to 3.99.15 and it
should work for you now. Please let me know. (Ignore any patch errors
for debian/changelog). Thanks for the report.
Raise an exception when the status area is locked. This will cause UIs
to go through their normal exception handling code. In particular, for
the Curses.Blinkenlights interface, the Curses module will be stopped
and the error message will be printed on the console. Previously, this
error message would not have been visible. Closes: #185709.
- Slight renaming in offlineimap.conf.minimal to clarify things.
- Documentation updated with information about new features. Closes:
#189771. + Described IMAP-IMAP syncing + Updated minimal example
with new offlineimap.conf.minimal + Updated UID information. Added
link to recent mailing list discussion. + Described KMail syncing,
which now works. + Added link to mailing list archives.
- Now checks that SELECT succeeded when entering a folder.
- Verifies that folders listed on folderincludes actually exist by
trying to enter them. Thus, if they do not exist, they can be
created on the first run.
Fixed line-ending code to deal with files with mixed \n and \r\n
codes. This is a rare case, but now is more onerous because we now
have to find headers.
Due to possibly having one account sleep while another is reading a
password, and other tricky situations, support for nice updating and
cancelling of a sleep in TTY.TTYUI has been removed. However, this is
not going to be a huge problem because the new Curses Blinkenlights
interface has this support, and does it a lot better than TTY.TTYUI
ever could have.
Reworked the canvas. Before, problem was the label and buttons to the
right of the lights would make the window too wide. When the button
got added, the window would get even wider. That was because the
canvas would not shrink. My workaround is to use a separate canvas for
each light. Seems to be OK here....
Fixed up the VerboseUI for new account system. All that really needed
updating with the "Sync immediately" button, to cope with syncing
different accounts at different times. It's better now.
More locking updates. Introduced a new MultiLock to threadutil. This
lock will let a single thread acquire the same lock more than once,
keeping track of how many times this happens, and will release the
actual lock only when the lock's lock count gets back to zero. By
using MultiLock, various functions in Curses.py and Blinkenlights.py
no longer need to pass around to other functions a parameter
indicating whether or not a lock should be obtained. This was a large
cause of complexity and errors, which is now eliminated. Everything
seems to be working properly wrt locking at this point. The
Curses.Blinkenlights interface has achieved basic working
functionality.
Updated the mbnames recorder to bring it back up-to-date with the new
account-centric system. It will now gather reports from account sync
threads, and when it has all that it's supposed to, it'll write out
the file.
Added some temporary debug code to help weed out a few race conditions
with the curses Blinkenlights interface. Think I've finally got it.
I'm leaving the debugging code in for now, though, to help in case
there are future problems.
More progress at debugging. The curses blinkenlights is now working
well, though it still has an occasional tendency to corrupt the light
display with comments from the log. I suspect a locking problem --
need to be more strict with iolock I suspect. Updated various modules
to register the threads' account names, etc.
Changed to a more account-centric behavior. The refresh time is now a
per-account variable. Implemented new account classes. User interfaces
must now be updated to take advantage of this.
Now properly handles folder names that contain parenthesis. Used patch
from Kyler Laird in http://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=173895. Closes: #173895.
Alter handling of messages flagged for deletion -- no longer
automatically delete them if expunge is 0. In Maildir folder, we will
now ignore the T flag entirely, and just pass it back and forth with
IMAP.
Moved password promting into imapserver.py. Passwords are now asked
for on-demand and typos will no longer crash the program (the user
will be re-prompted). Closes: #162672.
When an exception occurs, OfflineIMAP will attempt to print the last
50 debug messages, whether or not debugging was enabled for this
session. This way, even unexpected and non-repeatable errors stand a
chance of getting a more detailed log.
- Moved some code from offlineimap/init.py to new file
offlineimap/syncmaster.py to help dileneate between code that
performs different functions.
- Moved threadexited from offlineimap/init.py to
offlineimap/threadutil.py.
-d now takes a parameter to specify what kind of debugging to do.
imaplib now does debugging through the UI system. UIBase now has a
debugging process.
- Oops, incomplete commit from the last one:
- If a given Maildir folder is new, remove the associated local status
cache file, if any. That way, there will not be any chance of
propogating hordes of deletes and adds based on old status data.
If a given Maildir folder is new, remove the associated local status
cache file, if any. That way, there will not be any chance of
propogating hordes of deletes and adds based on old status data.
Better handling of read-only folders. We will now warn if there is a
change, but not propogate it. New config variable ignore-readonly can
suppress the warnings. This fixes [complete.org #10] and, for Debian,
Closes: #154769. changelog: noted the change IMAP.py: trap
imapobj.readonly more often UIBase.py: new methods to handle the
warnings offlineimap.conf: new ignore-readonly variable.
Backed out check for . in account names for now. Will put it back in
when we have a consensus on what exactly to do. Doubt that anyone has
a foldername that would conflict with Blinkenlights anyway.
This is release 3.1.1. Fixed a typo in the manual: Tk.TKUI -> Tk.TkUI.
Noted the new version and release time in the changelog and in
version.py. Rebuilt the documentation. Feed the cats, watered the
plants, backed up /dev/null.
- Modified imaputil.py and folder/Maildir.py to run faster. Eliminated
many regular expressions; pre-compiled many others.
- Fixed threadutil's exitnotifyloop to always handle threads in the
order they exited, rather than sometimes in the inverse order. This
way, make sure to handle thread's exception messages before a thread
exited.
