* Do not inherit LocalStatusSQLiteFolder class from the plaintext
one.
* Use some functions already in BaseFolder in both, plaintext and
sqlite classes.
* Add a saveall method. The idea is that saveall dumps the entire
messagelist to disk, while save only commits the uncommited
changes. Right now, save is noop for sqlite, and equivalent to
saveall for plaintext, but it enables to be more clever on when we
commit to disk in the future.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
* Implements Status Folder format v2, with a mechanism to upgrade an
old statusfolder.
* Do not warn about Gmail and GmailMaildir needing sqlite backend
anymore.
* Clean repository.LocalStatus reusing some code from
folder.LocalStatus.
* Change field separator in the plaintext file from ':' to '|'. Now
the local status stores gmail labels. If they contain field
separator character (formerly ':'), they get messed up. The new
character '|' is less likely to appear in a label.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Make external API of class/module to be smaller, explicitely mark
all internal functions. Also annotate methods that are implemented
as the part of the parent class interface.
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
While looking at the code to investigate if an why we sometimes don't
seem to honor the write lock, I made it use the more modern "with lock:"
pattern.
Still have not found out how we could ever be using 2 instances of the
LocalStatusFolder for the same folder.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Prevent savemessage(), and savemessageflags() to occur in dryrun mode in
all backends. Still need to protect against deletemessage().
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
'set' is builtin since python2.6, so remove the imports. Also 'ssl' exists
since 2.6 and has everything we need, so no need for conditional import
tests here anymore.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
As reported in https://github.com/spaetz/offlineimap/pull/2, we would
fail when files are empty because file.read() would throw attribute
errors.
Fix this by removing the superfluous read() check and additionally log
some warning message.
Reported-by: Ralf Schmitt
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
They have the Repository() which contains the root, so no need to pass
it in as an extra parameter. Rename repository.LocalStatus()'s
self.directory to self.root for consistency with other backends.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It is possible to get the config parameter from the Repository() which is
set in BaseFolder, so we set self.config there and remove the various
methods and 'config' parameters that are superfluous.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We passed in the accountname to all derivatives of BaseFolder, such as
IMAPFolder(...,repository,...,accountname), although it is perfectly
possible to get the accountname from the Repository(). So remove this
unneeded parameter. Each backend had to define getaccountname() (although
the function is hardly used and most accessed .accountname directly).
On the other hand BaseFolder was using getaccountname but it never defined
the function. So make the sane thing, remove all definitions from backends
and define accountname() once in Basefolder. It was made a property and not
just a (public) attribute, so it will show up in our developer
documentation as public API.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
As all Folders share these parameters, we can safely handle them in
BaseFolder. This makes sense, as BaseFolder has a getname() function
that returns self.name but nothing actually set self.name.
It also saves a few lines of code.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
As this is essentially what it is, a set of values. This allows as
to do set arithmetics to see, e.g. the intersection of 2 flag sets
rather than clunkily having to do:
for flag in newflags:
if flag not in oldflags:
oldflags.append(flag)
Also some more code documenting.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Empty foldernames (as they could be created through nametrans) were
failing as the uidvalidity and status files names as determined by
folder/Base.py:getfolderbasename() lead to invalid file names ''.
Fix this by handling empty file names and translating them to '.' which
leads to the special file name 'dot'. (this special value existed before
and was not invented by this patch)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
doautosave was a useless variable before (it was *always* 1). So we
remove the self.dofsync variable and store in doautosave whether we
should fsync as often as possible (which really hurts performance).
The sqlite backend could (at one point) use the doautosave variable to
determine if it should autocommit after each modification.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Based on patches by Stewart Smith, updated by Rob Browning.
plus:
- Inherit LocalStatusSQLFolder from LocalStatusFolder
This lets us remove all functions that are available via our ancestors
classes and are not needed.
- Don't fail if pysql import fails. Fail rather at runtime when needed.
- When creating the db file, create a metadata table which contains the
format version info, so we can upgrade nicely to other formats.
- Create an upgrade_db() function which allows us to upgrade from any
previous file format to the current one (including plain text)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When our LocalStatus cache is corrupt, ie e.g. it contains lines not in
the form number:number, we would previously just raise a ValueError
stating things like "too many values". In case we encounter clearly
corrupt LocalStatus cache entries, clearly raise an exception stating
the filename and the line, so that people can attempt to repair it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This was in the code for a very long time, it seems.
Remove one instance, no functional changes.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Earlier we would ignore *ALL* Exceptions that could occur during the
fsyncing of our LocalStatus database. Ignoring all Exceptions is not the
right thing here though. A recent commit improved the situation by
raising at least KeyboardInterrupt Exceptions, but that is still not
optimal.
os.fsync() is available on Unix, and Windows starting in python
2.2.3. so it should always work. If it doesn't, something is wrong.
It has been suggested to only catch EnvironmentError (ie SystemError and
OSError) here, but even those should be thrown. Something *is* wrong if
this fails and we should not ignore it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously we did not catch KeyboardInterrupts explicitly as all of the
code was executed in forked child threads which would never receive
Ctrl-c exceptions. With the upcoming single threaded modus, this code
can be run in the main thread however, so we need to take care of
KeyboardInterrupts explicitly.
This was done wherever we would catch *ALL* exceptions universally and
print out an error message.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
After tens of thousands of messages on the IMAP server were deleted it
takes offlineimap extremely long time (several hours of high CPU usage)
to delete them locally. It spends almost all the time modifying
LocalStatus. It processes the messages one by one, rewriting the
folder's status file in LocalStatus after each message.
It is much more efficient to save the status file only once, after
removing all the messages from the messagelist.
Deleting lots of messages now takes seconds instead of hours.
This should solve Debian bug #518093:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=518093
Signed-off-by: Michal Schmidt <mschmidt@redhat.com>
I created new IMAP folders on my server. In some moment I found offlineimap
crashed. The reason were empty LocalStatus files. From looking at the code I
have no explanation how those files came into existence.
fsync the Maildir file, its final directory when writing a new message.
fsync the localstatus file and its final directory when writing the
local status cache.
This should reduce duplication in the event of hardware trouble.
fixes#8
see thread at http://lists.complete.org/offlineimap@complete.org/2007/03/threads.html.gz
The attached patch adds syncing the INTERNALDATE of IMAP folders with
the mtime of messages in maildir folders.
I want this to happen, because I'm running a dovecot over the maildirs
synced by offlineimap, and that uses the mtime as the INTERNALDATE.
When using mutt to view messages I generally sort based on the received
date, which for IMAP folders is the INTERNALDATE.
Since this is the first real coding I've done in Python the patch may
need to be cleaned up some, but it's working pretty well for me. I've
added new messages to each side, and the received date has been
preserved going both ways.
