Commit Graph

2254 Commits

Author SHA1 Message Date
Nicolas Sebrecht
ac27c93c83 Merge branch 'ns/doc-install-from-git' into next 2011-05-07 13:51:59 +02:00
Sebastian Spaeth
f4081985dc Prettify and use new uidexists() helper function
Make the folder classes use uidexists() more. Add some code
documentation while going through.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:50:22 +02:00
Sebastian Spaeth
0318c6ad34 Create LocalStatus or LocalStatusSQLite folders
Depending on the configuration we use the plain text or the new
experimental sqlite backend for the LocalStatus cache. Make plain text
the default status backend but allow people to configure
status_backend=sqlite in their [Account ...] section.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:35:03 +02:00
Sebastian Spaeth
0af9ef70a7 Factor out SQL retries
Test if sqlite is multithreading-safe and bail out if not. sqlite
versions since at least 2008 are.
But, as it still causes errors when 2
threads try to write to the same connection simultanously (We get a
"cannot start transaction within a transaction" error), we protect
writes with a per class, ie per-connection lock. Factor out the retrying
to write when the database is locked.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
af25c2779f repository.LocalStatus: Remove code duplication
Make getfolders() invoke getfolder() for each folder rather than
duplicating code. Also add a forgetfolders() implementation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
e1e9c8e831 Use self.doautosave rather than self.dofsync
doautosave was a useless variable before (it was *always* 1). So we
remove the self.dofsync variable and store in doautosave whether we
should fsync as often as possible (which really hurts performance).

The sqlite backend could (at one point) use the doautosave variable to
determine if it should autocommit after each modification.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
bd3766eca5 Make getnicename work again for classes that derive from object()
Python's new style classes derive from object and str(class().__class__)
will return a slightly different format. class().__class.__name__ will
still work for both old and new style classes, so use that instead.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
2fc12e875a Experimental LocalStatus stored in SQLite database
Based on patches by Stewart Smith, updated by Rob Browning.

plus:
- Inherit LocalStatusSQLFolder from LocalStatusFolder
  This lets us remove all functions that are available via our ancestors
  classes and are not needed.

- Don't fail if pysql import fails. Fail rather at runtime when needed.

- When creating the db file, create a metadata table which contains the
  format version info, so we can upgrade nicely to other formats.

- Create an upgrade_db() function which allows us to upgrade from any
  previous file format to the current one (including plain text)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
535f4592fc Replace 2 regexes with a single one
No functional changes

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Nicolas Sebrecht
c6259fbb86 Merge branch 'master' into next
Conflicts:
	Changelog.draft.rst
2011-05-05 21:16:02 +02:00
Sebastian Spaeth
a40e506908 Fix typo to force singlethreading in debug mode
A typo prevented us from enforcing singlethreading mode when selecting debugging.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 21:15:01 +02:00
Sebastian Spaeth
f2ad4bc230 Fix typo to force singlethreading in debug mode
A typo prevented us from enforcing singlethreading mode when selecting debugging.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 21:12:10 +02:00
Sebastian Spaeth
deab62fbd8 Fix the broken thread debugging
Using threading._VERBOSE=1 is broken since python 2.6 till at least
python 3.2, (http://bugs.python.org/issue4188) so we can't use it for
our thread debugging.

Remove the usage of threading._VERBOSE, and implement a "light thread
debug log" that for now outputs information when a new thread is being
registered and when it is being unregistered. I am sure we will be able
to add more thread debugging information over the time.

Besides '-d thread' this will re-enable the usage of -d 'all' for the
most verbose debugging of all categories.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 21:07:24 +02:00
Sebastian Spaeth
5c7d7ee445 Output more detailed error on corrupt LocalStatus
When our LocalStatus cache is corrupt, ie e.g. it contains lines not in
the form number:number, we would previously just raise a ValueError
stating things like "too many values". In case we encounter clearly
corrupt LocalStatus cache entries, clearly raise an exception stating
the filename and the line, so that people can attempt to repair it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 20:09:57 +02:00
Sebastian Spaeth
f3774343db If we loop, exit the account synchronization after 3 failed attempts
This should get rid of intermittent network failures, but lets us bail
out on permanent errors.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 20:00:07 +02:00
Sebastian Spaeth
4608d14836 Handle OfflineImapError of severity REPO and CRIT
By aborting the account syncing, the looping and logging an error
message. We will introduce a ui.error() rather than a ui.warn() function
which saves all Exceptions in a Queue and outputs them at the end of the
program.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 20:00:07 +02:00
Sebastian Spaeth
0f45d89e34 Throw OfflineImapError on DNS error
In case we misconfigured a server name or are otherwise offline, a
socket.gaierror will be raised when attempting to connect. We catch that
case and raise an OfflineImapError with severity ERROR.REPO, meaning we
should stop syncing this account and continue with the next one.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 20:00:07 +02:00
Sebastian Spaeth
bc004a911a Add OfflineImapError class
This Exception can be thrown whenever a sync error occurs. It is used
for outputting sensible error messages to the user and it denotes a
"severity", it can tell offlineimap if we need to abort a message, a
folder, a repo sync, or whether we should indeed abort immediately.

The exception is not yet used in the code.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 20:00:07 +02:00
Sebastian Spaeth
3446b592f3 accounts: remove duplicated code paths
Currently, account.syncrunner() has 2 separate duplicated code paths
depending on whether we want to autorefresh after some waiting perios
or not. Unify those code paths by setting "looping = False" in case
self.refeshperiod == 0 after the first run. Behavior is identical to
before.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 19:52:24 +02:00
Sebastian Spaeth
b1ff307674 Output more detailed error on corrupt LocalStatus
When our LocalStatus cache is corrupt, ie e.g. it contains lines not in
the form number:number, we would previously just raise a ValueError
stating things like "too many values". In case we encounter clearly
corrupt LocalStatus cache entries, clearly raise an exception stating
the filename and the line, so that people can attempt to repair it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 19:00:46 +02:00
Nicolas Sebrecht
01b390bb71 INSTALL: explain how to install offlineimap using git
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-04 19:03:35 +02:00
Sebastian Spaeth
f3249e0856 True 1-way sync (backup)
This commit enables true 1-way syncing between repositories. This has
often been demanded for backup purposes when you do not want to cause
accidental modifications of your backup that would be propagated to the
other side.

This has been implemented by allowing to configure a Repository as
'readonly' to forbid any modification on it.

'readonly' applies to all the type of repositories.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 20:41:36 +02:00
Sebastian Spaeth
3d4dc11a8b Introduce an *empty* debug type
This debug type will always be enabled whenever any debugging is enables
and it outputs debug messages that cannot be categorized among any of
imap, maildir (e.g. things that concern the sync logic).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 20:41:36 +02:00
Sebastian Spaeth
50f0fbb421 More detailed error output on corrupt UID mapping files
This function will need much more "robustifying", but the very least we
can do is to print the file name and line that are giving trouble.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 20:18:51 +02:00
Sebastian Spaeth
6add201436 Improve the developer API documentation
Improve the code documentation (still much more to do) and also add some
more meat to the structure of the developer documentation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 20:16:45 +02:00
Nicolas Sebrecht
e9a7afda6d Merge branch 'ss/corrupted-uidmap-file' into next
Conflicts:
	Changelog.draft.rst
2011-05-02 19:09:52 +02:00
Sebastian Spaeth
2ed1c357a0 More detailed error output on corrupt UID mapping files
This function will need much more "robustifying", but the very least we
can do is to print the file name and line that are giving trouble.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 19:08:59 +02:00
Nicolas Sebrecht
cda932351c Merge branch 'ss/folder-name-whitspaces' into next
Conflicts:
	Changelog.draft.rst
2011-04-28 18:29:14 +02:00
Dan Christensen
9bd7a21f16 Don't strip whitespace in the -f option
Allow leading and trailing spaces in folder names specified on the
command line.

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Dan Christensen <jdc@uwo.ca>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-28 18:28:23 +02:00
Sebastian Spaeth
163f1eefc4 Drop all connections when a sync failed
in accounts.sync() we would holdordropconnections() after each sync. But
depending on the repository configuration that might imply that
offlineimap tries to keep the same connections. But when a sync failed,
e.g. after a user had his computer suspended, it might be that our
connections that we have are worthless. So definitely drop them after a
failed sync.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 23:04:24 +02:00
Nicolas Sebrecht
0bc1c9583e Merge branch 'ss/folder-cleanup-getmessage' into next
Conflicts:
	offlineimap/folder/IMAP.py
2011-04-27 22:52:17 +02:00
Sebastian Spaeth
1ff628bd5c folder/IMAP: cleanup getmessage()
Add some comments how the data structures actually look like.
Describe the function properly, and make sure we only hold on to the
data connection as quickly as possible.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 22:49:18 +02:00
Sebastian Spaeth
c8f80bc6d2 We had been setting this variable twice
This was in the code for a very long time, it seems.
Remove one instance, no functional changes.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 22:35:39 +02:00
Nicolas Sebrecht
e614838a1f Merge branch 'ss/improve_logging_using_str' into next 2011-04-27 19:07:49 +02:00
Sebastian Spaeth
e01968973d Use "%s" % account rather than account.name
uncritical patch, but we can make the code a bit shorter so why not.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:10 +02:00
Sebastian Spaeth
3e5d6f5450 Make str() of Repository() be its name
So we can simply hand a repository instance to error messages rather than
having to call Repository().getname() all the time.

This is not used yet.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:10 +02:00
Sebastian Spaeth
074ea14cf1 Make str() of Account() be its name
So we can simply hand an account instance to error messages rather than
having to call Account().getname() all the time.

This is not used yet.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:10 +02:00
Sebastian Spaeth
bb8546e846 Whitespace fixes
Just clean up lines which end in whitespaces.
Also adapt the copyright to the current year while touching the file.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 19:07:09 +02:00
Sebastian Spaeth
fa0b1ef8ee Add beginnings of sphinx-based code documentation
The docs still need some meat, but the infrastructure is in place. THis
allows us to generate the nice looking API documentation that many
python projects already have. We should document our API better, providing
an overview of the functionality available.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 18:57:41 +02:00
Nicolas Sebrecht
b2be0c3697 Merge branch 'ss/sync-remove-neguid-pass' into next 2011-04-25 13:01:59 +02:00
Sebastian Spaeth
e20d8b9679 Remove upload neguid pass from sync logic
In order to optimize performance, we fold the 1st and 2nd pass of our
sync strategy into one. They were essentially doing the same thing:
uploading a message to the other side. The only difference was that in
one case we have a negative UID locally, and in the other case, we have
a positive one already.

This saves some time, as we don't have to run through that function on
IMAP servers anyway (they always have positive UIDs), and 2nd were we
stalling further copying until phase 1 was finished. So uploading a
single new message would prevent us from starting to copy existing
regular messages.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-25 13:01:53 +02:00
Sebastian Spaeth
36eb37b47d IMAP: reduce quickchanged() checks
For each folder we were making a second IMAP request asking for the
latest UID and compared that with the highest UID in our
statusfolder. This catched the case that 1 mail has been deleted by
someone else and another one has arrived since we checked, so that the
total number of mails appears to not have changed.

We don't capture anymore this case in the quickchanged() case.

It improves my performance from 8 to about 7.5 seconds per check (with lots of
variation) and we would benefit even more in the IMAP<->IMAP case as we do one
additional IMAP lookup per folder on each side then.

Do cleanups on whitespaces while in this file.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-25 12:58:47 +02:00
Nicolas Sebrecht
12e11429b5 Changelog: add missing date for v6.3.3
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-24 18:43:35 +02:00
Nicolas Sebrecht
fb3abaa6a5 v6.3.2.2
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-24 18:37:23 +02:00
Nicolas Sebrecht
3b09695236 v6.3.3
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-21 18:34:24 +02:00
Thomas Kahle
9136bdb0ba Adding an entry to offlineimap.conf that explain how to use python code to query for a password.
Signed-off-by: Thomas Kahle <tomka@gentoo.org>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-20 18:46:36 +02:00
Nicolas Sebrecht
84db2c50ac v6.3.3-rc3
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-19 19:43:50 +02:00
Nicolas Sebrecht
bf1c9e3233 Merge branch 'ss/maildir-quickchanged-cleanup' into next 2011-04-15 19:52:35 +02:00
dtk
38dfa1d684 Improve documentation on quick syncs
Make clear that quick syncs do not happen inbetween full syncs (ie they
are part of the regular autorefresh intervals and don't happen within
them). This part of the documentation had confused many.

Signed-off-by: dtk <d.t.k@gmx.de>
Modified-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-14 19:14:19 +02:00
Ethan Glasser-Camp
9e734006f6 Fix IMAP4_Tunnel to work with imaplib2
* IMAP4_Tunnel constructor should support base-class arguments, in
  order to support the timeout argument.

* IMAP4_Tunnel needs to store the member IMAP4.host, which is normally
  done in IMAP4.open().

* Update IMAP4_Tunnel.read() and IMAP4_Tunnel.send().  We turn on
  nonblocking mode for these sockets, so we can return immediately
  with whatever data is available.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-13 18:41:53 +02:00