Commit Graph

94 Commits

Author SHA1 Message Date
Sebastian Spaeth
d981d66305 Add changelog entry for reworked UI backends
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-10-28 09:35:23 +02:00
Sebastian Spaeth
76b0d7cf25 imapserver: Make noop() more resitent against dropped connections.
Drop a connection, if the NOOP to keep a connection open fails due to
broken connections.

Note that I believe this function is not working as intended. We grab
one random connection and send a NOOP. This is not enough to keep all
connections open, and if we invoke this function multiple times, we
might well always get the same connection to send a NOOP through.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-10-05 18:18:55 +02:00
Sebastian Spaeth
3885acf87d Implement server diagnostics
This outputs a handy summary of your server configuration and version
strings etc, which is useful for bug reporting.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-30 09:41:45 +02:00
Sebastian Spaeth
4e89bbfe6e Recache folder list after creating a new folder
If we create a new folder we would previously not update our folder
list, which led to us skipping the synchronization of those new folders
during the initial run (subsequent runs would pick it up).

Invalidate the folder cache when we create a folder during folder
structure sync. Regetting the whole list from an IMAP server might be
slightly suboptimal from a performance point, but it is easy and will
lead to consistent results. Hopefully we will not have to create new
folders on each new run.

Reported-by: Daniel Shahaf <d.s@daniel.shahaf.name>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-30 09:05:49 +02:00
Sebastian Spaeth
3b647d65be Output sync timing
Modify the UI:acct and acctdone functions to keep tab of the time
inbetween. Put self.ui.acct() and acctdone() at the right places in
accounts.py so that the timing happens at the right places.

While modifying that loop, flatten the nested try: try: except: finally:
constructs, we require python 2.5 now which copes with that.

At the end of each account sync you will now see something like:
*** Finished account 'test' in 0:05

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 17:26:46 +02:00
Sebastian Spaeth
ba396cf0ef More verbose loggin
Thread names are used to determine the logging header in the TTY ui. A
recent change made them too terse (basically only changing the account
name and not the folder names). Unbreak.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 16:59:36 +02:00
Sebastian Spaeth
8970a1500b UIBase: Fix and cleanup register/unregisterthread
Registering a thread (associating it with a certain account name) would
fail if it was already registered. However, as we a) never unregister most
threads (bad) and b) single-threaded mode reuses threads, we failed when
syncing multiple accounts in single-threading mode.

This commit cleans up the functions to not make re-registering a thread
fatal (it could be legitimate, however it *should* not occur). Future
work needs to be done to unregister new threads at the appropriate places.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 15:44:30 +02:00
Sebastian Spaeth
275dbfa3fb Release 6.3.5-rc3
Update Changelog and __VERSION__ for 6.3.5-rc3.
Also add in the notes for rc2 to the Changelog.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-21 02:32:50 +02:00
Sebastian Spaeth
347e1eaa32 update CAPABILITIES after login
Some Webservers (I am looking at you Gmail) send different capabilities
before and after login, so they can tailor their server capabilities to
the user. While legal, this is uncommon and we were not updating our
server capabilities. Doing so allows us to detect that Gmail actually
supports the UIDPLUS extension, and we will stop mangling headers when
uploading to Gmail. This could lead to some performance gains when we
upload many messages to Gmail.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-21 02:32:45 +02:00
Sebastian Spaeth
050bbb3d2d Reset draft Changelog post-release
Empty it, so new stuff can pour in.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 15:21:27 +02:00
Sebastian Spaeth
fd6261a50f Create new folders on srcrepo if needed
This will ignore any nametrans rules, so we might want to limit this
only to cases where no nametrans has been specified, or we might want to
use the nametrans setting of the dest repo.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 14:23:08 +02:00
Sebastian Spaeth
7d34060217 Enable folderfiltering for MailDir repositories too
Currently we only filtered IMAP repositories, this patch enables filtering
for Maildir repositories too.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 14:23:08 +02:00
Sebastian Spaeth
1ac9dc7fb4 Implement RFC 2595 LOGINDISABLED
Warn the user and abort when we attempt a plaintext login, but the
server has explicitly disabled plaintext logins.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:55:27 +02:00
Sebastian Spaeth
c7938dc081 Per-account locking
Previously, we were simply locking offlineimap whenever it was
running. Howver there is no reason why we shouldn't be able to invoke it
in parallel, e.g. to synchronize several accounts in one offlineimap
each.

This patch implements the locking per-account, so that it is possible to
sync different accounts at the same time. If in refresh mode, we will
attempt to loop three times before giving up.

This also fixes Debian bug #586655

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:34:25 +02:00
Sebastian Spaeth
f5366343b9 Fix default Maildir File permissions.
open() and os.open() lead to different file permissions by default, and
while we have not changed the os.open that had been used, some code
changes led to these permissions slipping through. Fix this by setting
the permissions explicitly to 0666 (minus the users umask).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:25:48 +02:00
Sebastian Spaeth
ee1706fa90 documentation: Clarify restoreatime setting
Better document what this actually does and that most people won't be
needing it.

(Especially as mount setting such as relatime|noatime now reduce the
amount of atime changes anyway)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-15 19:28:48 +02:00
Nicolas Sebrecht
1a4b7c337c v6.3.5-rc1
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-12 20:10:44 +02:00
Sebastian Spaeth
8800fa37a3 Implement Server SSL fingerprint check
If we connect to a SSL server (not STARTTLS) and no CA cert has been
specified for verification, we check the configured SSL fingerprint and
bail out in case it has not been set yet, or it does not match.

This means one more mandatory option for SSL configuration, but it
improves security a lot.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-12 19:45:15 +02:00
Nicolas Sebrecht
2749374a93 Merge branch 'ss/imaplib2-2.28' into next
Conflicts:
	Changelog.draft.rst
2011-08-15 13:57:37 +02:00
Sebastian Spaeth
3a91e296f0 Adapt the code to work with the new imaplib2
imaplib renamed self.sslobj to self.sock and our overriden open()
functions were failing for that reason when updating imaplib2 to
v2.28. It turns out that all of our custom initializations are being
done by stock imaplib2 now anyway, so there is no need to override them
anymore. This lets us simplify the code we have to worry about.

Move the verifycert() function to the imapserver.py file, it is now a
callback function that is being handed to imaplib from there, so it
makes sense to also define it in our imapserver function...
(this also lets us easily make use of the verifycert function in the
starttls case in the future)

TODO: we need to examine if and why we still need to override the
select() function, it is the only reason why we still wrap the IMAP4
classes.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:21:38 +02:00
Sebastian Spaeth
634b6cd49e Don't make Blinkenlight statuschar configurable
Allowing to specify the char to use in the BLinkenlights is a bit over
the top and bloats our default offlineimap.conf. The dot is just fine,
so let us settle for it and cut the example config file by an unneeded
section.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:10:20 +02:00
Nicolas Sebrecht
a38c6da49d Merge branch 'ss/improved-folderfilter-errors' into next
Conflicts:
	Changelog.draft.rst
2011-08-12 18:29:23 +02:00
Sebastian Spaeth
fa18968642 Catch folderfilter errors in repository.IMAP.getfolders()
Rather than throwing ValueError, we now properly throw OfflineImapError
when selecting a folder in folderincludes. So we also need to catch
OfflineImapErrors here. If they are of severity FOLDER, just ignore the
invalid folder and continue. If the error is more severe, bubble it up.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-12 18:28:49 +02:00
Sebastian Spaeth
131298c2b1 Remove superfluous class ConfigedIMAPServer
Remove a level of wrapper abstraction that is not needed. Just use
IMAPserver and be done with it.

We do this by passing in the IMAPRepository() instance rather than a
long list of single paramters to the IMAPServer instanciation. This way
we can retrieve all repository parameters ourselves, rather than passing
a dozen paramters into IMAPServer. Also, this enables us to pass the
repository() object into our WrappedIMAP4() instance, so that it can
query, e.g. the SSL fingerprint configuration.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-12 18:18:26 +02:00
Sebastian Spaeth
b47bc44783 accounts.py: Use ui.error when raising exceptions
Rather than using ui.warn, use ui.error() which outputs Exceptions to
the error log, saving them to a stack, so we get notified again at the
end of the sync run.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-11 19:14:28 +02:00
Nicolas Sebrecht
61e50b3b1a v6.3.4
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-10 21:44:58 +02:00
Vladimir Marek
9d5f805981 FAQ: add commant about UID not found on saved messages
This add missing doc as requested on the mailing list for commit d5cbdc4c0e.
Also, add an entry in the changelog.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-01 19:19:15 +02:00
Nicolas Sebrecht
beef888ed7 Changelog: fix missing note about iddle support
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-07 22:32:42 +02:00
Nicolas Sebrecht
129b703bf2 v6.3.4-rc3
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-07 18:43:18 +02:00
Sebastian Spaeth
6311716edb imapserver.py: Implement STARTTLS
If we do not use a SSL connection anyway and if the server supports it,
authenticate automatically with STARTTLS.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-25 17:09:52 +02:00
Sebastian Spaeth
856982a4e6 Throw OfflineImapError when we try to request an inexistant message
During a sync run, someone might remove or move IMAP messages. As we
only cache the list of UIDs in the beginning, we might be requesting
UIDs that don't exist anymore. Protect folder.IMAP.getmessage() against
the response that we get when we ask for unknown UIDs.

Also, if the server responds with anything else than "OK", (eg. Gmail
seems to be saying frequently ['NO', 'Dave I can't let you do that now']
:-) so we should also be throwing OfflineImapErrors here rather than
AssertionErrors.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-15 22:17:04 +02:00
Nicolas Sebrecht
5328f10b39 v6.3.4-rc2
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-15 18:15:11 +02:00
Nicolas Sebrecht
543c7b2fb7 Merge branch 'ss/maildir-simplify-save' into next
Conflicts:
	Changelog.draft.rst
2011-06-13 16:30:06 +02:00
Sebastian Spaeth
d8026c5308 Simplify Maildir message saving
Previously we were attempting to save out mails according to
 http://www.qmail.org/man/man5/maildir.html in 4 steps:

1 Create a unique filename
2 Do stat(tmp/<filename>). If it found a file, wait 2 sec and go back to 1.
3 Create and write the message to the tmp/<filename>.
4 Link from tmp/* to new/*

(we did step 2 up to 15 times) But as stated by
http://wiki1.dovecot.org/MailboxFormat/Maildir (see section 'Issues with
the specification'), this is a pointless approach, e.g. there are race
issues between stating that the filename does not exist and the actual
moving (when it might exist).

So, we can simplify the steps as suggested in the dovecot wiki and
tighten up our safety at the same time.

One improvement that we do is to open the file, guaranteeing that it did
not exist before in an atomic manner, thus our simplified approach is
really more secure than what we had before.

Also, we throw an OfflineImapError at MESSAGE level when the supposedly
unique filename already exists, so that we can skip this message and
still continue with other messages.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-13 16:29:00 +02:00
Sebastian Spaeth
e18bb4b2b2 Add Changelog recommendation for commit 520e39d355
Recommend to upgrade from the previous release. I forgot to change the
Changelog previously.  [Update imaplib2 to 2.24]

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-12 17:48:51 +02:00
Nicolas Sebrecht
2d31abde76 v6.3.4-rc1
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-16 18:54:22 +02:00
Sebastian Spaeth
0318c6ad34 Create LocalStatus or LocalStatusSQLite folders
Depending on the configuration we use the plain text or the new
experimental sqlite backend for the LocalStatus cache. Make plain text
the default status backend but allow people to configure
status_backend=sqlite in their [Account ...] section.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:35:03 +02:00
Nicolas Sebrecht
c6259fbb86 Merge branch 'master' into next
Conflicts:
	Changelog.draft.rst
2011-05-05 21:16:02 +02:00
Sebastian Spaeth
deab62fbd8 Fix the broken thread debugging
Using threading._VERBOSE=1 is broken since python 2.6 till at least
python 3.2, (http://bugs.python.org/issue4188) so we can't use it for
our thread debugging.

Remove the usage of threading._VERBOSE, and implement a "light thread
debug log" that for now outputs information when a new thread is being
registered and when it is being unregistered. I am sure we will be able
to add more thread debugging information over the time.

Besides '-d thread' this will re-enable the usage of -d 'all' for the
most verbose debugging of all categories.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-05 21:07:24 +02:00
Sebastian Spaeth
f3249e0856 True 1-way sync (backup)
This commit enables true 1-way syncing between repositories. This has
often been demanded for backup purposes when you do not want to cause
accidental modifications of your backup that would be propagated to the
other side.

This has been implemented by allowing to configure a Repository as
'readonly' to forbid any modification on it.

'readonly' applies to all the type of repositories.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 20:41:36 +02:00
Nicolas Sebrecht
e9a7afda6d Merge branch 'ss/corrupted-uidmap-file' into next
Conflicts:
	Changelog.draft.rst
2011-05-02 19:09:52 +02:00
Sebastian Spaeth
2ed1c357a0 More detailed error output on corrupt UID mapping files
This function will need much more "robustifying", but the very least we
can do is to print the file name and line that are giving trouble.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-02 19:08:59 +02:00
Nicolas Sebrecht
cda932351c Merge branch 'ss/folder-name-whitspaces' into next
Conflicts:
	Changelog.draft.rst
2011-04-28 18:29:14 +02:00
Dan Christensen
9bd7a21f16 Don't strip whitespace in the -f option
Allow leading and trailing spaces in folder names specified on the
command line.

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Dan Christensen <jdc@uwo.ca>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-28 18:28:23 +02:00
Sebastian Spaeth
163f1eefc4 Drop all connections when a sync failed
in accounts.sync() we would holdordropconnections() after each sync. But
depending on the repository configuration that might imply that
offlineimap tries to keep the same connections. But when a sync failed,
e.g. after a user had his computer suspended, it might be that our
connections that we have are worthless. So definitely drop them after a
failed sync.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-27 23:04:24 +02:00
Sebastian Spaeth
e20d8b9679 Remove upload neguid pass from sync logic
In order to optimize performance, we fold the 1st and 2nd pass of our
sync strategy into one. They were essentially doing the same thing:
uploading a message to the other side. The only difference was that in
one case we have a negative UID locally, and in the other case, we have
a positive one already.

This saves some time, as we don't have to run through that function on
IMAP servers anyway (they always have positive UIDs), and 2nd were we
stalling further copying until phase 1 was finished. So uploading a
single new message would prevent us from starting to copy existing
regular messages.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-25 13:01:53 +02:00
Nicolas Sebrecht
84db2c50ac v6.3.3-rc3
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-19 19:43:50 +02:00
Ethan Glasser-Camp
9e734006f6 Fix IMAP4_Tunnel to work with imaplib2
* IMAP4_Tunnel constructor should support base-class arguments, in
  order to support the timeout argument.

* IMAP4_Tunnel needs to store the member IMAP4.host, which is normally
  done in IMAP4.open().

* Update IMAP4_Tunnel.read() and IMAP4_Tunnel.send().  We turn on
  nonblocking mode for these sockets, so we can return immediately
  with whatever data is available.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-13 18:41:53 +02:00
Sebastian Spaeth
d762175af4 Make -f option with with folder names with spaces.
Previously ALL spaces had been stripped off. Now, only strip spaces
around the comma, so -f "INBOX, Deleted Mails" will work. You will still
need to quote or escape spaces so the shell hand the list as one command
line argument to offlineimap.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-11 18:29:44 +02:00
Nicolas Sebrecht
69d2185c23 v6.3.3-rc2
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-04-07 19:46:10 +02:00