Commit Graph

1494 Commits

Author SHA1 Message Date
Sebastian Spaeth
d8026c5308 Simplify Maildir message saving
Previously we were attempting to save out mails according to
 http://www.qmail.org/man/man5/maildir.html in 4 steps:

1 Create a unique filename
2 Do stat(tmp/<filename>). If it found a file, wait 2 sec and go back to 1.
3 Create and write the message to the tmp/<filename>.
4 Link from tmp/* to new/*

(we did step 2 up to 15 times) But as stated by
http://wiki1.dovecot.org/MailboxFormat/Maildir (see section 'Issues with
the specification'), this is a pointless approach, e.g. there are race
issues between stating that the filename does not exist and the actual
moving (when it might exist).

So, we can simplify the steps as suggested in the dovecot wiki and
tighten up our safety at the same time.

One improvement that we do is to open the file, guaranteeing that it did
not exist before in an atomic manner, thus our simplified approach is
really more secure than what we had before.

Also, we throw an OfflineImapError at MESSAGE level when the supposedly
unique filename already exists, so that we can skip this message and
still continue with other messages.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-13 16:29:00 +02:00
Sebastian Spaeth
e18bb4b2b2 Add Changelog recommendation for commit 520e39d355
Recommend to upgrade from the previous release. I forgot to change the
Changelog previously.  [Update imaplib2 to 2.24]

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-12 17:48:51 +02:00
Sebastian Spaeth
e023f190b0 folder/Maildir: Store only relative filename components
MaildirFolder.messagelist[*]['filename'] was storing the absolute file
paths for all stored emails. While this is convenient, it wastes much
space, as the folder prefix is always the same and it is known to the
MaildirFolder. Just 40 chars in a folder with 100k mails waste >4MB of
space. Adapt the few locations where we need the full path to construct
it dynamically.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-12 17:41:13 +02:00
Sebastian Spaeth
c1c200a487 folder/Maildir: cache getfullname() value
We use getfullname() very often (thousands to millions), yet we
dynamically calculate the very same value over and over. Optimize this
by caching the value once and be done with it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-12 17:40:11 +02:00
Nicolas Sebrecht
4c8abb0bc6 Merge branch 'ss/imaplib2-v2.24' into next 2011-06-09 18:28:23 +02:00
Sebastian Spaeth
520e39d355 Update imaplib2 to 2.24
In some cases we had offlineimap trying to delete emails that shouldn't
be deleted. E.g. https://bugzilla.redhat.com/show_bug.cgi?id=708898.

It turns out that imaplib2 does not like FETCH responses that are
interrupted by other unsolicited server responses, e.g.

* OK Searched 43% of the mailbox, ETA 0:12\r\n

Bump imaplib2 to a version that can cope with these (legal) responses by
the IMAP server.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-09 18:26:39 +02:00
Sebastian Spaeth
e8b633b884 folder/IMAP: Remove buggy duplicate assignment
we do:
  for msgid in imapdata:
      maxmsgid = max(long(msgid), maxmsgid)
and then basically immediately:
   maxmsgid = long(imapdata[0])

throwing away the first assignment although the first method of
assigning is the correct one. The second had been forgotten to be
removed when we introduced the above iteration. This bug would fix a
regression with those broken ZIMBRA servers that send multiple EXISTS
replies.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:52:24 +02:00
Sebastian Spaeth
846070b240 Fix typos in months names
All months names are 3-letter abbreviated, but accidentally June and
July slipped through. Thanks to the heads up by
Philipp Kern <pkern@debian.org>.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:20:25 +02:00
Sebastian Spaeth
b20f5192fb Allow to specify remote hostname even for the Gmail case
Previously we hard-coded the imap server name in the case of Gmail
repositories, but often we need a different host name. So, allow people
to specify the hostname via the regular "remotehosteval" and
"remotehost" settings, and only falling back to imap.gmail.com when
nothing has been specified.

Cache the hostname, so we don't evaluate the whole thing each time we
query the host name.

Make the remotehosteval processing more robust, by catching any
Exceptions that occur, and throw a OfflineImapError, that explains where
exactly the error had occured. You can test this, e.g. by setting
remotehosteval to 1/"n" or some other invalid expression.

The whole IMAP.gethost() function has been documented code wise while
going through.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:19:07 +02:00
Sebastian Spaeth
1754bf4110 Allow to create the root MailDir directory
We currently do not allow nametrans rules such as
nametrans = lambda foldername: re.sub('^INBOX$', '', foldername)

because we crash with a traceback when running:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=499755

The underlying reason is that we cannot create the "top level" root
directory of the Maildir in the function makefolders(), it will bail
out. John Goerzen intentionally prevented offlineimap from creating the
top-level dir, so that a misconfiguration could not arbitrarily create
folders on the file system. I believe that it should be perfectly
possible to automatically create the root dirctory of the maildir. We
still protect against folder creations at arbitrary places in the file
system though.

This patch cleans up makefolders(), adds documentation, allows to
automatically create rootfolders if needed (using absolute paths) and
adds some robustness in case the folders already exist that we want to
create (rather than simply crapping out).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:15:31 +02:00
Sebastian Spaeth
f3ec6d9c7d Assert error hint when trying to create MailDir root
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=499755 shows the
cryptic output we have when, e.g. trying to put somethin in our mailDir
root via the nametrans options. This commit adds at least some hint as
to what went wrong using an "assert" message, although the correct thing
is to allow the creation of a maildir in the root folder.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:10:31 +02:00
Sebastian Spaeth
3dc9fc519a Throw errors on connection refused and on non-standard SSL ports
We were "crashing" with tracebacks when we could not connect to a host,
(e.g. because no service was on the port) and we were getting mysterious
SSL tracebacks when someone tried to connect via SSL to a non-ssl port.

In these cases, we will now throw an nice error message. On python<2.6
where no ssl module exists, we simply won't throw those errors.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-06 21:33:16 +02:00
Nicolas Sebrecht
a5a4cb214b Merge branch 'ns/fix-magicline-import' into next 2011-05-31 18:42:08 +02:00
Nicolas Sebrecht
e87f213046 fix magicline import
commit 0318c6a [Create LocalStatus or LocalStatusSQLite folders] changes import
of LocalStatus but doesn't preserve magicline.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-25 21:54:55 +02:00
Nicolas Sebrecht
fc6c12d96c improve message "error 111" if connection failed
Raise OfflineImapError with severity REPO explaining that the connection failed.
Before, no valuable information was given to the user.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-25 20:32:41 +02:00
Tom Lawton
da36c5c1e7 Handle abort messages from GMail
Without this patch, we try to NOOP on a bad connection and crash messily.

Signed-off-by: Tom Lawton <tlawton@gmx.de>
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Ethan Glasser-Camp
2cc2ead503 Let the user configure how long to IDLE for
This commit was originally by James Bunton <jamesbunton@fastmail.fm>.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Ethan Glasser-Camp
36a0680191 Add check for IDLE in capabilities
If the server doesn't support IDLE, we fall back to the standard
noop() keepalive.

This commit was originally by James Bunton <jamesbunton@fastmail.fm>.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Ethan Glasser-Camp
d47bd1ff89 Change keepalive() to spawn IdleThreads
This is the commit that enables IDLE support. In order to do this, we
hijack the keepalive method. Instead of just sending NOOPs, it now
sends IDLE and responds accordingly, thanks to the IdleThread class.

This code was originally by James Bunton <jamesbunton@fastmail.fm>.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Ethan Glasser-Camp
9fa6bcdf82 Add IdleThread class
This encapsulates the logic for sending a keepalive/IDLE call,
including starting a sync if needed.

This code was originally by James Bunton <jamesbunton@fastmail.fm>.  I
modified the idle() method to put the select() call after
acquireconnection().

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Ethan Glasser-Camp
52cefb582c Recognize configuration for idlefolders
Mark this option as experimental and document its shortcomings in
MANUAL.rst.

This code was originally by James Bunton <jamesbunton@fastmail.fm>.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Nicolas Sebrecht
89a5d25263 fix broken ui Blinkenlights in multi-threaded mode
This is a regression introduced by commit d5493fe894
[threadutil: explicitly import get_ident from thread].

The threadid attribute was wrongly removed from the ExitNotifyThread class.
Restore it.

Tested-by: Mark Foxwell <fastfret79@archlinux.org.uk>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-18 18:38:05 +02:00
Nicolas Sebrecht
2d31abde76 v6.3.4-rc1
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-16 18:54:22 +02:00
Nicolas Sebrecht
ea8c1c0b6a Merge branch 'ns/threadutil-remove-os._exit' into next 2011-05-16 18:26:40 +02:00
Nicolas Sebrecht
41c21f5a9f fix: allow debugtype 'thread' from command line
commit f7e9d95 [Imply single-threaded mode with -d(ebug) command line option]
was broken. It pretends to imply singlethreading each time _unless_ user
explicitly asks for the debug 'thread' option.

Change the "force singlethreading" check from

  (force single threading if last column is true)

+------------------------------------------+----------------+
|                                          |                |
+---------------------------------+        |                |
|((SINGLETHREADING   THREAD_MODE) | "AND") | "NOT"          |
+------------------+--------------+--------+----------------+
| True             |  True        |  True  |  False (wrong) |
+------------------+--------------+--------+----------------+
| True             |  False       |  False |  True          |
+------------------+--------------+--------+----------------+
| False            |  True        |  False |  True (wrong)  |
+------------------+--------------+--------+----------------+
| False            |  False       |  False |  True          |
+------------------------------------------+----------------+

To the correct one

+--------------------------------------------------+-------+
|                                                  |       |
+-----------------------------------------+        |       |
|(("NOT" SINGLETHREADING    THREAD_MODE)  | "AND") | "NOT" |
+-------------------------+---------------+--------+-------+
| False                   | True          | False  | True  |
+-------------------------+---------------+--------+-------+
| False                   | False         | False  | True  |
+-------------------------+---------------+--------+-------+
| True                    | True          | True   | False |
+-------------------------+---------------+--------+-------+
| True                    | False         | False  | True  |
+-------------------------+---------------+----------------+

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-16 18:18:21 +02:00
Nicolas Sebrecht
694e5772da threadutil: remove broken os.exit()
We are missing the import of 'os' python module since commit d839be3c61
(Sat Apr 16 20:33:35 2005 +0100) which was when John switched from SVN to Git.

Happily, it help us today: we still had no feedback for this missing import,
6 years later.  So, we can remove the os.exit() call safely.

That beeing said, we still don't know if the above sys.exit() was ever touched.
My guess is that it never was. Keep this (hopefully) commented statement to
ensure the thread terminate and not play too much with the Murphy's law. :-)

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-13 00:56:52 +02:00
Nicolas Sebrecht
21875852eb cleanup: remove unused initextnotify() function and inited variable
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-12 18:38:45 +02:00
Nicolas Sebrecht
136237b7dc refactoring: simplify the semaphorewait logic
The semaphorewait()/waitforthread() logic is usefull for IMAP starting
connections. We actually use it in imapserver only.

This patch removes the over-engineered factorized methods. It tend to simplify
the code by cleaning out a chain of two direct calls with no other processes.

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-12 18:30:39 +02:00
Nicolas Sebrecht
d5493fe894 threadutil: explicitly import get_ident from thread
The python module thread is the low-level module we should avoid to use in favor
of threading. We still need it to support old python because Thread.ident
doesn't exist before python 2.6:

	http://docs.python.org/library/threading.html#threading.Thread.ident

Make it clear we should avoid it.

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-12 18:28:49 +02:00
Nicolas Sebrecht
8a34edc8ca cleanup import satements
- conform to PEP8
- explicitly define symbols instead of 'import *'
- remove unused import

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-12 18:26:29 +02:00
Nicolas Sebrecht
a39128516b explicitly define symbols to import instead of 'import *'
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-09 22:43:03 +02:00
Nicolas Sebrecht
b25a72f8c8 cleanup: remove uneeded imports
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-09 22:42:15 +02:00
Sebastian Spaeth
c70a621709 Limit msg body length in debug output
We were outputting full message bodies to the debug log (often stderr),
and then again (as they go over the imaplib2 wire, imaplib logs
everything too). Not only is quite a privacy issue when sending in debug
logs but it can also freeze a console for quite some time. Plus it
bloats debug logs A LOT.

Only output the first and last 100 bytes of each message body to the
debug log (we still get the full body from imaplib2 logging). This
limits privacy issues when handing the log to someone else, but usually
still contains all the interesting bits that we want to see in a log.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-09 20:58:46 +02:00
Nicolas Sebrecht
146b2abb6b Merge branch 'ns/testing-offlineimap' into next 2011-05-09 19:52:44 +02:00
Nicolas Sebrecht
bdeaebbb7d Merge branch 'ns/working-with-git' into next 2011-05-09 19:52:39 +02:00
Daniel Shahaf
632f1fe61f FAQ: add two entries concerning 'sslcacertfile'
Add a FAQ entry about non-verifying SSL certificates by default,
and another about how to generate a certificates file to feed to
the 'sslcacertfile' repository configuration item.

Signed-off-by: Daniel Shahaf <d.s@daniel.shahaf.name>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-09 19:36:18 +02:00
Nicolas Sebrecht
8465c832c3 FAQ: insert entry to explain how to test OfflineIMAP
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 18:42:30 +02:00
Nicolas Sebrecht
9a9379f8f3 add documentation about Git
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 17:17:07 +02:00
Nicolas Sebrecht
e7439b5d77 doc: introduce a new HACKING file
This aims to help newcomers to be very fast to involve into the project.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 17:17:07 +02:00
Nicolas Sebrecht
40900dcfb6 Merge branch 'ss/remove-siglistener-class' into next
Conflicts:
	offlineimap/accounts.py
2011-05-08 14:37:47 +02:00
Sebastian Spaeth
cff792d381 LocalStatusSQLite: Fix bug when deleting messages
The syntax was not right, and deleting messages from the LocalStatus
failed. (We passed in the full list of uids and we need to pass in one
uid at a time (as a tuple). Deleting messages works now.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 14:34:55 +02:00
Sebastian Spaeth
89619838b0 Remove weird SigListener class
The SigListener class was used to queue folders that we need to sync and
to receive "resync" and "abort" signals. It was undocumented and weird
and we had to pass "siglisteners" through the whole program.

Simply do away with it, and make 2 functions in the Account() class:
set_abort_event and get_abort_event which can be used to set and check
for such signals. This way we do not need to pass siglisteners all over
the place. Tested Blinkenlights and TTYUI uis to make sure that SIGUSR1
and SIGUSR2 actually still work.

Document those signals in MANUAL.rst. They were completly undocumented.

This simplifies the code and interdependencies by passing less stuff
around. Removes an undocumented and weirdly named class.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 14:25:16 +02:00
Sebastian Spaeth
b3a383d151 accounts: handle OfflineImapError severity FOLDER
Throw an OfflineImapError when SELECTing a folder is unsuccessful and
bail out with a FOLDER serverity. In accounts.py catch all
OfflineImapErrors and either just log the error and skip the folder or
bubble it up if it's severe.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 13:56:04 +02:00
Sebastian Spaeth
dd82f213f0 accounts: fix broken warn statement
Use two %s in the message for both string parameters.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 12:48:59 +02:00
Nicolas Sebrecht
ac27c93c83 Merge branch 'ns/doc-install-from-git' into next 2011-05-07 13:51:59 +02:00
Sebastian Spaeth
f4081985dc Prettify and use new uidexists() helper function
Make the folder classes use uidexists() more. Add some code
documentation while going through.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:50:22 +02:00
Sebastian Spaeth
0318c6ad34 Create LocalStatus or LocalStatusSQLite folders
Depending on the configuration we use the plain text or the new
experimental sqlite backend for the LocalStatus cache. Make plain text
the default status backend but allow people to configure
status_backend=sqlite in their [Account ...] section.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:35:03 +02:00
Sebastian Spaeth
0af9ef70a7 Factor out SQL retries
Test if sqlite is multithreading-safe and bail out if not. sqlite
versions since at least 2008 are.
But, as it still causes errors when 2
threads try to write to the same connection simultanously (We get a
"cannot start transaction within a transaction" error), we protect
writes with a per class, ie per-connection lock. Factor out the retrying
to write when the database is locked.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
af25c2779f repository.LocalStatus: Remove code duplication
Make getfolders() invoke getfolder() for each folder rather than
duplicating code. Also add a forgetfolders() implementation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00
Sebastian Spaeth
e1e9c8e831 Use self.doautosave rather than self.dofsync
doautosave was a useless variable before (it was *always* 1). So we
remove the self.dofsync variable and store in doautosave whether we
should fsync as often as possible (which really hurts performance).

The sqlite backend could (at one point) use the doautosave variable to
determine if it should autocommit after each modification.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-07 13:29:11 +02:00