Commit Graph

1409 Commits

Author SHA1 Message Date
Eygene Ryabinkin
6cbd2498ae Refactoring: make functions to be private if we can
Make external API of class/module to be smaller, explicitely mark
all internal functions.  Also annotate methods that are implemented
as the part of the parent class interface.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-04-30 19:35:56 +04:00
Stefan Huber
af2d6dc5e1 Fix: Undefined variable 'this' should be 'self'
Pointyhat-to: Eygene Ryabinkin
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-03-06 10:48:34 +04:00
Eygene Ryabinkin
829c9cf4be Introduce dynamic filtering for repository folders
Allow people who want folder filtering to depend on the external
conditions or to make it dynamic for other reasons, to do what
they want.

New repository configuration knob 'dynamic_folderfilter' was
introduced; it defaults to 'False' that matches historical behaviour.

GitHub: #73
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-03-02 16:37:41 +04:00
Eygene Ryabinkin
b01274ce38 Fix double release of IMAP connection object
This commit fixes the case when we're invoking releaseconnection()
for a given imapobj twice.

This bug manifests itself as
{{{
ValueError: list.remove(x): x not in list

  File "[...]/offlineimap/folder/IMAP.py", line 615, in savemessage
    self.imapserver.releaseconnection(imapobj)
  File "[...]/offlineimap/imapserver.py", line 130, in releaseconnection
    self.assignedconnections.remove(connection)
}}}

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-02-26 17:51:48 +04:00
Eygene Ryabinkin
8bc2f35bf6 OfflineIMAP 6.5.5 is out
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-10-04 17:03:06 +04:00
Eygene Ryabinkin
254e848140 Rolling in release candidate #3 for 6.5.5
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-09-20 14:35:37 +04:00
Eygene Ryabinkin
57adfc23a5 Fix parsing of quoted strings
When imaputil was parsing quoted strings, it treated "abcd\\"
as incomplete quoted string having escaped quote, rather than
properly-quoted string having escaped backslash.

GitHub issue: https://github.com/OfflineIMAP/offlineimap/issues/53
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-09-19 22:56:55 +04:00
Eygene Ryabinkin
be1c72ea5f Updated bundled imaplib2 to 2.36
2.36 it includes support for SSL version override that was integrated
into our code before, no other changes.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-09-19 17:19:37 +04:00
Eygene Ryabinkin
2df5b716b5 Updated bundled imaplib2 to 2.35:
- Fix for Gmail sending a BYE response after reading >100 messages
   in a session.

 - Includes fix for GitHub#15: patch was accepted upstream.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-09-02 09:08:50 +04:00
Eygene Ryabinkin
1ef506655c Catch two instances of untested regexp matches
They could possibly lead to the problems mentioned in
  https://github.com/OfflineIMAP/offlineimap/issues/6
though there are no sound evidences for this.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-30 21:50:08 +04:00
Eygene Ryabinkin
69765a3ef0 Properly call error() function from ui.UIBase
Second argument is exception traceback, not the message; without this
tracebacks like mentioned in
  http://permalink.gmane.org/gmane.mail.imap.offlineimap.general/5712
were happening when this exception handling block was hit.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-30 21:49:09 +04:00
Eygene Ryabinkin
f2c858330f Always show full tracebacks at the end of the run
This greatly simplifies developer's life and will, possibly, allow
users familiar with Python to debug and fix the problems by
themselves.

We, possibly, should not give tracebacks for the problems like
"can't open connection", but this is up to the caller of this
routine not to provide traceback in this case.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-27 20:25:30 +04:00
Eygene Ryabinkin
d55e4ef15e imaplib2: fix handling of zero-sized IMAP objects
self._expecting_data was used both as the expected data length and the
flag that we expect some data.  This obviously fails when advertized
data length is zero, so self._expecting_data_len was introduced to
hold the length of the expected data and self._expecting_data was left
as the flag that we expect the data to come.

GitHub issue: https://github.com/OfflineIMAP/offlineimap/issues/15
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-27 20:25:30 +04:00
Eygene Ryabinkin
95aea5e489 Add new expansion key for mbnames.peritem config variable
It is called localfolders and holds expanded name for the same
variable for the local repository of the account that is being
processed.

GitHub issue: https://github.com/OfflineIMAP/offlineimap/issues/21
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-27 20:25:23 +04:00
Eygene Ryabinkin
8b6f10e2e7 Pass folder names for the foldersort function
Bring the description in the template offlineimap.conf in sync to the
actual implementation: pass folder names to the sorting function, not
the offlineimap.folder.IMAP.IMAPFolder objects themselves.

GitHub issue: https://github.com/OfflineIMAP/offlineimap/issues/27
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-27 16:27:36 +04:00
X-Ryl669
3bc66c0858 Add support for alternative message date synchronisation
Global or per-repository option utime_from_message tells OfflineIMAP
to set file modification time of messages pushed from one repository
to another basing on the message's "Date" header.

This is useful if you are doing some processing/finding on your
Maildir (for example, finding messages older than 3 months),
without parsing each file/message content.

From: Cyril RUSSO <boite.pour.spam@gmail.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-09 10:48:00 +04:00
Andreas Mack
e26827c1cb Make authentication mechanisms configurable
Added configuration option "auth_mechanisms" to the config file:
it is a list of mechanisms that will be tried in the specified order.

Author: Andreas Mack <andreas.mack@konsec.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-08 22:34:30 +04:00
Steve Purcell
968d5520da Allow transport-type tunnels to the IMAP servers
It's nice to set up an ssh tunnel command which forwards an IMAP tcp
port inside an encrypted session, e.g. with ssh's "-W" flag.  In this
case the tunnelled connection still requires authentication inside
IMAP session, because this is transport-only tunnel that substitutes
normal TCP/SSL connection.

New directive, 'transporttunnel' was added: it specifies the command
that will create the tunnel.  Only one type of tunnel must be
specified for a single repository: we can't have both preauthenticated
and transport-type tunnels, they won't chain together.

From: Steve Purcell <steve@sanityinc.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-06 12:18:21 +04:00
Andreas Mack
acaa96291d Add SASL PLAIN authentication method
- this method isn't as deprecated as IMAP LOGIN;

 - it allows to keep hashed passwords on the server side;

 - it has the ability to specify that the remote identity
   is different from authenticating username, so it even
   can be useful in some cases (e.g., migrated mailboxes);
   configuration variable "remote_identity" was introduced
   to leverage this functionality.

From: Andreas Mack <andreas.mack@konsec.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-06 01:51:41 +04:00
Eygene Ryabinkin
7d313f49dc Refactored authentication handling
- created helper routine that will do authentication;

 - routine tries each method in turn, first successful
   one terminates it: makes things easier to read
   and handle;

 - renamed plainauth() inside offlineimap/imapserver.py
   to loginauth(): the function does IMAP LOGIN authentication
   and there is PLAIN SASL method, so previous name was
   a bit misleading;

 - slightly improved error reporting: all exceptions during
   authentication will be reported at the end of the run;

 - now loginauth() is never called if LOGINDISABLED is advertized
   by the server; it used to be invoked unconditionally when
   CRAM-MD5 fails, but we should respect server's opinion on
   how to handle its users.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-06 01:10:10 +04:00
Adam Spiers
d39a1f864f make savemessage() handle NO response to APPEND correctly
IMAP servers can return `NO` responses to the `APPEND` command,
e.g. here's an example response from Groupwise's IMAP server:

    NO APPEND The 1500 MB storage limit has been exceeded.

In this case, savemessage() should abort the repository sync rather
than returning UID 0 which would cause the local copy of the message
being saved to get irreversibly deleted.

Signed-off-by: Adam Spiers <offlineimap@adamspiers.org>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-02 16:41:54 +04:00
Johan Herland
56b0c5dbac Allow custom sorting of mailboxes for mbnames
mutt-sidebar and, probably, other MUA show mailboxes in the order
they are listed in the file written by mbnames.  Therefore, to allow
customization of the order with which mailboxes are listed, introduce
the new 'sort_keyfunc' directive in the [mbnames] section.

'sort_keyfunc' must be a function that will be called once for each
mailbox.  It must accept the only argument -- a dict with 2 items,
'accountname' and 'foldername', and should return an object that
will be used as the sorting key for each mailbox.

Default key function returns (d['accountname'], d['foldername']),
thus sorting by account name and then by the folder name.

Signed-off-by: Johan Herland <johan@herland.net>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-02 14:46:44 +04:00
mxgr7
0d992ee7d3 Execute pre/post hooks for IDLE-toggled syncs
Make IDLE syncs be equal to the regular synchronisations
in respect to pre-sync and post-sync hooks.

From: mxgr7 <maxgerer@gmail.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-02 13:53:21 +04:00
Eygene Ryabinkin
67d68c2fc8 IMAP: simplify locking and fix potential lock loss
Run the locked code under 'with': this guarantees that lock
will be released in any case.

This modification also avoids the case when our thread wasn't running
locked when exception was caught, another thread got the lock, our
code checked it via self.connectionlock.locked() and errorneously
released the lock thinking that is was running locked.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-07-22 00:08:39 +04:00
Eygene Ryabinkin
41cb0f577f Prune trailing whitespaces from code and documentation
They are redundant in all pruned cases and sometimes even create some
problems, e.g., when one tries to jump through paragraphs in vi.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-07-21 23:00:23 +04:00
Dmitrijs Ledkovs
e8c40a9285 Merge branch 'master' into next 2013-07-10 02:49:52 +01:00
xnox
05284b4df8 Merge pull request #28 from aspiers/fix-typos
fix typos
2013-07-09 18:30:28 -07:00
Dmitrijs Ledkovs
7a21cab6c5 Correct whitespace & add docs about remoteporteval. 2013-07-10 02:17:01 +01:00
Ryan Kavanagh
2bacdb7fa3 Allow setting IMAP servers' SSL version
We now allow setting the SSL version used when connecting to IMAPS servers, and
do so via the `ssl_version` configuration option. We default to the current
practice (letting python's "ssl" library automatically detect the correct
version). There are however rare cases where one must specify the version to
use.

Signed-off-by: Ryan Kavanagh <rak@debian.org>
2013-07-08 10:57:58 -04:00
Kim Lidström
dde021ce52 Added remoteporteval 2013-06-20 11:38:45 +02:00
Adam Spiers
aa5202396c fix typos 2013-03-27 12:42:32 +00:00
Eygene Ryabinkin
611f6e89c0 IMAP class: don't suggest multithreading in single-threaded mode
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-02-11 23:34:28 +04:00
Eygene Ryabinkin
f4140cbbed Create global instance of command-line options
This eases testing of option values inside the code.  This instance
is implemented as the read-only copy of the obtained 'options' object,
so callers won't be able to modify its contents.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-02-11 23:34:18 +04:00
X-Ryl669
5c842c01bd Fix deadlock for IMAP folder synced in singlethreaded mode
The problem lies in the fact that offlineimap.folder.Base's method
syncmessagesto_copy() uses threaded code everytime it is suggested by
the derived class's suggeststhreads() (currently, only IMAP does this
suggestion), but offlineimap/init.py will not spawn the
exitnotifymonitorloop() from offlineimap.threadutil.

The root cause is that ExitNotifyThread-derived threads need
offlineimap.threadutil's exitnotifymonitorloop() to be running the
cleaner for the exitthreads Queue(), because it fills the queue via
the run() method from this class: it wants to put() itself to the
Queue on exit, so when no exitnotifymonitorloop() is running, the
queue will fill up.  And if this thread is an instance of
InstanceLimitedThread that hits the limit on the number of threads,
then it will hold the instancelimitedsems[] semaphore will prevent
other InstanceLimitedThread()s of the same name to pass its start()
method.

The fix is to avoid using threaded code if we're running
single-threaded.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Obtained-from: X-Ryl669 <boite.pour.spam@gmail.com>
2013-02-05 07:53:25 +04:00
Eygene Ryabinkin
1c8917b036 And add forgotten __init__.py for offlineimap.utils
I should learn to check my commits when rebasing and reordering.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-02-05 07:52:25 +04:00
Eygene Ryabinkin
de84c3941c Implement stack trace dump for all running threads on SIGQUIT
This is handy when we're debugging the thread locks: we can try to
understand which thread does what and how it was called.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-02-02 16:25:37 +04:00
Dmitrijs Ledkovs
83e8fca2e0 Make SIGHUP singal handler equivalent to SIGTERM and SIGINT.
offlineimap has several frontends that encourage running it from a
terminal under an X session. When X session closes for a system
shutdown, the terminals exit, after sending SIGHUP to their children.

Previously SIGHUP was treated to be equivalent to SIGUSR1, i.e. wake
up and sync all accounts. This causes delays during shutdown.

According to Wikipedia [0], SIGHUP has been repurposed from a
historical meaning to one of:
 * re-read configuration files, or reinitialize (e.g. Apache, sendmail)
 * controlling pseudo or virtual terminal has been closed

I believe second meaning is more appropriate for offlineimap, and
hence this patch makes SIGHUP to be handled in the same way SIGTERM
and SIGINT are handled.

[0] http://en.wikipedia.org/wiki/SIGHUP

Debian-Bug: http://bugs.debian.org/670120
Reported-By: Steve Langasek <steve.langasek@canonical.com>
Signed-off-by: Dmitrijs Ledkovs <xnox@debian.org>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-01-26 23:05:46 +04:00
Tobias Thierer
caef9a72fc Fix IMAP folder throwing away time zone when parsing email Date headers
Fix imapfolder.getmessageinternaldate misparsing the Date:
header from emails due to a bug or surprising behaviour by
email.utils.parsedate. This is because email.utils.parsedate's
return value contains the unadjusted hour value from the string
parsed but does not include information about the time zone in
which it is specified. For example (Python 2.7.3):

$ python -c "import email.utils;
  print email.utils.parsedate('Mon, 20 Nov 1995 19:12:08 -0500')"
 (1995, 11, 20, 19, 12, 8, 0, 1, -1)

(the -1 is the isdst field); the -0500 time zone is completely
ignored, so e.g. the same input with time "19:12:08 +0300" has
the same result. When passed to time.struct_time as allowed per
the parsedate documentation, this time is interpreted in GMT and
thus deviates from the correct value by the timezone offset
(in this example, -5 hours).

I consider this a bug in email.utils.parsedate: In my opinion,
since the return value of the parsetime doesn't include a timezone,
it should be expressed in terms of UTC rather than in terms of the
time zone from the Date header; the existence of
email.utils.parsedate_tz, to which I've switched, indicates that
maybe the authors were aware of this problem.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-10-03 11:16:13 +02:00
Sebastian Spaeth
721035eace Release v6.5.5-rc2
(Same as rc1, but with Changelog updated and version number bumped)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-09-05 17:40:09 +02:00
Sebastian Spaeth
3476e9ab36 Fix fallout when filtering folders
Previous commit e7ca5b25cb combined
checks for filtered folders in one place. However, it turns out there
was a reason to have them separate. getfolder() on a non-existent Maildir
fails and there might not be an equivalent local Maildir folder for a
filtered out IMAP folder.

Fix this by first checking if the remote folder should be filtered, and
only then retrieving the local folder (which should exist then).

This bug was found by our test suite!

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-09-01 03:15:08 +02:00
Sebastian Spaeth
49c11fc8f0 LocalStatus._folders needs to be {} not None
the LocalStatus._folders cache was changed to be a dict that can be
searched for names. One instance were _folders was set to "None" was
accidentally left over. Fix this.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-09-01 02:58:14 +02:00
Dave Abrahams
7d1d5283f8 No need to filter Maildir folders here
commit e94642bb4d centralized folder filtering by using the
repository.should_sync_folder() function. Therefore there is no need
to check for folderfilter in the Maildir backend separately.

Origina patch by Dave, split into 3 by Sebastian

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-09-01 02:49:00 +02:00
Dave Abrahams
e94642bb4d Determine folder syncing on Folder initialization
Rather than later when retrieving them by IMAP, we determine the
"sync_this" value of a folder on Folder() initialization.

This also fixes a bug where folder structure was not propagated when
a folder was filtered out but included in the folderincludes list.

Patch by Dave, split and modified slightly by Sebastian

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-09-01 02:30:46 +02:00
Dave Abrahams
e7ca5b25cb Combine checks for ignored folders in one place
- Factor out the code to find a local folder given a remote folder

Patch by Dave, split and modified by Sebastian.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-09-01 02:16:06 +02:00
Sebastian Spaeth
1b54b85f20 Reuse LocalStatus() folders rather than recreate instances
If we ask twice for a LocalStatusFolder via getfolder(), we would
get a newly created instance each time. This can lead to problems,
as e.g. write locks protecting files only work within the same Folder
instance. Make it so, that we cache all Folder instances that we have
asked for and hand back the existing one if we ask again for it,
rather than recreate a new instance.

Also, make getfolders() a noop for LocalStatus. We attempted to
derive the foldername from the name of the LocalStatusfile. However,
this is not really possible, as we do file name mangling
(".$" -> "dot", "/" -> ".") and there is no way to get the original folder
name from the LocalStatus file name anyway.

This commit could potentially solve the "file not found" errors, that people
have been seeing with their LocalStatusCache files. If we have 2
instances of a LocalStatusFolder pointing to the same file, our locking
system would not work.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-09-01 01:19:19 +02:00
Sebastian Spaeth
a1dc76ae91 Don't output initial blurb in "quiet" mode
When logging to a file using the -l switch, we would still write an initial
banner to the file. This was never intended. Quiet should be really quiet
unless it experiences an error. Simplify the logging statement, to do nothing
if logevel is set to "WARNING" aka quiet.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-08-31 23:11:11 +02:00
Sebastian Spaeth
3cb2ddccb8 Use "with lock" pattern
While looking at the code to investigate if an why we sometimes don't
seem to honor the write lock, I made it use the more modern "with lock:"
pattern.

Still have not found out how we could ever be using 2 instances of the
LocalStatusFolder for the same folder.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-08-31 22:34:53 +02:00
Sebastian Spaeth
04ffae2f00 Exit with nice error message failing to get repositories
If we throw an OfflineImapError in case of the Repository()
initialization, we display the nice error message and exit rather
than bomb out with a traceback. Misconfiguring a repository name in
the configuration file is now nicely pointed out to the user.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-08-31 20:34:54 +02:00
Sebastian Spaeth
103524c979 Throw OfflineImapErrors rather than weird Exceptions
When misconfiguring OLI, e.g. by specifying a repository name that was
not configured anywhere, we would bomb out with cryptic "NoSectionError".
Throw OfflineImapError that explains what has happened. We still need to
avoid throwing exceptions with Tracebacks here though.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-08-31 20:31:43 +02:00
Sebastian Spaeth
925a5bcae1 Fix cut-off line
Commit 6e5fbebc introduced a cut-off line. Fixing that.
All tests ran successfully.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-08-21 17:00:39 +02:00
Sebastian Spaeth
a134301ac5 Improve 'find first quotation' regex
Reported by http://www.dfranke.com/blog/2012/08/20/offlineimap-error-beim-syncen-mit-lotus-domino/
our 'find the first quote possible containing encoded quotation
marks' regex did not seem to have caught all cases. E.g. "\\".

Verified the fix as good. Thanks Daniel Franke.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-08-21 16:58:32 +02:00
Vladimir Nesov
6e5fbebc0e fix: createfolders setting is ignored if readonly is True
If 'readonly' is True, folders shouldn't be created (regardless of
'createfolders' option). With old behavior, instead folders were always created
when 'readonly' is True (even if 'createfolders' was also False), which is a
serious bug (offlineimap was creating folders in all read-only repositories).
'createfolders' should only play a role if 'readonly' is False, in which case
folders should only be created if 'createfolders' is True.

Submitted-by: Vladimir Nesov <robotact@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2012-08-20 21:12:56 +02:00
Eygene Ryabinkin
131ac5c827 IMAPlib mixin class: pass 'readonly' exception to our callers
This will allow our callers who are capable of dealing with
readonly folders to properly detect this condition and act
accordingly.

One example is Gmail's "Chats" folder that is read-only,
but contains logs of the quick chats.

Minor Changelog improvements.

Tested-by: Abdó Roig-Maranges <abdo.roig@gmail.com>
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2012-08-11 11:59:26 +02:00
Dmitrijs Ledkovs
d45872e59c Add missing OfflineImapError import in folder/UIDMaps.py
Bug-Debian: http://bugs.debian.org/671279
Reported-by: Ansgar Burchardt <ansgar@debian.org>
Signed-off-by: Dmitrijs Ledkovs <xnox@debian.org>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2012-07-30 20:10:11 +02:00
Sebastian Spaeth
36156fa985 Remove the APPENDUID hack, previously introduced
As Gmail was only announcing the presence of the UIDPLUS extension
after we logged in, and we were then only getting server
capabilities before, a hack was introduced that checked the
existence of an APPENDUID reply, even if the server did not claim
to support it.

However, John Wiegley reports problems, where the APPENDUID would
be None, and we attempt to go this path (it seems that imaplib2
returns [None] if there is no such reply, so our test here for "!="
might fail. Given that this is an undocumented imaplib2 function
anyway, and we do fetch gmail capabilities after authentication,
this hack should no longer be necessary.

We had problems there earlier, where imapobj.response() would
return [None] although we had received a APPENDUID response from
the server, this might need more debugging and testing.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-06-06 10:13:42 +02:00
David Logie
a614f9a735 Fix str.format() calls for Python 2.6.
Python 2.6 requires a slightly different string formatting that >2.7.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-06-02 18:25:51 +02:00
Sebastian Spaeth
686561b42c Release v6.5.4
See Changelog for details

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-06-02 13:41:20 +02:00
Sebastian Spaeth
0752c123f5 Implement the "createfolders" setting for repositories
By default OfflineImap propagates new folders in both
directions. Sometimes this is not what you want. E.g. you might want
new folders on your IMAP server to propagate to your local MailDir,
but not the other way around. The 'readonly' setting on a repository
will not help here, as it prevents any change from occuring on that
repository. This is what the `createfolders` setting is for. By
default it is `True`, meaning that new folders can be created on this
repository. To prevent folders from ever being created on a
repository, set this to `False`. If you set this to False on the
REMOTE repository, you will not have to create the `Reverse
nametrans`_ rules on the LOCAL repository.

Also implement a test for this

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-05-08 16:56:45 +02:00
Sebastian Spaeth
644b9f0bb9 Implement .readonly property for repositories
Set the value once on repository initialization to centralize the
default value.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-05-08 16:31:19 +02:00
Sebastian Spaeth
61e754c65e Do not try to release IMAP connection twice
Reported by sharat87 in https://github.com/spaetz/offlineimap/pull/38,
he would often get an unhandled Exception when trying to
releaseconnection() a connection that was not in the pool of
connections.

The reason this could happen is that when folder.IMAP.quickchanged()
raises an Exception in select(), we would release the connection in the
"except" handling, and than release the same connection in the "finally"
clause, which led to the error. The right thing is to only release the
connection once, of course.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-04-21 13:32:14 +02:00
Sebastian Spaeth
895e709bf2 Fix SSL fingerprint was not always checked
As reported by James Cook, we would not check the fingerprint of the SSL
server, as we were looking for the 'ssl' module in locals() rather than
globals(). Ooops!

Rather than using globals() though, I simply remove the by-now
superfluous check. We now rely on python2.6 and we unconditionally
import the SSL module in any case, so it needs to be there.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-04-19 18:26:29 +02:00
Christoph Höger
b9af72ea11 Curses UI: Reset the warn method before terminate
The warn() method tries to set the color to red. This leads to a garbled
tty after endwin() has been called. So lets simply use the UIBase
implementation.

Signed-off-by: Christoph Höger <christoph.hoeger@tu-berlin.de>
2012-04-19 17:20:49 +02:00
Sebastian Spaeth
8fe8bbe366 imaplib2: bump upstream version 2.29 --> 2.33
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-04-19 13:22:03 +02:00
Sebastian Spaeth
679c491c56 bugfix release v6.5.3.1
a single bugfix when no dry-run option was given in the config file.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-04-03 01:39:28 +02:00
Sebastian Spaeth
c708e36ff7 Don't fail if no dry-run setting has been specified
Somehow we failed if no dry-run setting had been specified in the config
file. This got caught thanks to extending the test suite with a stock
configuration.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-04-03 01:13:27 +02:00
Sebastian Spaeth
d3427a664e v6.5.3 release
see changelog

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-04-02 23:47:32 +02:00
Eygene Ryabinkin
2f88b0296a Fix Curses interface for Python 2.6
In 2.6 all logging classes are not the new-style ones, so they
have no mro() method and, thus, we can't use super() for them.
Since CursesLogHanler is singly-inherited method, there will
be no problems in usage of the explicit superclass name.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-04-02 23:39:36 +02:00
Sebastian Spaeth
29ba2fc523 Warn about nonsensical config option 'sep' for IMAP repositories
We autodetect the folder separator on IMAP servers and ignore any 'sep'
setting in the repository section for IMAP servers. Detect if there is
such a setting and warn the user about it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-04-02 23:26:59 +02:00
Sebastian Spaeth
3c481d9ce5 -f command line option only works on the untranslated remote names
Previously folderfilters had to match both the local AND remote
name which caused unwanted behavior in combination with nametrans
rules. Make it operate on the untranslated remote names now and
clarify in the command line option help text.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-24 14:52:30 +01:00
Sebastian Spaeth
925538f02d docs: Fix docstrings to proper .rst syntax
Prevents compile errors when creating the user documentation.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-24 11:20:22 +01:00
Sebastian Spaeth
0e6b4ae798 Revert "Clean up and improve APPENDUID handling"
This reverts commit 4d47f7bf3c.

This is one of two candidates for introducing the instabilities that
John Wiegley observed. We need to reintroduce with careful testing only.

The original patch has been mostly reverted.
2012-02-24 08:35:59 +01:00
Sebastian Spaeth
bc73c11239 Revert "Don't CHECK imapserver after each APPEND"
This reverts commit 47390e03d6.

It is one of two potential candidates for the APPENDUID
regression that John Wiegley reported. We need to examine this
carefully before reintroducing this patch.

Resolved Changelog.draft.rst conflict.
2012-02-24 08:31:31 +01:00
Sebastian Spaeth
aa01fe754a Improve command line option help text
Shuffle around and update command line option descriptions in the
docs.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 13:17:05 +01:00
Sebastian Spaeth
256a26a649 dry-run mode: Protect us from actually deleting a message in dry-run mode
Document which functions honor dry-run mode and which don't.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 13:17:05 +01:00
Sebastian Spaeth
5ef69e95c0 Prevent modifications on a folder level to occur in dry-run
Prevent savemessage(), and savemessageflags() to occur in dryrun mode in
all backends. Still need to protect against deletemessage().

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 13:17:05 +01:00
Sebastian Spaeth
b6807355b5 Implement ui.makefolder and abort repo.makefolder() in dry-run mode
IMAP, Maildir, and LocalStatus abort if in dry-run mode. IMAP and Maildir
will log that they "would have" created a new folder.

This will probably fail later on as we can not cache messagelists on
folder that don't exist, so --dry-run is not yet safe when new folders
have been created.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 13:17:03 +01:00
Sebastian Spaeth
33f55b5362 Implement dry-run on Account() level
1) Set attribute self.dryrun depending on whether we are in dry-run mode.
2) Don't actually call hooks in --dry-run (just log what you would
   invoke
3) Don't write out the mbnames file in --dry-run mode.

Repository, and Folder levels still need to be protected in dry-run mode
as of now.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 10:53:10 +01:00
Sebastian Spaeth
b7e0a51751 Add command line option --dry-run
And set the [general]dry-run=True setting if yes. It is not used yet.
Also set ui.dryrun to True so we can output what WE WOULD HAVE DONE in
dryrun mode.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 10:32:04 +01:00
Sebastian Spaeth
a8c6407f50 Implement CustomConfig.set_if_not_exists()
A convenience helper function that allows to set a configuration value
if the user has not explicitly configured anything ie the option does
not exist yet in the configuration. It won't do anything, if the option
exists.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 10:28:59 +01:00
Sebastian Spaeth
bf44d30b46 UIDMaps: Better error message when not finding a mapping
Bail out with a better Exception and error text. The whole mapped
UID situation needs to be improved though.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 08:48:59 +01:00
Sebastian Spaeth
7da50e638d folder/IMAP: better error when savemessage fails
If we cannot identify the new UID after a sendmessage(), log a better error
message, including the server response for better debugging.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-16 16:49:06 +01:00
Sebastian Spaeth
ac033c68fd Improve nametrans local->remote folder syncing
While improving the test suite, I noticed that we would not create folders on
the remote in some cases when we should (yay for test suites!). This is because
we were testing the untransposed LOCAL foldername and check if it existed on
the remote side when deciding whether we should potentially create a new folder.

Simplify the code by transposing the LOCAL folder names in dst_hash, saving us
to create another confusing "newsrc" temp variable. Make the code a bit more
readable by using dst_name_t to indicate we operate a transposed folder name.

This now passes test 03 (using invalid nametrans rules) when test 03 would pass
before.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-16 11:45:18 +01:00
Sebastian Spaeth
189d78cc5c sync_folder_structure: make more readable
Rename variable src_name to src_name_t to indicate that it is the transposed
name. Also rather than testing the hash thingie, we can simply test for
"if source_name_t in dst_folders" now.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-16 11:12:07 +01:00
Sebastian Spaeth
10dd317026 Folder: Implement __eq__ for folders
This allows to compare folders directly with strings. It also allows
constructs such as "if 'moo' in repo.getfolders()".

See the code documentation for the exact behavior (it basically is equal if
it's the same instance *or* a string matching the untranslated folder name.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-16 11:03:33 +01:00
Sebastian Spaeth
19c014c6cd Implement foldersort in a python3 compatible way
By default we sort folders alphabetically (for IMAP) according to their
transposed names. For python3, we need to bend a bit backwards to still
allow the use of a cmp() function for foldersort. While going through, I
discovered that we never sort folders for Maildir.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-13 08:36:28 +01:00
Sebastian Spaeth
5c598d7e74 dict.has_key(a) --> a in dict
has_key() is gone in python3, so use a more modern syntax here.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
8aba2800e6 long(0) -> 0
There is no need to cast 0 to 'long' even if we want to compare it to long
numbers in modern pythons.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
046210b93d Fix mixed space/tabs
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
e6e708ec78 Don't use sort() on dict values()
This won't work in python3 anymore, so just use sorted() when needed.
In one case, we could remove the sort() completely as were were sanity checking
one line above, that we only having one UID as response which makes sorting
unneeded.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
b33f2452f0 Use "from . import" for relative imports
Will fail in python3 otherwise.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
014caddee6 folder/Base: Create unambigous MRO inheritence
class BaseRepository(object, CustomConfig.ConfigHelperMixin):

led to TypeError: Cannot create a consistent method resolution
order (MRO) for bases ConfigHelperMixin, object. Switching the inherited
classes helps.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
03566b2037 Replace thread.get_ident()
Replace low-level thread.get_ident() with threading.currentThread().ident.
This works both in python2.6 and python3. (thread is renamed _thread and its
direct use is not recommended)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
ba3a698a67 No need to test for types.StringType
all we want to know is if we got some string'ish type and testing for isinstance
 'basestring' is sufficient for that. Remove the import.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:43 +01:00
Sebastian Spaeth
dc67f515b6 no need for type(s) == types.StringType
All we want to do here is to test whether we got a string'ish type or a list
(literal), so testing for basestring will be fine.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
81fc20c7ca Remove python<2.6 import workarounds (set & ssl)
'set' is builtin since python2.6, so remove the imports. Also 'ssl' exists
since 2.6 and has everything we need, so no need for conditional import
tests here anymore.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
93f4a19778 python3: urlencode is in a different module
import urlencode from urllib.parse if neeeded for python3.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
8b6af63d83 [MachineUI] Remove unneeded "print 't'" statement
1) it's print('t') now and 2) this was a superfluous statement.
This broke python3

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
74fc902967 python3: import absolute package name
This import failed in python3, we need to either specify "." (relative) or
from OfflineImap.ui. (absolute). Done the latter.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
06a78b6112 python3: Queue -> queue
import queue (python3) if Queue is not available (python2)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
c5468ae599 raise Exception, "text" --> raise Exception("text")
To have the code work in python3, we need to convert all occurences of

raise Exception, "text" to be proper functions. This style also adheres to PEP8.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
9df7f34d4c Remove unused locked() function
We do not use ui.locked() anymore to output an error message, the text comes
directly from the exception.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
a8ab269ada Import configparser for python3 compatability
Attempt to load first ConfigParser and then configparser. At some point this
should be switched to do the python3 thing first.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
55da31c84b octal notation 0700 -> 0o700
Use octal notation that python3 understands. Works >=python2.6

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
0844d27f9f except Ex, e: --> except Ex as e:
Nudge us towards python3 compatability by converting deprecated python2 syntax.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
7f22d89872 fix type: logging.info --> logging.INFO
We mean the (numeric) logging level here and not the info() function.
logger.isEnabledFor() takes the logging level as argument,
obviously. This was a stupid typo that failed under python3.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:40:02 +01:00
Sebastian Spaeth
a242b985bf Release v6.5.2.1
See changelog for details

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-04 21:11:44 +01:00
Sebastian Spaeth
51728ed815 Remove Gmail realdelete option
It can lead to potential dataloss (see recent commit log where I added a
scary warning about it to offlineimap.conf).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-04 21:08:44 +01:00
Sebastian Spaeth
58450fb22e Enforce "Basic" UI when using --info switch
Blinkenlights does not work well (at all) when using the --info
switch. All we really want here is an output that can be pasted as
debugging information.

Enforce the usage of the "Basic" ui, when the --info switch is used.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 14:53:18 +01:00
Sebastian Spaeth
47390e03d6 Don't CHECK imapserver after each APPEND
Most servers support the UIDPLUS extension, and we don't have to search
headers after each uploaded message. There is no need to CHECK the imap
server after each message when there is no need to search headers.

I have not measured the performance impact on real world servers, but
this lets us do less unneeded work in the common case.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 14:43:12 +01:00
Sebastian Spaeth
4d47f7bf3c Clean up and improve APPENDUID handling
We were not cleaning out possibly existing APPENDUID messages before
APPENDing a new message. In case an old message were still hanging
around, this could *possibly* lead to retrieving and old UID. Things
should have been fine, but we do want to play safe here.

Also, make use of the "official" imaplib2 .response() command rather
than the internal _get_untagged_response() function.

Remove the hack that we would be looking for APPENDUID responses even if
the server claimed not to support the UIDPLUS ext. We now poll server
CAPABILITIES after login, and Gmail does provide us with the UIDPLUS
capability after login.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 14:33:16 +01:00
Sebastian Spaeth
36dd46abfc Fix warn() output for MachineUI
We used "self" but the paramter was called "s". Fixes a crash when we
ui.warn() (only when using the MachineUI).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 11:33:30 +01:00
Sebastian Spaeth
09ef5ab042 Remove stray debug line
This line was accidentally left laying around.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 11:31:41 +01:00
Sebastian Spaeth
fae48f309e Release 6.2.1-rc1
See Changelog.rst for details

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-20 11:10:42 +01:00
Sebastian Spaeth
eb04036da5 Make folders containing quotes work
imaputil.imapsplit did not cope with strings that contained encoded
quotation marks, e.g. a folder name '"Make" Magazine' would fail and
crash OfflineImap. Make it work by adapting the regex that we use to
extract the first quote to also work with encoded \" quotes. (We do no
sanity checks that there is an even number of such marks within a string
though)

This commit makes such folders work. This was reported and analyzed by
Mark Eichin.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>

Conflicts:

	Changelog.draft.rst
2012-01-20 11:07:33 +01:00
Sebastian Spaeth
165cf5cb4f Use modern "with open() as file"
Saves a LOC and guarantees that the file will be closed

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
de1e03b065 Tweak code documentation
Minor comment changes

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
d3c693be7d Rename folder.saveuidvalidity()
Simply rename to folder.save_uidvalidity()

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
eb616da602 Rename folder.isuidvalidityok()
Use nicer name folder.check_uidvalidity() and improve code documentation

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
297a618441 Rename folder.getsaveduidvalidity(), folder.getuidvalidity()
Modernize names to: folder.get_saveduidvalidity(), folder.get_uidvalidity()

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
9453e1d955 Fix getuidvalidity crash (UIDVALIDITY returning None)
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.

1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
   from the imaplib2 result stack, so subsequent calls would return "None".
   As various functions can invoke this, this led to some errors that we
   avoid by caching the current UIDVALIDITY value in the Folder instance.

There are more simplifications and improvements to be made.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:02:13 +01:00
Sebastian Spaeth
48802ae4b6 TTYUI: Fix missing "time" import
commit f2a94af5 introduced the use of time.sleep in ui/TTY.py without
importing it. This caused a regression in 6.5.2, crashing OfflineIMap
when in refresh mode.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 02:02:55 +01:00
Sebastian Spaeth
344b2f0b7a TTYUI: Fix python 2.6 compatibility
We were using super() on a class derived from logging.Formatter() which
worked fine in python 2.7. Apparently python 2.6 uses old-style classes
for this, so the TTYUI broke and crashed OfflineImap. This was
introduced in OLI 6.5.0, I think.

Fix it by calling logging.Formatter.... directly, rather than the
elegant super() (which I happen to like a lot more than is appropriate
in the python world).

Reported by Nik Reiman as github issue 23, should fix that issue.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-18 00:39:04 +01:00
Sebastian Spaeth
6dc74c9da5 Improve delete performance with SQLITE backend
When deleting many (eg 2000) mails using the SQLITE backend, this takes
a long time durig which OfflineImap can not be aborted via
CTRL-C. Thinking it had frozen permanently, I killed it hard, leaving a
corrupted db journal (which leads to awkwards complaints by OLI on
subsequent starts!). That shows that delete performance is critical and
needs improvement.

We were iterating through the list of messages to delete and deleted
them one-by-one execute()'ing a new SQL Query for each message. This
patch improves the situation by allowing us to use executemany(), which
is -despite still being one SQL query per message- much faster. This is
because rather than performing a commit() after each mail, we now do
only one commit() after all mails have been deleted.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-17 04:17:15 +01:00
Sebastian Spaeth
7a5768e471 Release v6.5.2
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-17 01:40:12 +01:00
Sebastian Spaeth
f2a94af522 Don't emit regular sleeping announcements in Basic UI
They are too noisy in the Basic UI, but are still included in TTYUI.
Reported by John Wiegley.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-10 07:20:49 +01:00
Cédric Krier
3028648e8f Don't fail if /etc/netrc is not readable 2012-01-09 16:39:12 +01:00
Sebastian Spaeth
9e65cfca21 Release v6.5.2-rc1
OfflineIMAP v6.5.2-rc1 (2012-01-09)
===================================
Commits v6.5.1.1 - v6.5.2-rc1:
note: Proper Changelog still in Changelog-draft.rst
d72bb88 Improve error message
3284e01 Revert "use .response() rather _get_untagged_response()"
81f194a mbnames should write out local and not nametransformed box names
7184ec2 Sanity check return value of UIDVALIDTY response
50de217 Allow to pass 'force' arg to selectro() to enforce a new select
ed71805 Changelog entry about "realdelete" option
0a275b9 Add scary warnings about "realdelete" option

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-09 10:04:05 +01:00
Sebastian Spaeth
d72bb88729 Improve error message
Add *what* UID was returned in case savemessage did not return a UID>0

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-09 09:57:36 +01:00
Sebastian Spaeth
3284e010ff Revert "use .response() rather _get_untagged_response()"
Recently the internal function use of imaplib2's _get_untagged_response()
was switched to use the public documented .response() function (which
should return the same data). However within a few fays we received reports
that both uses of a) the UIDVALIDITY fetching and b) the APPENDUID fetching
returned [None] as data although the IMAP log definitely shows that data
was returned. Revert to using the undocumented internal imaplib2 function,
that seemed to have worked without problems. This needs to be taken up to
the imaplib2 developer.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-09 09:51:43 +01:00
Sebastian Spaeth
81f194adca mbnames should write out local and not nametransformed box names
Rather than to write out the nametrans'lated folder names for mbnames,
we now write out the local untransformed box names. This is generally
what we want. This became relevant since we support nametrans rules on
the local side since only a short time. Reported by Paul Collignan.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 12:48:21 +01:00
Sebastian Spaeth
7184ec28cc Sanity check return value of UIDVALIDTY response
We have a reported case where response('UIDVALIDITY') returned [None]
which results in an ugly non-intuitive crash. Sanity check and report
something nicer.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 12:26:47 +01:00
Sebastian Spaeth
50de2174bf Allow to pass 'force' arg to selectro() to enforce a new select
Pass through the 'force' argument from selectro() to select() so that it
can also enforce a new SELECT even if we already are on that folder.

Also change the default parameter from '0' to 'False' to make clear that
this is a Bool.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 11:29:54 +01:00
Sebastian Spaeth
15b4a7a5b9 OfflineIMAP v6.5.1.2
See changelog for details

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 23:34:00 +01:00
Sebastian Spaeth
137e11c88e Improve previous comment & Changelog
Add a changelog to W. Trevor King's previous commit. Also make wording a
bit more consistent and and remove a now unneeded comparison (dirname is
always set when extension is set).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 23:26:02 +01:00
W. Trevor King
061712fe1c Folder could not be created
On Tue, Dec 13, 2011 at 12:00:57PM -0500, W. Trevor King wrote:
> I've attached a patch that does fix the problem…

Oops, *now* I've attached the patch and logs ;).

--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

From 3067b1b4dfb00d165bd9480ea49f446adb12991d Mon Sep 17 00:00:00 2001
From: W. Trevor King <wking@drexel.edu>
Date: Tue, 13 Dec 2011 11:26:00 -0500
Subject: [PATCH] Only scan children in _getfolders_scandir if extension is set.

When sep is '/', MaildirRepository._getfolders_scandir recursively
checks sub-directories for additional maildirs.  The old loop logic
always checked the top directory and its children.  This lead to
children being found twice, once from their parent, with dirname
matching their directory name, and once from themselves, with a
dirname of ''.

This patch fixes the problem by only checking the top directory when
extension is not set (i.e. for the root directory).
2012-01-07 23:21:40 +01:00
Sebastian Spaeth
6c1d80f727 Merge branch 'master' into next 2012-01-07 22:04:25 +01:00
Sebastian Spaeth
6e9e256946 Ignore lower-case custom Maildir flags
Do not read in custom maildir flags, or we would try to sync them over
the wire. The next step will be to merge flag writes with existing
custom flags, so we don't lose information.

The long term goal will be to attempt to sync flags to the other side,
of course.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 22:00:45 +01:00
Sebastian Spaeth
d549cbdac9 Fix code comment on flag stripping
Fix code comment that we preserve custom dovecot flags. We discard them.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 21:34:57 +01:00
Sebastian Spaeth
241e8f838b Add filter information to --info output
Prettify the folder listing in --info and add the "filtered" information
to it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 21:17:36 +01:00
Sebastian Spaeth
e267dcd24d Need to import OfflineImapError for --info function
An import was missing.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 20:33:34 +01:00
Sebastian Spaeth
679977e5d3 Need to import OfflineImapError for --info function
An import was missing.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 20:32:31 +01:00
Sebastian Spaeth
9cba9b8f3a OfflineIMAP v6.5.1.1
Blinkenlights UI 6.5.0 regression fixes only.

* Sleep led to crash ('abort_signal' not existing)

* Make exit via 'q' key work again cleanly

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 15:14:30 +01:00
Sebastian Spaeth
968dcd5780 Make exit via 'q' key work again in Blinkenlights UI
With the new abort signal handler, we can send a signal that lets us
exit cleanly. Make use of this, rather than crashing out in ugly ways.

This affects only the Blinkenlights UI when pressing 'q'.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 15:02:22 +01:00
Sebastian Spaeth
415c7d7979 Fix abort_sleep missing crash in Blinkenlights UI
This is a regression introduced when renaming signals due to the
improved CTRL-C handling. Regression in 6.5.0

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 14:54:16 +01:00
Sebastian Spaeth
45782ca3ac Release v 6.5.1
OfflineIMAP v6.5.1 (2012-01-07) - "Quest for stability"
=======================================================

* Fixed Maildir regression "flagmatchre" not found. (regressed in 6.5.0)

* Have console output go by default to STDOUT and not STDERR (regression
  in 6.5.0)

* Fixed MachineUI to urlencode() output lines again, rather than
  outputting multi-line items. It's ugly as hell, but it had been that
  way for years.

* Remove the old global locking system. We lock only the accounts that
  we currently sync, so you can invoke OfflineImap multiple times now as
  long as you sync different accounts. This system is compatible with
  all releases >= 6.4.0, so don't run older releases simultanous to this
  one.
2012-01-07 13:15:44 +01:00
Sebastian Spaeth
0b85a34fc7 Maildir: flagmatchre --> re_flagmatch
Overlooked a variable renaming when merging in a branch.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 13:10:41 +01:00
Sebastian Spaeth
9d6a2dec37 Have log output go to STDOUT by default
THe new logging framwork spit putput to STDERR by default (as that is
pythons default), but we used to have STDERR, so make it go there again.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 03:04:26 +01:00
Sebastian Spaeth
3a1eab7383 Remove a stray debug output that slipped in the previous commit.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 01:31:24 +01:00
Sebastian Spaeth
ec63b4fe6b replace imaplib internal _get_untagged_response with public functions
We were using the internal imaplib2 _get_untagged_response() functions a
few times. Replace 3 of these calls with 2 calls to the public function
response() rather than fudging with internals that could change anytime.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-07 01:28:20 +01:00
Sebastian Spaeth
1b85e35256 Disable old global lock system
Remove the old global locking system. We lock only the accounts that we
currently sync, so you can invoke OfflineImap multiple times now as long
as you sync different accounts. This system is compatible with all
releases >= 6.4.0, so don't run older releases simultanous to this one.

This mostly reverts commit 0d95651417,
disabling the old global lock system that we had in parallel to the new one.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 23:54:28 +01:00
Sebastian Spaeth
946386a8e7 Restore previous MachineUI output a bit more
The logging rework led to multipline output as we stopped urlencoding
the output lines. Urrg. Fixed this, so output is urlencoded again.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 23:40:31 +01:00
Sebastian Spaeth
17f60f7233 Remove from __future__ import with_statements
These were needed for python <2.6 compatability, but since we depend on
python 2.6 now, these can go.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 23:13:55 +01:00
Sebastian Spaeth
d223905068 BUG FIX: Release v6.5.0
This is a CRITICAL bug fix release for everyone who is on the 6.4.x
series. Please upgrade to avoid potential data loss! The version has
been bumped to 6.5.0, please let everyone know to stay away from 6.5.x!
I am sorry for this.

See details in the Changelog and even more gory details in commit
message for commit 8fc7227189.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 22:23:39 +01:00
Sebastian Spaeth
8fc7227189 SEVERE: Fix getting wrong UID back on IMAP upload
This change looks harmless, but it fixes a severe bugfix, potentially
leading to data loss! It fixes the "on n new uploads, it will redownload
n-1, n-2, n-3,... messages during the next syncs" condition, and this is
what happens:

If there are more than one Mails to upload to a server, we do that by
repeatedly invoking folder.IMAP.savemessage(). If the server supports
the UIDPLUS extension we query the resulting UID by doing a:

imapobj._get_untagged_response('APPENDUID', True)

and that is exactly the problem. The "True" part causes the reply to
remain in the "response stack" of the imaplib2 library. When we do
the same call on a subsequent message and the connection is still on the
same folder, we will get the same UID response back (imaplib2 only looks
for the first matching response and returns that). The only time we
clear the response stack, is when the IMAP connection SELECTS a
different folder.

This means that when we upload 10 messages, the IMAP server gives us
always the same UID (that of the first one) back. And trying to write
out 10 different messages with the same UID will confuse OfflineIMAP.

This is the reason why we saw the ongoing UPLOADING/DOWNLOADING behavior
that people reported. And this is the reason why we saw the
inconsistency in the UID mapping in the IMAP<->IMAP case.

I urge everyone to upgrade ASAP. Sorry for that, I don't know why the
problem only became prevalent in the recent few releases as this code
has been there for quite a while.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 21:57:48 +01:00
Sebastian Spaeth
0ea6c6ed47 Forgot to change a variable name in all cases
the newuid var was renamed new_uid but there was one leftover. Fixed.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 21:41:04 +01:00
Sebastian Spaeth
78a37f27ef WIP, revamp how we treat top-level dirs
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 19:08:48 +01:00
Sebastian Spaeth
ff50585007 Folder-Maildir.savemessageflags(): Only parse file name if flags changed
Rather than always parsing the filename, we only need to do so if the flags
have actually changed, otherwise we can keep the filename.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 19:08:48 +01:00
Sebastian Spaeth
de537dc09c Implement change_message_uid
Previously, assigning a new UID to a mapped IMAP or Maildir repository
was done by loading the "local" item, saving it under a new UID and
deleting the old one. This involved lots of disk activity for nothing
more than an effective file rename in Maildirs, and lots of network
usage in the MappedUID cases.

We do this on every upload from a local to a remote item, so that can
potentially be quite expensive. This patch lets backends that support it
(Maildir, MappedUID) efficiently rename the file rather than having to
read the mail content, write it out as a new file and delete the old
file. This speeds up uploads from Maildir and the MappedUID server.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 19:08:48 +01:00
Sebastian Spaeth
09ce56c594 Factor out creating a Maildir message filename
Various functions (such as change_message_uid) will want to construct
maildir filenames, so factor out the code into a helper.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 19:08:48 +01:00
Sebastian Spaeth
6fe808338c Refactor parsing out maildirs filename components
Create a helper function that retrieves the UID, folder MD5, and Flags from
a message filename.

We need these items when we simply want to rename (=new UID) a Maildir
message file later. The new function can give us these components.

Rework, so we cache the calculation of the folder's md5 value once, it
never changes and we call it a lot.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 19:08:45 +01:00
Sebastian Spaeth
de4f8c8605 Remove unneeded import in MachineUI
The urllib module is not needed anymore.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 14:06:03 +01:00
Sebastian Spaeth
d75a7a862f Fix regression of MachineUI
The recent UI overhaul led to the breakage of the machineui. This bug
fixes that.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 14:02:22 +01:00
Sebastian Spaeth
69838b81a5 Release 6.4.4
This is a bugfix release that fixes regressions since 6.4.0.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 13:22:34 +01:00
Sebastian Spaeth
d5666ce91d Don't create invalid maildir names with lower case maildir flags
If someone had a custom :2,a flag, adding a new flag would lead to the
invalid maildir filename ...a:2,... due to regex deficiencies not coping
with this. Fix this so we alway produce valid maildir names.

Note that custom flags are still problematic: as the syncing to the
remote IMAP server will fail, the next sync will assume that they have
been removed from the remote IMAP side and they will be removed from the
local Maildir then. We will need to think about how to handle this. At
least, with this patch we won't lose standard flags and won't produce
invalid maildir names.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 13:05:08 +01:00
Sebastian Spaeth
64f5e557bc Fix (harmless) regex flaw when determining maildir flags
The regex for catching Maildir message flags was
self.infosep + '.*2,([A-Z]+)' (infosep being ':').

The .* is bogus, as there is nothing between the : and the 2, per
maildir name specification, so remove that unneeded piece.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-05 14:39:09 +01:00
Sebastian Spaeth
a280a905ca Fix "getfolders() asked to get nonexisting folder" bug
When a new remote folder was detected, we tried to create the folder
locally on the Maildir and called repository.forgetfolders() to force a
new scanning of the Maildir. However, that implementation used the
inherited base function that did nothing. We simply needed to implement
forgetfolders() to set self.folder=None, so we would force a new read in
of the updated local folder structure.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-05 14:23:33 +01:00
Sebastian Spaeth
3e28073f98 Do not create folders on readonly repositories
1) Rename the unintuitive repository.syncfoldersto() to
sync_folder_structure()

2) We were checking if the local repository is readonly and then turning
off any folder creation. But as we can create folders on a remote
repository too, we need to be more fine grained here. Just don't create
a folder on the repository that is marked readonly=True.

This still does not do away with the error message that one currently
gets on missing local folders.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-05 14:05:51 +01:00
Sebastian Spaeth
6f361c4d9a Don't output "Finished in x seconds" in quiet ui
The quiet UI should only output errors, and the final "Finished account
X in 2 seconds" clearly is none, so the message debug level needed to be
reduced to INFO to suppress it in the quiet ui.

Fixes https://github.com/spaetz/offlineimap/issues/6
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-05 13:21:08 +01:00
Sebastian Spaeth
9a7c700248 Release 6.4.3
Merge in the new logging mechanism, and provide an --info feature that
will help with debugging. There is still some unstableness, so there
will be another release soon, but this should be no worse than 6.4.2 at
least...

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-04 19:34:13 +01:00
Sebastian Spaeth
0ccf06d5e6 Implement clean CTRL-C termination
Previously, we would simply bail out in an ugly way, potentially leaving
temporary files around etc, or while writing status files. Hand SIGINT
and SIGTERM as an event to the Account class, and make that bail out
cleanly at predefined points. Stopping on ctrl-c can take a few seconds
(it will e.g. finish to transfer the ongoing message), but it will shut
down cleanly.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-04 19:31:27 +01:00
Sebastian Spaeth
8ec6980c96 Don't fail on empty LocalStatus cache files
As reported in https://github.com/spaetz/offlineimap/pull/2, we would
fail when files are empty because file.read() would throw attribute
errors.

Fix this by removing the superfluous read() check and additionally log
some warning message.

Reported-by: Ralf Schmitt
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-12-01 23:57:54 +01:00
Sebastian Spaeth
50e78a8d41 Release v6.4.2
This is a bugfix release over 6.4.1. Upgrading is recommended.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-12-01 17:14:52 +01:00
Sebastian Spaeth
9a654968fc Don't append trailing slash to maildir foldernames
When sep='/' in a Maildir, we were doing a os.path.join(dirname,'') on
the top level maildir, which results in a "dirname/", so all our maildir
folder names had slashes appended. Which is pretty much wrong, so this
fixes it by only using os.path.join when we actually have something to
append.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-12-01 17:01:43 +01:00
Sebastian Spaeth
c93f8710a3 Init folder list early enough
We need the list of folders and the folder delimiter, but it was not
always retrieved early enough. E.g. when doing IMAP<->IMAP sync and the
local IMAP being readonly, we would bunk out with a mysterious error
message become repository.getsel() would still return None.

This commit fixes this error.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-12-01 10:31:43 +01:00
Sebastian Spaeth
bf4127c2d6 Remove unused imapserver getdelim()
imapserver.getdelim() was not used at all, so remove this function. The
folder delimiter is available via the repository.getsep() call.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-12-01 10:12:54 +01:00
Sebastian Spaeth
20f2edfcec Sanity check to notify us when we call IMAPRepository.getsep() too early
The folder delimiter is only initialized after a call to
acquireconnection(), so we must never call this function too
early. Include an assert() to make sure we get notified when we do.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-12-01 09:49:09 +01:00
Sebastian Spaeth
98e82db505 Release current master as 6.4.1
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>

Conflicts:

	Changelog.draft.rst
2011-11-17 09:00:58 +01:00
Sebastian Spaeth
a93c80292d Curses UI: Simplify text buffer handling
Rather than keeping a separate queue of all logged lines in memory, we
rely on the curses window scrolling functionality to scroll lines. On
resizing the terminal this means, we'll clear the screen and start
filling it afresh, but that should be acceptable.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-08 13:51:36 +01:00
Sebastian Spaeth
f8d5f1890c Make sleep abort request working again for Curses UI
1) Rework the sleep abort request to set the skipsleep configuration
   setting that the sleep() code checks.
2) Only output 15 rather than 50 debug messages on abort...

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-03 14:21:25 +01:00
Sebastian Spaeth
f4a32bafd6 Pass ui.registerthread an Account() and not a name as string
This way, we can use all the account functions such as set_abort_event()
from the ui if needed.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-03 13:45:44 +01:00
Sebastian Spaeth
ab184d84e2 Reduce parameter list to account.syncfolder call
The remote|local|statusrepo is an anttribute of each SyncableAccount()
anyway, so we don't need to pass it in, we can simply get it from the
Account().

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-03 13:29:43 +01:00
Sebastian Spaeth
ef28d5dac0 More Blinkenlights UI cleanup
Rename some variables, simplify the hotkeys treatment. Refresh/exit
signals still don't work as of yet, but will come.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-03 12:25:55 +01:00
Sebastian Spaeth
70125d58e6 Curses UI: make resize behave better
Resizing a Blinkenlights terminal doesn't crash anymore, and actually
seems to be changing the size, with this patch.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 17:03:40 +01:00
Sebastian Spaeth
212e50eb4b Fix sleeping in the Blinkenlights UI
We were still referring to s.gettf() in sleeping(self, ...) causing each
attempt to sleep to crash. Fix this, and the CursesAccountFrame.sleeping()
method. I am sure, there is still wrong and broken but we are getting there.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 12:48:22 +01:00
Sebastian Spaeth
8dbf62cfdb Add a TODO comment
This function can IMHO lead to possible deadlocks when waiting for the
connectionlock. Do add a comment to that regard, this will need to audit.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 11:56:04 +01:00
Sebastian Spaeth
74d580bc68 Exit "infinite" monitorloop when SyncRunner thread exits
The huge UI rework patch removed some obscure logic and special handling of
thread exit messages. It turns out that this was in fact still needed as a
specific exit message of the SyncRunner thread signified the threatmonitor
to quit.

We will want a nicer machinery for this in the future I guess, but fix the
eternal hang on exit by reintroducing a special exit message for the
SyncRunner thread, and return from the infinite monitor loop if SyncRunner
finishes.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 11:55:05 +01:00
Sebastian Spaeth
52ca66f055 Reduce log verbosity while scanning Maildir
no need to be that verbose here...

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 11:37:29 +01:00
Sebastian Spaeth
ccfa747ce6 Use lock with 'with' statement
To make sure, the lock gets released even if we raise an exception between
acquire() and release()

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 11:30:16 +01:00
Sebastian Spaeth
d54859a931 Don't setDaemon explicitly, it's done inherently
All ExitNotifyThreads and InstanceLimitThreads are setDaemon(True) in their
constructor, so there is no need to do that again in the code.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 11:29:23 +01:00
Sebastian Spaeth
4eeb88dd8f Append, don't truncate, log file
The new logging handler was truncating the log file on each start.
Append by default.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 10:56:42 +01:00
Sebastian Spaeth
54117e702d Bump bundled imaplib2 to 2.29
It has fixed some bugs, so update to current upstream.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 10:55:42 +01:00
Sebastian Spaeth
656405616d Drop connection if it might be bad on APPEND
1) Differentiate error messages between imaplib.abort and imaplib.error
exceptions in the log.

2) Drop connections in the case of imapobj.error, it also might denote a
   broken connection.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 10:36:30 +01:00
Sebastian Spaeth
006f19aba9 Make releaseconnection a NOOP when conn is None
During cleanup we often call releaseconnection in a finally: block. But
in cases of error, we might have dropped the connection earlier already
and set it to "None". In this case don't fail releaseconnection() but
make it a NOOP.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 10:36:30 +01:00
Sebastian Spaeth
33b4a16dac Fix mbox.select(foldername) readonly parameter comparison
The default parameter value was "None", and we were comparing that
directly to the imaplib2 value of is_readonly which is False or True, so
the comparison always returned "False".

Fix this by setting the default parameter to "False" and not
"None". Also convert all users of that function to use False/True.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 08:57:01 +01:00
Sebastian Spaeth
7c45e05428 Fix getExitStackTrace call
It was changed
2011-10-27 17:45:00 +02:00
Sebastian Spaeth
d992c66156 Rework the whole unused get/setExitCause machinery
It is basically unused by now. Rework to be able to make use of it
later, no functional changes.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-10-27 17:23:43 +02:00
Sebastian Spaeth
8195d1410f Prettify Blinkenlights.sleep()
Prettify the function.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-10-27 16:58:44 +02:00
Sebastian Spaeth
f6f8fc8528 Simplify the keepalive code a bit
Should not change behavior.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-10-27 16:56:18 +02:00
Sebastian Spaeth
cbec8bb5b2 Rework UI system to make use of the logging module
Logging was flawed as the output was e.g. heavily buffered and people
complained about missing log entries. Fix this by making use of the
standard logging facilities that offlineimap offers.

This is one big ugly patch that does many things. It fixes the
Blinkenlights backend to work again with the logging facilities.

Resize windows and hotkeys are still not handled absolut correctly, this
is left for future fixing. THe rest of the backends should be working fine.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-10-27 16:23:55 +02:00
Sebastian Spaeth
097b7ab2b0 createfolder: Fix wrong error output
If folder creation failed, we would output the wrong repository and
folder name (copy'n paste error). Fix this so we actually output the
correct values.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-10-05 18:42:47 +02:00
Sebastian Spaeth
b0fd6c6ab8 Do not import threading.*
Only import the lock, that we actually need. Also import the with statement
for use with python 2.5. We'll need it for sure in this file.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-10-05 18:34:32 +02:00
Sebastian Spaeth
76b0d7cf25 imapserver: Make noop() more resitent against dropped connections.
Drop a connection, if the NOOP to keep a connection open fails due to
broken connections.

Note that I believe this function is not working as intended. We grab
one random connection and send a NOOP. This is not enough to keep all
connections open, and if we invoke this function multiple times, we
might well always get the same connection to send a NOOP through.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-10-05 18:18:55 +02:00
Sebastian Spaeth
9578f29195 Use 'reference' value when creating an IMAP directory
A repositories 'reference value is always prefixed to the full folder
path, so we should do so when creating a new one. The code had existed
but was commented out since 2003, I guess the "reference" option is not
too often used.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-30 16:57:07 +02:00
Sebastian Spaeth
0ba9634205 Fix string formatting when port is not given.
If port is None, we would try to format an empty string with %d wich
fails. Fix it by using %s.

Reported-by: Iain Dalton <iain.dalton@gmail.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-30 10:59:07 +02:00
Sebastian Spaeth
3885acf87d Implement server diagnostics
This outputs a handy summary of your server configuration and version
strings etc, which is useful for bug reporting.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-30 09:41:45 +02:00
Sebastian Spaeth
6b3f429c81 Split offlineimap.run()
Was getting too large, split into an parse_cmd_options and a sync()
function. Moving config and ui to self.config and self.ui to make them
available through the OfflineImap instance.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-30 09:27:29 +02:00
Sebastian Spaeth
a5eebd4b6b Make "syncing folder structure" a debug level log entry
It happens quick, and clutters the log. So we can usually skip this. We
will output a log entry when we actually create a new folder anyway.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-30 09:21:03 +02:00
Sebastian Spaeth
4e89bbfe6e Recache folder list after creating a new folder
If we create a new folder we would previously not update our folder
list, which led to us skipping the synchronization of those new folders
during the initial run (subsequent runs would pick it up).

Invalidate the folder cache when we create a folder during folder
structure sync. Regetting the whole list from an IMAP server might be
slightly suboptimal from a performance point, but it is easy and will
lead to consistent results. Hopefully we will not have to create new
folders on each new run.

Reported-by: Daniel Shahaf <d.s@daniel.shahaf.name>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-30 09:05:49 +02:00
Sebastian Spaeth
ac94de3449 Only output a sleeping notice once every minute
Rather than every ten seconds, which is a bit chatty.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 17:58:07 +02:00
Sebastian Spaeth
93b05215fb Condense ui.connecting() function a bit
Just tiny non-functional code cleanup.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 17:47:14 +02:00
Sebastian Spaeth
3b647d65be Output sync timing
Modify the UI:acct and acctdone functions to keep tab of the time
inbetween. Put self.ui.acct() and acctdone() at the right places in
accounts.py so that the timing happens at the right places.

While modifying that loop, flatten the nested try: try: except: finally:
constructs, we require python 2.5 now which copes with that.

At the end of each account sync you will now see something like:
*** Finished account 'test' in 0:05

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 17:26:46 +02:00
Sebastian Spaeth
ba396cf0ef More verbose loggin
Thread names are used to determine the logging header in the TTY ui. A
recent change made them too terse (basically only changing the account
name and not the folder names). Unbreak.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 16:59:36 +02:00
Sebastian Spaeth
eb0b546927 Output progress indication when copying files
Output (2 of 500) when logging message copying. This required moving of
self.ui.copyingmessage into a different function where we actually have
the information about the progress handy.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 16:51:48 +02:00
Sebastian Spaeth
642880b404 Don't use global variable profiledir
Rather than setting a global threadutil/profiledir variable, we make
set_profiledir a class function that sets the class variable profiledir.

While touching theprofiledir code, add warning to the user if the
profile directory existed before.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 16:16:53 +02:00
Sebastian Spaeth
8970a1500b UIBase: Fix and cleanup register/unregisterthread
Registering a thread (associating it with a certain account name) would
fail if it was already registered. However, as we a) never unregister most
threads (bad) and b) single-threaded mode reuses threads, we failed when
syncing multiple accounts in single-threading mode.

This commit cleans up the functions to not make re-registering a thread
fatal (it could be legitimate, however it *should* not occur). Future
work needs to be done to unregister new threads at the appropriate places.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 15:44:30 +02:00
Sebastian Spaeth
c1120c9158 6.4.0 release
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 11:20:57 +02:00
Sebastian Spaeth
4bfc1e8227 except imapobj.abort() -> except imapobj.abort
When checking for the IMAP4.abort() exception, we need of course to
perform:

except imapobj.abort:

and not

except imapobj.abort():

Thanks to Johannes Stezenbach <js@sig21.net> for pointing to the glitch.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-27 14:08:20 +02:00
Sebastian Spaeth
ae8a1cb79f Remove deprecated calls to apply()
apply() has been deprecated since Python 2.3, and won't be working in
python 3 anymore. Use the functional equivalent throughout.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-27 13:00:12 +02:00
Sebastian Spaeth
e145beb394 Fix "command CHECK illegal in state AUTH"
Dave identified a case where our new dropped connection handling did
not work out correctly: we use the retry_left variable to signify
success (0=success if no exception occured).

However, we were decrementing the variable AFTER all the exception
checks, so if there was one due to a dropped connection, it
could well be that we 1) did not raise an exception (because we want to
retry), and 2) then DECREMENTED retry_left, which indicated "all is
well, no need to retry".

The code then continued to check() the append, which failed with the
above message (because we obtained a new connection which had not even
selected the current folder and we were still in mode AUTH). The fix is
of course, to fix our logic: Decrement retry_left first, THEN decide
whether to raise() (retry_left==0) or retry (retry_left>0) which would
then correctly attempt another loop. I am sorry for this newbie type of
logic error. The retry count loop was too hastily slipped in, it seems.

Reported-by: Dave Abrahams <dave@boostpro.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-26 16:13:52 +02:00
Sebastian Spaeth
953c58a9c9 Robustify (&fix) error throwing on APPEND
If APPEND raises abort(), the (typ, dat) variables will not be set, so
we should not be using it for the  OfflineImapError Exception
string. Fixing and prettifying the string formatting a bit at the same
time.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-26 15:57:35 +02:00
Sebastian Spaeth
1b6d76345a Robustify error msg against more failure
When syncfolder() fails, we output an error message containing the
foldername per the localfolder variable. However, the localfolder
variable is assigned inside our try: block and when the error occurs
there, we will have no localfolder variable to use for output. This
caused the errormsg to cause an Exception itself which unhelpfully
distracts from the root cause of the error.

Reconstruct the folder name in a bit more complex way, but in a way so
it is guaranteed to work (by relying on parameters passed in to the
function).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-26 15:32:24 +02:00
Sebastian Spaeth
248f23afd6 Do not fail calling dequote() with empty string
A report by Dave Abrahams showed that the dequote() function failed when
invoked with an empty string. This fixes the function to be more robust.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-26 15:16:59 +02:00
Sebastian Spaeth
e30ae53b2a Folder: Force top-level folder visiblename to ''
nametrans rules can lead to different visiblename names for the
top-level directory, specifically both '.' and '' (the latter was
recently introduced). However, we need to be able to compare folder
names to see if we need to create a new directory or whether a directory
already exists, so we need to be able to compare a repositories
visiblename (=transposed via nametrans rule) with another folder.

To make the top-level directory comparison happen, we enforce a
top-level name of '', so that comparisons work.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-23 14:31:49 +02:00
Sebastian Spaeth
275dbfa3fb Release 6.3.5-rc3
Update Changelog and __VERSION__ for 6.3.5-rc3.
Also add in the notes for rc2 to the Changelog.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-21 02:32:50 +02:00
Sebastian Spaeth
347e1eaa32 update CAPABILITIES after login
Some Webservers (I am looking at you Gmail) send different capabilities
before and after login, so they can tailor their server capabilities to
the user. While legal, this is uncommon and we were not updating our
server capabilities. Doing so allows us to detect that Gmail actually
supports the UIDPLUS extension, and we will stop mangling headers when
uploading to Gmail. This could lead to some performance gains when we
upload many messages to Gmail.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-21 02:32:45 +02:00
Sebastian Spaeth
db28cb77e7 Fix another visiblename() glitch in determining the foldername to be created
Commit b0e88622c4 changed dst_hash[folder.visiblename] to
dst_hash[folder.name] but we did not adapt all places where it is needed
to use visiblename again. This led to attempting to create a name on
REMOTE ignoring the nametrans setting on the LOCAL repository.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-20 22:25:45 +02:00
Sebastian Spaeth
1c58ebe348 Do not create folder on REMOTE if it would fall in REMOTE's folderfilter
Previously, we only checked if a LOCAL folder falls under the local
repositories folderfilter rule when deciding whether a folder should be
created on REMOTE.

However, we also do not want to create the folder on REMOTE if it would
fall under a folderfilter rule there. This patch prevents us from doing
so.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 21:25:50 +02:00
Sebastian Spaeth
b0e88622c4 Fix think on how to compare folder names on src and dest
It is not easy to think through when to use visiblenames() and whatnot. It seems I managed to not think it through properly. Which might be fixed now.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 21:06:54 +02:00
Sebastian Spaeth
ee82cd135b Fix typo atter->latter
Simply typo in user error message.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 20:07:56 +02:00
Sebastian Spaeth
48fdb14418 Don't output thread ID in log
The thread ID is not really useful and looks ugly. It also makes lines
longer than needed, there is more useful information we can put in the
log. So do away with it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 19:59:39 +02:00
Sebastian Spaeth
eeef8f4bab Don't ask for hostname if using a tunnel
In tunnel mode, we don't need to ask for a hostname because there is no
need for one. Fix this.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 15:07:34 +02:00
Sebastian Spaeth
b92e453f0e sanity check so that nametrans does not lead to infinite folder creation 2011-09-19 14:23:08 +02:00
Sebastian Spaeth
fd6261a50f Create new folders on srcrepo if needed
This will ignore any nametrans rules, so we might want to limit this
only to cases where no nametrans has been specified, or we might want to
use the nametrans setting of the dest repo.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 14:23:08 +02:00
Sebastian Spaeth
6b2ec956cf Apply nametrans to all Foldertypes
getvisiblename() was only defined on IMAP(derived) foldertypes, but we
want it on eg. Maildirs too, so we define it centrally in Folder.Base.py
rather than only in folder.IMAP.py.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 14:23:08 +02:00
Sebastian Spaeth
7d34060217 Enable folderfiltering for MailDir repositories too
Currently we only filtered IMAP repositories, this patch enables filtering
for Maildir repositories too.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 14:23:08 +02:00
Sebastian Spaeth
9c5f73d3b3 Move self.folderfilter|nametrans|sort from IMAPFolder to BaseFolder
We want to have these functions available for Maildir folders too, so we
can folderfilter a Maildir repository too (which is currently not possible)

This commit only move the corresponding functions from the IMAP to the Base
implementation. It should not change behavior in any way yet.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 14:23:08 +02:00
Sebastian Spaeth
3157a8d793 repository.Gmail.getfolder()
Also remove the removed parameters in the Gmail folder
initialization. This is one spot where I had forgotten to also strip the
parameters.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 14:22:03 +02:00
Sebastian Spaeth
a279aa7307 Maildir: Call top-level directory '', not '.'
If nametrans translates to an empty directory we want to find the
top-level directory by name '' and not by name '.'. This unbreaks
nametrans rules that result in empty folder names.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 14:14:44 +02:00
Sebastian Spaeth
792243c78f Improve repository/Maildir.getfolder() to use cached objects
Previously, getfolder() would always construct new MaildirFolder()
objects, independent of whether the folder exists or not. Improve the
function to:

1) Scan and cache the folders if not already done
2) Return the same cached object if we ask for the same foldername twice
3) Reduce a tiny bit of code duplication

This is important because we handle stuff like folderfilter in the
scandir function and if we discard the scanned dir and create a new
object on folderget(), we will lose the folderfilter information.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 13:24:42 +02:00
Sebastian Spaeth
c7420e6ad4 Debug log/error proof the creation of new IMAP folders
Output a debug log line whenever we create a new folder on an IMAP
server. Also raise an OfflineImap Error in case we failed to create it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 11:36:03 +02:00
Sebastian Spaeth
8ba17c5bd1 add sync_this variable to all Folder() instances
This variable shows if this folder should be synced or is disabled due to
a folderfilter statement. This lets us distinguish between a non-existent
folder and one that has been filtered out. Previously any filtered folder
would simply appear to be non-existing.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 11:35:52 +02:00
Sebastian Spaeth
32ca20d0da Folder.Maildir: No need to store 'uid' in messagelist dict.
The Message UID is already the key to self.messagelist, so we have that
information. It is redundant to save the UID again as
self.messagelist[uid]{'uid': uid} and we never made use of the
information anyway.

The same thing should be done with the other 2 backends.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 10:01:31 +02:00
Sebastian Spaeth
dfdc4d457b Folder.Maildir: Simplify getmessagetime
No need to use os.stat()['st_mtime'] when there is os.path.getmtime()

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:59:04 +02:00
Sebastian Spaeth
a43677a3e6 Shorten self.infosep assignment
A more pythonic and less verbose way to do the same. Add a comment what the
variable is all about.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:58:20 +02:00
Sebastian Spaeth
1ac9dc7fb4 Implement RFC 2595 LOGINDISABLED
Warn the user and abort when we attempt a plaintext login, but the
server has explicitly disabled plaintext logins.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:55:27 +02:00
Sebastian Spaeth
19ff636390 Properly output errors when the main thread receives some
Use the ui.error infrastructure that has been put in place and use
ui.terminate even if we received an Exception, so that we can output the
list of errors that we have. This does away with 2 now unused functions
in ui/UIBase.py

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:34:25 +02:00
Sebastian Spaeth
0d95651417 Perform legacy global lock in addition to new per-account lock
We simply lock OfflineImap the same global way that we have always done
in addition to the previously implemented per-account lock. We can keep
both systems in parallel and then after a few stable releases, drop the
old-style global lock. by reverting this patch

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:34:25 +02:00
Sebastian Spaeth
89c705bb26 init.py: Import OfflineImapError
The next commit will make use of OfflineImapError but is transient (the
old-style lock). The commit is supposed to be reverted after a few
releases. So add the new import in a separate commit, because we might
need this even when reverting the commit.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:34:25 +02:00
Sebastian Spaeth
c7938dc081 Per-account locking
Previously, we were simply locking offlineimap whenever it was
running. Howver there is no reason why we shouldn't be able to invoke it
in parallel, e.g. to synchronize several accounts in one offlineimap
each.

This patch implements the locking per-account, so that it is possible to
sync different accounts at the same time. If in refresh mode, we will
attempt to loop three times before giving up.

This also fixes Debian bug #586655

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:34:25 +02:00
Sebastian Spaeth
42c9e04966 Merge branch 'pending/FixGetUsername' into next 2011-09-19 09:30:05 +02:00
Sebastian Spaeth
f5366343b9 Fix default Maildir File permissions.
open() and os.open() lead to different file permissions by default, and
while we have not changed the os.open that had been used, some code
changes led to these permissions slipping through. Fix this by setting
the permissions explicitly to 0666 (minus the users umask).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:25:48 +02:00
Sebastian Spaeth
82e47896cf Don't ask for username in the preauthtunnel case
repos.getuesr() asks for a username if none is specified, but in the
case of a tunnel connection, we don't need one, so we need to skip the
repos.getuser() call here.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 08:31:14 +02:00
Sebastian Spaeth
0d3303ec12 Remove visiblename as parameter to IMAPFolder creation
IMAPFolder has the repository and foldername values so it can get the
transposed (aka visiblename) of a folder itself just fine. There is no
need to pass it in as an separate parameter.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:29 +02:00
Sebastian Spaeth
80e87d0d99 Don't pass in 'root' as para to LocalStatusFolders
They have the Repository() which contains the root, so no need to pass
it in as an extra parameter. Rename repository.LocalStatus()'s
self.directory to self.root for consistency with other backends.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:12 +02:00
Sebastian Spaeth
ee75e0921f Remove 'config' as parameter from BaseFolder & derivatives
It is possible to get the config parameter from the Repository() which is
set in BaseFolder, so we set self.config there and remove the various
methods and 'config' parameters that are superfluous.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:12 +02:00
Sebastian Spaeth
410e2d35e9 Set accountname in BaseFolder, and don't pass it in initialization
We passed in the accountname to all derivatives of BaseFolder, such as
IMAPFolder(...,repository,...,accountname), although it is perfectly
possible to get the accountname from the Repository(). So remove this
unneeded parameter. Each backend had to define getaccountname() (although
the function is hardly used and most accessed .accountname directly).

On the other hand BaseFolder was using getaccountname but it never defined
the function. So make the sane thing, remove all definitions from backends
and define accountname() once in Basefolder. It was made a property and not
just a (public) attribute, so it will show up in our developer
documentation as public API.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:12 +02:00
Sebastian Spaeth
c93cd9bb1a BaseFolder(): Save name and repository
As all Folders share these parameters, we can safely handle them in
BaseFolder. This makes sense, as BaseFolder has a getname() function
that returns self.name but nothing actually set self.name.

It also saves a few lines of code.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:54:12 +02:00
Sebastian Spaeth
7941ea7e7d folder.IMAP: Make use of the new connection discarding
In getmessage() we were releaseing a connection when we detected a
dropped connection, but it turns out that this was not enough, we need
to explicitely discard it when we detect a dropped one. So add the
drop_conn=True parameter that was recently introduced to force the
discarding of the dead conection.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:53:00 +02:00
Sebastian Spaeth
fe4f385e2c folder.IMAP: Improve dropped connection handling in quickchanged()
The quickchanged() function was not handling dropped connections yet. If
IMAP4.select() throws a FOLDER_RETRY error, we will now discard the
connection, reconnect and retry.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-16 12:53:00 +02:00
Sebastian Spaeth
5bcfbc1525 MaildirRepository: Beautify restore_atime code
Beauty of code is probably a subjective measure, but this patch hopefully
is an improvement over the previous incarnation without changing
functionality.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-15 19:28:52 +02:00
Sebastian Spaeth
c1a2b1559d repository: Simplify restore_atime code
repository.BaseRepository().restore_atime() was testing in complex ways
that it only operates on a Maildir and that the 'restoreatime' setting
is set. This is unecessary, we can simply make the base implementation a
NoOp, and move the implementation to MaildirRepository().

This will save a tad of work for everyone doing IMAP<->IMAP
synchronization and simplify the code. Also document the functions.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-15 19:28:42 +02:00
Sebastian Spaeth
d2af21d57d Simplify testing for 'subscribedonly' setting
We only explicitly tested for 'yes' when we have a nice function to get
boolean settings which also works with Treu/False/NO, etc...

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-15 19:27:01 +02:00
Sebastian Spaeth
fe57140224 Fix repository 'readonly' configuration
The readonly feature was introduced to safeguard repositories from
accidental modifications. Unfortunately, my patch treated the readonly
setting as a string and not as a boolean, so if *anything* was set in
the configuration file as 'readonly', this value evaluated to True

Fortunately this was safe, we never treated a repository that we wanted
read-only as read-write. We always treated them readonly if something
was configured as "readonly=..." even if that was False.

The fix is simply to use getconfboolean() rather than getconf() which
checks for True/False/On/Off/yes/no/1/0 and hands back the correct boolean.

Reported-by: Tomasz Nowak <nowak2000@poczta.onet.pl>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-15 19:20:27 +02:00
Sebastian Spaeth
7c83d505f8 IMAP cachefolder: Fix returning None on select
We rely on the number of mails being returned by the imapobj.select()
call, however that only happens if we "force" a real select() to occur.
Pass in the force parameter that I dropped earlier (we did not make use
of the return value when I dropped it, that is how it slipped through).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-13 18:31:28 +02:00
Sebastian Spaeth
24db42916c IMAP savemessage(): Don't loop indefinitely on failure
We were retrying indefinitely on imapobj.abort() (as that is what
imaplib2 suggests), but if the failure occurs repeatedly, we'll never
quit this loop. So implement a counter that errs out after unsuccessful
retries.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-13 18:29:22 +02:00
Nicolas Sebrecht
1a4b7c337c v6.3.5-rc1
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-12 20:10:44 +02:00
Nicolas Sebrecht
78a67ac73c Merge branch 'ss/cleanup-idle' into next 2011-09-12 20:02:26 +02:00
Sebastian Spaeth
ce8471a011 IMAP IDLE cleanup(5): Really discard connections when they are dropped
Finally, actually discard dropped connections when we detect them as an
imapobj.abort() has been thrown. In this case, invoke releaseconnection
with drop_conn=True.

We don't need the self.aborted attribute to get signified of dropped
connections. An Execption during the noop will do.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-12 19:55:14 +02:00
Sebastian Spaeth
59753fc06f IMAP IDLE cleanup(4): Simplify code
while True: if a: return

is equivalent to

   while not a:

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-12 19:55:14 +02:00
Sebastian Spaeth
6ad0de08ef IMAP IDLE cleanup(3): Rename self.event to self.stop_sig
Variable name 'event' is as bad as it gets. Rename it to something that
actually describes what it is about.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-12 19:55:14 +02:00
Sebastian Spaeth
0bebd65ba0 IMAP IDLE cleanup(2): Add code documentation
Add code documentation throughout the idle() function.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-12 19:55:14 +02:00
Sebastian Spaeth
f369961a87 IMAP IDLE cleanup(1): Move idle callback out of loop
Don't redefine the idle callback function on every run in the while
loop, define it once when we enter the function.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-12 19:55:13 +02:00
Sebastian Spaeth
bc10e05600 Check for SSL module existence and only do fingerprint check then
Python 2.5 has no ssl module, and we can therefor not get the server
certificate for fingerprint verification. Add a check that disables
fingerprint verification for python 2.5.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-12 19:45:16 +02:00
Sebastian Spaeth
eafea0c880 IMAP4.sslobj -> sock
imaplib2 has changed internally to use self.sock for its ssl socket when
it used to be sslobj. Reflect that change.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-12 19:45:16 +02:00
Sebastian Spaeth
8800fa37a3 Implement Server SSL fingerprint check
If we connect to a SSL server (not STARTTLS) and no CA cert has been
specified for verification, we check the configured SSL fingerprint and
bail out in case it has not been set yet, or it does not match.

This means one more mandatory option for SSL configuration, but it
improves security a lot.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-12 19:45:15 +02:00
Sebastian Spaeth
5cbec30b3e Sanity check for maxage setting
If maxage is set too large, we would even SEARCH for negative
years. With devastating results. So implement some sanity check and err
out in case the year does not make sense.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-07 19:32:43 +02:00
Sebastian Spaeth
135f8c45cf Simplify constructing the SEARCH date
We can use Imaplib's monthnames and shorten the construction of the date
by using them rather than hardcoding them again.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-07 19:32:08 +02:00
Sebastian Spaeth
1b99c019e5 Fix handling the search results
Results are delivered in a 1-element list, and somehow I managed to drop
a [0] in the previous patches. We need to look at the element of course,
or our string splitting will fail horribly. Sorry this somehow slipped
through.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-07 19:31:37 +02:00
Sebastian Spaeth
26721c60d4 Don't cache empty IMAP folders
If a folder is empty, most servers will return EXISTS 0 and imaplib2
passes back ['0'] as return value to a select(). It returns [None] if
no EXISTS response was given by the server at all.

Attempting to fetch the UIDs of 0 emails which leads to
various error messages (One server responds with "NO No matching
messages", Gmail seems to say "BAD Bad message sequence 1:*" for some
(although it is working fine for me with Gmail, so it might behave
different for different people).

In case we get an None or 0 back, we simply stop caching messages as the
folder is empty. This should fix the various error reports that have
popped up.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-07 19:13:13 +02:00
Nicolas Sebrecht
67863017e2 Merge branch 'ss/better-error-throwing-and-id-sequence' into next
Conflicts:
	offlineimap/folder/IMAP.py
	offlineimap/imaputil.py

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:15:05 +02:00
Sebastian Spaeth
4c558c1b69 Error proof IMAP.APPEND against dropped connections
Make sure that when a connection is dropped during append, we really
discard the broken connection and get a new one, retrying. We retry
indefinitely on the specific abort() Exception, as this is what imaplib2
suggests us to do.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
971ed3adac Allow Imapserver.releaseconnection() to drop a connection
If a connection is broken, we want to have it really dropped and not be
reused. So far, we are checking the .Terminate attribute for this, but
according to the imaplib2 author, it is only set on normal shutdown and
it is an undocumented attribute whose meaning could change any time.

This patch introduces the parameter drop_conn which allows to tell
releaseconnection() that we really want to connection being dropped from
the pool of available connections and properly destroy it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
1917be8e83 Shorten list of messages to be deleted in UI output
Rather than output the full list of messages, coalesce it into number
ranges wherever possible.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
fb8017991c Rework undocumented listjoin to create UID sequences
This function was badly named and completely undocumented. Rework it to
avoid copying the full UID list using an iterator. Make it possible to
hand it a list of UIDs as strings rather than implicitely relying on the
fact that they are numeric already. Document the code.

The behavior off the function itself remained otherwise unchanged.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
3302940382 Proper error handling for SEARCH and FETCH failures from the server
SEARCH and FETCH were never checking that the IMAP server actually
returned OK. Throw OfflineImapErrors at severity FOLDER in case one of
them fails.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
f755c8b423 Use range 1:* if we want to examine all messages in a folder
Some code cleanup. If we want to examine all messages of a folder, don't
try to find out how many there are and request a long list of all of them,
but simply request 1:*. This obliviates us from the need to force a select
even if we already had the folder selected and it requires us to send a
few less bytes over the wire.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
8532a69458 Clean up the maxsize maxage code
Do away with the wrapping of this code in a try...except KeyError, as
this code cannot conceivably throw a KeyError. Even if it could, it
should be documented why we should simply return() in this case.

Shorten some of the variable names and minor code cleanup while taking
the git blame anyway.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Sebastian Spaeth
06bfff7c04 Coalesce SEARCH uid list into sequence set
Rather than passing in huge lists of continuous numbers which eventually
overflow the maximum command line length, we coalesce number ranges
before passing the UID sequence to SEARCH. This should do away with the
error that has been reported with busy mailing lists and 'maxage'.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 20:03:33 +02:00
Nicolas Sebrecht
a21bb949af Merge branch 'ss/improve-collapsing' into next 2011-09-06 19:59:28 +02:00
Nicolas Sebrecht
8c730cf509 Merge branch 'ss/folder.imap-dont-import-copy' into next
Conflicts:
	offlineimap/folder/IMAP.py

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:57:02 +02:00
Sebastian Spaeth
63b9dcd896 folder.IMAP: Don't import copy
It is not needed. list(ALIST) will create a new copy of the list just
fine.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:53:44 +02:00
Sebastian Spaeth
2cf6155282 Error proof IMAP.APPEND against dropped connections
Make sure that when a connection is dropped during append, we really
discard the broken connection and get a new one, retrying. We retry
indefinitely on the specific abort() Exception, as this is what imaplib2
suggests us to do.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:51:49 +02:00
Sebastian Spaeth
9c678c9d6b Allow Imapserver.releaseconnection() to drop a connection
If a connection is broken, we want to have it really dropped and not be
reused. So far, we are checking the .Terminate attribute for this, but
according to the imaplib2 author, it is only set on normal shutdown and
it is an undocumented attribute whose meaning could change any time.

This patch introduces the parameter drop_conn which allows to tell
releaseconnection() that we really want to connection being dropped from
the pool of available connections and properly destroy it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:51:49 +02:00
Sebastian Spaeth
0906d0db70 IMAP.cachemessagelist(): Protect against empty folders
When invoked with FETCH 1:* (UID), imaplib returns [None] for empty
folders. We need to protect against this case and simply 'continue' here.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:07:54 +02:00
Nicolas Sebrecht
ff9b941f42 Merge branch 'ss/protect-select-for-idle' into next 2011-09-06 19:01:11 +02:00
Sebastian Spaeth
dc103ab9ce Protect IMAP.getmessage() against dropped connections
If a connection is dropped for some reason while fetching a message, the
imapobj.uid command throws an imapbj.abort() Exception which means we are
supposed to retry. Implement a fail loop that drops the connection, gets a
new one and attempts the command another time.

Remove obsolete comment that we need to catch nonexisting messages. We do
now.

GMail seems to drop connections left and right. This patch is a response to
the reported mail "4E5F8D8C.1020005@gmail.com" by zeek
<ezekiel.das@gmail.com>.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:00:48 +02:00
Sebastian Spaeth
f1555ec893 Catch terminated connections on the IDLE SELECT operation
Handle the case gracefully where a server has closed an IMAP connection
that we want to use for IDLEing. Simply have it dropped and get a new one
in this case. THis should get rid of the errors reported by John Wiegley
in mail id:"m2sjohd16t.fsf@gmail.com".

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:00:21 +02:00
Sebastian Spaeth
61fffd5c5f imaplibutil: Have SELECT throw an OfflineImapError when the connection died
Imapserver.acquireconnection will succeed even whent the server connection
has been terminated and the first IMAP operation will throw an exception.

Often this is the folder SELECT operation (e.g. after an idle timeout), as
has been reported by John Wiegley. Catch this case and throw an
OfflineImapError with severity FOLDER_RETRY to notify consumers that they
are supposed to retry.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:00:21 +02:00
Sebastian Spaeth
7bc45507cb Introduce new error level FOLDER_RETRY
In cases where processing a folder failed, but a retry might well succeed
e.g. when the server connection had simply timed out and was disconnected,
we can throw a FOLDER_RETRY (which is less severe than FOLDER).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-06 19:00:21 +02:00
Sebastian Spaeth
fe1391084b Improve repo.Base.py:syncfoldersto parameter/var names
Name parameter that hands us a Status Repository 'status_repo' and not
'copyfolders' as was before:

a) make it clear that we pass in a repository and not folder
instances. That was very confusing before.
b) We were always only using one 'copyfolders' item anyway, so let us
not make it a list.

Go through the list and make the variable nameing consistent:
dst_repo rather than dest (is it a folder?) to match the status_repo
naming scheme.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-03 17:16:42 +02:00
Sebastian Spaeth
54d3307499 imaputil: make uid_sequence sort all items to improve collapsing
To preserve previous behavior and save a few CPU cycles, we were not sorting UID
lists and only collapsed them if they were alreay sorted.  Vincent pointed out
that this is not always the case and unsorted lists lead to non-optimally
collapsing.

Force lists to numeric types and sort them before collapsing.

Reported-by: Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-02 22:02:51 +02:00
Sebastian Spaeth
f0869e5c0f Fix sqlite upgrade code to use correct API call
We were using self.getfolderbasename(self.name) but the API is simply
getfolderbasename(). Fix this glitch.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-02 21:59:35 +02:00
Sebastian Spaeth
8cce178fb8 Improve developer documentation
Well, this patch... improves it. :)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-30 22:19:16 +02:00
Sebastian Spaeth
4db5913492 Use SafeConfigParser for the configuration
SafeConfigParser is very similar to the currently used ConfigParser but
it supports interpolation. This means values can contain format strings
which refer to other values in the same section, or values in a special
DEFAULT section. For example:

[My Section]
foodir: %(dir)s/whatever
dir=frob

would resolve the %(dir)s to the value of dir (frob in this case). All reference expansions are done on demand.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-30 22:16:35 +02:00
Sebastian Spaeth
dcc50efc4f Don't call set().sort()
as that method doesn't exist on sets. Rather call the inbuilt
sorted(flags). This fixes Exceptions being thrown when using the sqlite
backend.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:59:48 +02:00
Sebastian Spaeth
a6480d4959 Fix string formatting
We were omitting an '%' where we needed it. Also include the traceback
information where it belongs in the new ui.error infrastructure.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:58:55 +02:00
Sebastian Spaeth
2b9b6be6be Fix standard port for SSL/TLS
443 is of course the https and not the IMAPS standard port. Fix.

Thanks to Daniel Shahaf <d.s@daniel.shahaf.name> for the heads up.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:57:38 +02:00
Vladimir Marek
9ecac29aaa Maildir relative paths change was not complete
Commit e023f190b0 changed the storing of
file paths in the messagelist variable to be relative paths, but we were
using the full absolute path anyway as we missed one spot.

Adapt this and construct the full file path in the one place where we
need it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:56:24 +02:00
Sebastian Spaeth
6295d64d54 Shorten list of messages to be deleted in UI output
Rather than output the full list of messages, coalesce it into number
ranges wherever possible.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:55:28 +02:00
Sebastian Spaeth
0a25408199 Rework undocumented listjoin to create UID sequences
This function was badly named and completely undocumented. Rework it to
avoid copying the full UID list using an iterator. Make it possible to
hand it a list of UIDs as strings rather than implicitely relying on the
fact that they are numeric already. Document the code.

The behavior off the function itself remained otherwise unchanged.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-23 20:55:28 +02:00
Vladimir Marek
8de326b29f Catch correct type of exception
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 19:05:16 +02:00
Vladimir Marek
df62bb61a5 Create exception when file rename fails
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 19:02:47 +02:00
Sebastian Spaeth
466ded04d9 Make flags a set rather than a list
As this is essentially what it is, a set of values. This allows as
to do set arithmetics to see, e.g. the intersection of 2 flag sets
rather than clunkily having to do:

for flag in newflags:
  if flag not in oldflags:
    oldflags.append(flag)

Also some more code documenting.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 18:56:10 +02:00
Sebastian Spaeth
373e7cdbc1 Allow empty foldernames
Empty foldernames (as they could be created through nametrans) were
failing as the uidvalidity and status files names as determined by
folder/Base.py:getfolderbasename() lead to invalid file names ''.

Fix this by handling empty file names and translating them to '.' which
leads to the special file name 'dot'. (this special value existed before
and was not invented by this patch)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 18:37:00 +02:00
Sebastian Spaeth
4f1cd05fdc Fix copyingmessage() in Machine UI
I changed the API to pass in a folder rather than a list of folders, but
used getnicename() on the wrong object. It is not used on the folder but
on the ui object. Fix this and give the variable somewhat better names.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-19 18:25:04 +02:00
Vladimir Marek
c2fc81dd3d Make syncfoldersto to accept a single folder only
It is just historic relict

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-17 21:02:21 +02:00
Vladimir Marek
b6ac1aecb1 Another way of locating UID of just saved message
It works by fetching all headers of new messages from IMAP server and
searching for our X-OfflineIMAP marker by using regular expression.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-17 20:58:41 +02:00
Sebastian Spaeth
6f9b171ffd Don't pass a list to ui.copyingmessage()
We only copy to a single folder anyway, so clean up the code to only
pass in a single folder.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-17 20:41:32 +02:00
Vladimir Marek
38b1d7b085 Replaced tabs with spaces to unify python sources
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-16 22:33:19 +02:00
Sebastian Spaeth
8d858a0b78 Disable certificate verification if there is no certificate
imaplib2 always attempts to verify a certificate if a verification
callback function is passed in, even the certificate is None
specified. Disable the verification excplictly by setting the
verification function to None in that case.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-16 22:22:47 +02:00
Nicolas Sebrecht
82bf23ee57 Merge branch 'ss/import-errno' into next
Conflicts:
	offlineimap/imapserver.py
2011-08-15 14:46:52 +02:00
Sebastian Spaeth
4a0d17b566 Import errno
we need errno.CONNREFUSED, but through some merging mishaps(?) the part
that actually imported errno was missing. Import the errno module.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 13:58:44 +02:00
Nicolas Sebrecht
2749374a93 Merge branch 'ss/imaplib2-2.28' into next
Conflicts:
	Changelog.draft.rst
2011-08-15 13:57:37 +02:00
Sebastian Spaeth
3a91e296f0 Adapt the code to work with the new imaplib2
imaplib renamed self.sslobj to self.sock and our overriden open()
functions were failing for that reason when updating imaplib2 to
v2.28. It turns out that all of our custom initializations are being
done by stock imaplib2 now anyway, so there is no need to override them
anymore. This lets us simplify the code we have to worry about.

Move the verifycert() function to the imapserver.py file, it is now a
callback function that is being handed to imaplib from there, so it
makes sense to also define it in our imapserver function...
(this also lets us easily make use of the verifycert function in the
starttls case in the future)

TODO: we need to examine if and why we still need to override the
select() function, it is the only reason why we still wrap the IMAP4
classes.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:21:38 +02:00
Sebastian Spaeth
1c0c19ad86 imaplib2: bump to upstream version 2.28 (rev 8228a0f)
Imaplib2 2.28 can deal with ID sequences, such as 1:*, so we need to
bump upstream in order to make use of these features.

Note that this revision will not run correctly as it requires
adaptations to our code, which happens in the next commit.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:21:38 +02:00
Sebastian Spaeth
fe388400c4 Better error message when FETCH fails.
We were not including the full server reply into our error message. Fix
that so we get better error logs.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:16:29 +02:00
Sebastian Spaeth
9f23fea74e Fix error handling in folder.Base.copymessageto()
This is a bug fix on several levels. 1) We were lacking the import of
OfflineImapError. 2) OfflineImap.ERROR.MESSAGE was misspelled as ERROR.Message.
3) COntinuing with the next message only worked in single-thread mode
(using debug) and not in multi-thread mode. The reason is that we were
invoking a new thread and catching Exceptions in the main thread. But
python immediately aborts if an Exception bubbles up to a thread start.
This was fixed by catching exceptions directly in copymessageto() which
is the new thread, rather than catching them in the main thread.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:16:28 +02:00
Sebastian Spaeth
634b6cd49e Don't make Blinkenlight statuschar configurable
Allowing to specify the char to use in the BLinkenlights is a bit over
the top and bloats our default offlineimap.conf. The dot is just fine,
so let us settle for it and cut the example config file by an unneeded
section.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:10:20 +02:00
Sebastian Spaeth
194aa1db3c Catch 'BAD' replies on append()ing a message
append() raises an Exception, in case the IMAP server replies with 'BAD'
(but not when it responds with 'NO') but we were not catching that. Do
catch the situation and also raise an OfflineImapError at MESSAGE
severity, so that we can continue with the next message.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-15 12:05:08 +02:00
Nicolas Sebrecht
b896e6f13b Merge branch 'ss/remove-processmessagesflags-function' into next 2011-08-14 17:23:49 +02:00
Nicolas Sebrecht
21b10bbf55 Merge branch 'ss/remove-mustquote' into next 2011-08-14 17:23:44 +02:00
Sebastian Spaeth
51629b8ffe Update example in code documentation of ui.error()
sys.exc_traceback is long deprecated and is seems removed in python2.7,
so document the legitimate use of sys.exc_info()[2] instead.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-14 17:18:51 +02:00
Sebastian Spaeth
3c89a72be9 remove unused 'mustquote' regex
We set an imapobj.mustquote which apparently was used in previous
incarnations of imaplib or imaplib2, however, nothing in our codebase
makes use of that. So let us remove it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-14 17:15:38 +02:00
Sebastian Spaeth
306f584c86 Remove custom Gmail/folder/processmessagesflags()
This function was overridden as the IMAP version apparently had been
using imapobj.myrights() at some point in time, which was not
implemented in the Gmail version. However, IMAP is not using myrights()
anymore, and as that is an extension that needs to be advertised in
CAPABILITIES we should not unconditionally use it anyway.

So remove the function that is identical to it's ancestor's
function.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-14 17:14:35 +02:00
Sebastian Spaeth
97dbd18e12 LocalStatus.py: Fix getfolders()
This code was unused and broken. It is still unused but this commit
fixes it. (We should retain the method in case we ever start calling
getfolders() on LocalStatus.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-12 18:31:31 +02:00
Nicolas Sebrecht
a38c6da49d Merge branch 'ss/improved-folderfilter-errors' into next
Conflicts:
	Changelog.draft.rst
2011-08-12 18:29:23 +02:00
Sebastian Spaeth
fa18968642 Catch folderfilter errors in repository.IMAP.getfolders()
Rather than throwing ValueError, we now properly throw OfflineImapError
when selecting a folder in folderincludes. So we also need to catch
OfflineImapErrors here. If they are of severity FOLDER, just ignore the
invalid folder and continue. If the error is more severe, bubble it up.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-12 18:28:49 +02:00
Sebastian Spaeth
131298c2b1 Remove superfluous class ConfigedIMAPServer
Remove a level of wrapper abstraction that is not needed. Just use
IMAPserver and be done with it.

We do this by passing in the IMAPRepository() instance rather than a
long list of single paramters to the IMAPServer instanciation. This way
we can retrieve all repository parameters ourselves, rather than passing
a dozen paramters into IMAPServer. Also, this enables us to pass the
repository() object into our WrappedIMAP4() instance, so that it can
query, e.g. the SSL fingerprint configuration.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-12 18:18:26 +02:00
Sebastian Spaeth
f6b9c68333 IMAP: Don't use assert() in folder.savemessage()
We simply assert()ed that APPENDing a message returned OK, but in some
cases (e.g. Google chat messages) APPEND might return BAD or NO too. We
should be throwing an OfflineImapError here at MESSAGE level, so that we
can continue to sync all other messages, and still give the user some
details on what went wrong at the end of the sync run.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-11 19:14:28 +02:00
Sebastian Spaeth
b47bc44783 accounts.py: Use ui.error when raising exceptions
Rather than using ui.warn, use ui.error() which outputs Exceptions to
the error log, saving them to a stack, so we get notified again at the
end of the sync run.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-11 19:14:28 +02:00
Sebastian Spaeth
f937a86571 Implement ui.error() and output them at end of offlinimap sync
Output all raised Exceptions error strings to the error log. If we are
in debug mode, we also output the traceback of the exception.

Save all exceptions that occur during the run time to a Queue and output
them all again when the offlineimap sync is finished.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-11 19:14:28 +02:00
Nicolas Sebrecht
61e50b3b1a v6.3.4
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-10 21:44:58 +02:00
Vladimir Marek
e58399ac0b Fixed typo
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-27 18:54:53 +02:00
Vladimir Marek
d5cbdc4c0e Handle when UID can't be found on saved messages
Message was stored to dstfolder, but we can't find it's UID. This means we can't
link current message to the one created in IMAP. So we just delete local message
and on next run we'll sync it back. Also fixed imap.savemessage description.

This was broken by e20d8b9679.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-27 18:54:53 +02:00
Vladimir Marek
86e9c7442b Include message header at better place
It's not enough to place header after first newline, since this might break
multiline rfc0822 folded long header lines. Those are difined as CRLF followed
by white space. Instead we'll search for two successive CRLF sequences which
mark end of mail headers and place our header just before that.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-27 18:54:52 +02:00
Nicolas Sebrecht
86cfd31ce9 v6.3.4-rc4
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-27 18:53:46 +02:00
Vladimir Marek
6995eeb92e Support maildir for windows
That makes OfflineIMAP to use exclamation mark (!) instead of colon for storing
messages. Such files can be written to windows partitions. But you will probably
loose compatibility with other programs trying to read the same Maildir.

Signed-off-by: Vladimir Marek <vladimir.marek@oracle.com>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-12 21:53:33 +02:00
Nicolas Sebrecht
129b703bf2 v6.3.4-rc3
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-07 18:43:18 +02:00
Sebastian Spaeth
c293e64119 Fix caching inconsistencies in getselectedfolder()
getselectedfolder was using a cached variable that we were setting in
select(), but sometimes the IMAP4 instance got into the SELECTED state
without explicitely select()ing, it seems, and our variable was unset.

Let us just use the self.mailbox variable that imaplib2 is setting when
select()ing rather than doing our own caching. Also remove the part
where we were setting the cache.

Just access self.state rather than looking up self.state via
self.getstate() every time, it is just an unnecessary layer of
redirection.

Original-patch-by: Arnaud Fontaine <arnau@debian.org>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-07 18:33:45 +02:00
Arnaud Fontaine
09ba269576 Add missing import of SSLError exception.
In commit 89cbdc9, usage of SSLError was dropped but later reintroduced
without importing SSLError exception.

Signed-off-by: Arnaud Fontaine <arnau@debian.org>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-07 18:31:38 +02:00
Sebastian Spaeth
aec35eebd3 Reduce initial license blurb
Rather than the extremly verbose NO WARRANTY blurb, we output a somewhat
smaller initial text which should still make the GPL happy.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-07-04 18:08:51 +02:00
Sebastian Spaeth
8e2589982c Don't use CStringIO to format a traceback
The traceback module has format_exc() for this purpose so let's use that.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-30 21:28:06 +02:00
Sebastian Spaeth
928c363044 imapserver.py: Make severity var available where it is needed
We we using the variable 'severity' in a few places to throw
OfflineImapErrorrs of severity REPO. Somehow, that variable is now not
accessible in all places that refer to it, so we move where it is
defined to before all the 'if' checks which might make use of it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-27 18:43:45 +02:00
Haojun Bao
9d95d7bc62 folder/IMAP: fix typo with maxsize and maxage.
Signed-off-by: Bao Haojun <baohaojun@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-27 18:26:43 +02:00
Sebastian Spaeth
6311716edb imapserver.py: Implement STARTTLS
If we do not use a SSL connection anyway and if the server supports it,
authenticate automatically with STARTTLS.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-25 17:09:52 +02:00
Sebastian Spaeth
416df0fc44 imaputil.imapsplit: Remove overzealous debug statement
In an IMAP run where we did not have to sync anything, I spend nearly a
fulls second in imaputil.debug() without even having debug output
enabled. imapsplit is mainly called by flagsplit() which will also do
debug output, so we get TONS of nearly duplicate debug output in the log
which makes it really hard to analyze.

Cut down the debug logging in imapsplit, we should be debug logging
stuff at a slightly higher level than here anyway.

This one-line change sped up my folder sync (without having to sync
anything) by 0.5 seconds even when debugging is disabled.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-25 17:06:53 +02:00
Sebastian Spaeth
3ce514e92b Simplify MappedIMAPFolder, fixing bugs
Previously, we instanciated an MappedImapFolder, and would cleverly (too
cleverly?) invoke methods on it casting it to an IMAPFolder by calling
methods such as: self._mb.cachemessages(self) where self._MB is the class
IMAPFolder and self and instance of MappedImapFolder. If
e.g. cachemessages() invokes a method uidexists() which exists for
MappedImapFolder, but not directly in IMAPFolder, I am not sure if
Python would at some point attempt to use the method of the wrong class.
Also, this leads to some twisted thinking as our class would in same
cases act as an IMAPFolder and in some cases as an MappedImapFOlder and
it is not always clear if we mean REMOTE UID or LOCAL UID.

This commit simplifies the class, by a)doing away with the complex Mixin
construct and directly inheriting from IMAPFOlder (so we get all the
IMAPFOlder methods that we can inherit). We instantiate self._mb as a
new instance of IMAPFolder which represents the local IMAP using local
UIDs, separating the MappedIMAPFolder construct logically from the
IMAPFolder somewhat.

In the long run, I would like to remove self._mb completely and simply
override any method that needs overriding, but let us take small and
understandable baby steps here.

Reported-and-tested-by: Vincent Beffara <vbeffara@gmail.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-25 17:03:22 +02:00
Sebastian Spaeth
b40d02e801 repository/Maildir.py: Fix typo 'sudir'
sudir->subdir in a debug statement. Thanks ccxCZ on IRC for the heads
up.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-23 18:30:55 +02:00
Nicolas Sebrecht
994a134133 Merge branch 'ss/handle-deleted-uid-while-syncing' into next
Conflicts:
	offlineimap/imapserver.py
2011-06-17 23:43:47 +02:00
Sebastian Spaeth
7570f71880 Throw OfflineImapError when we try to request an inexistant message
During a sync run, someone might remove or move IMAP messages. As we
only cache the list of UIDs in the beginning, we might be requesting
UIDs that don't exist anymore. Protect folder.IMAP.getmessage() against
the response that we get when we ask for unknown UIDs.

Also, if the server responds with anything else than "OK", (eg. Gmail
seems to be saying frequently ['NO', 'Dave I can't let you do that now']
:-) so we should also be throwing OfflineImapErrors here rather than
AssertionErrors.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-17 23:41:31 +02:00
Nicolas Sebrecht
aba13912c4 Merge branch 'ss/init-always-call-ui-terminate' into next 2011-06-16 23:42:43 +02:00
Nicolas Sebrecht
89cbdc9244 Revert "Throw errors on connection refused and on non-standard SSL ports"
This reverts commit 3dc9fc519a.
2011-06-16 23:41:36 +02:00
Sebastian Spaeth
8634b0030d Always call ui.terminate()
I discovered that we do not run ui.terminate in all circumstances, so
make sure that we call with properly at the end of each run (whether in
threaded or single-thread mode).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-16 18:57:35 +02:00
Sebastian Spaeth
335b320d9a Fix recursively scanning Maildir folders
Commit 1754bf4110 introduced a blunder:

-        for dirname in os.listdir(toppath) + ['.']:
+        for dirname in os.listdir(toppath) + [toppath]:
...
-            if self.getsep() == '/' and dirname != '.':
+            if self.getsep() == '/' and dirname:

This change was plainly wrong and would never have worked, so this
commit reverts above bit. While touching the function, some minor code
documentation, cleanup and limiting line length to 80 chars.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-16 18:37:40 +02:00
Sebastian Spaeth
2180f5fbf4 UIDMappedFolder.savemessage() returned nothing, breaking API conventions
Folder.savemessage() is supposed to return the new UID that a backend
assigned, and it BaseFolder.copymessageto() fails if we don't return a
non-negative number in the savemessage() there.

For some reason, the UIDMappedFolder was not returning anything in
savemessage, despite clearly stating in the code docs that it is
supposed to return a UID. Not sure how long this has already been the
case. This patch fixes the UIDMappedFolder to behave as it should.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-16 18:34:05 +02:00
Sebastian Spaeth
856982a4e6 Throw OfflineImapError when we try to request an inexistant message
During a sync run, someone might remove or move IMAP messages. As we
only cache the list of UIDs in the beginning, we might be requesting
UIDs that don't exist anymore. Protect folder.IMAP.getmessage() against
the response that we get when we ask for unknown UIDs.

Also, if the server responds with anything else than "OK", (eg. Gmail
seems to be saying frequently ['NO', 'Dave I can't let you do that now']
:-) so we should also be throwing OfflineImapErrors here rather than
AssertionErrors.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-15 22:17:04 +02:00
Scott Henson
41fad17125 Fix gssapi with multiple connections
Fix a gssapi issue where threads beyond the first would not
 be able to authenticate against the imap server. This is
 done by using the connection lock around the gssapi
 authentication code and resetting (and releasing) the
 kerberos state after success so that subsequent connections
 may make use of kerberos.

Signed-off-by: Scott Henson <sjh@foolishpride.org>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-15 18:29:57 +02:00
Nicolas Sebrecht
5328f10b39 v6.3.4-rc2
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-15 18:15:11 +02:00
Sebastian Spaeth
d122dde93f Check SSL certificate for expiration
We currently don't care about expiration dates of the servers SSL
certificate. This patch adds a check that fails Cert verification when
it is past its due date. There is no way or option to override this
check.

Unfortunately we only seem to be able to get SSL certificate data when
we passed in a CA cert file? How do we get that date when we don't have
a ca cert file?

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-13 16:47:03 +02:00
Sebastian Spaeth
d8026c5308 Simplify Maildir message saving
Previously we were attempting to save out mails according to
 http://www.qmail.org/man/man5/maildir.html in 4 steps:

1 Create a unique filename
2 Do stat(tmp/<filename>). If it found a file, wait 2 sec and go back to 1.
3 Create and write the message to the tmp/<filename>.
4 Link from tmp/* to new/*

(we did step 2 up to 15 times) But as stated by
http://wiki1.dovecot.org/MailboxFormat/Maildir (see section 'Issues with
the specification'), this is a pointless approach, e.g. there are race
issues between stating that the filename does not exist and the actual
moving (when it might exist).

So, we can simplify the steps as suggested in the dovecot wiki and
tighten up our safety at the same time.

One improvement that we do is to open the file, guaranteeing that it did
not exist before in an atomic manner, thus our simplified approach is
really more secure than what we had before.

Also, we throw an OfflineImapError at MESSAGE level when the supposedly
unique filename already exists, so that we can skip this message and
still continue with other messages.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-13 16:29:00 +02:00
Sebastian Spaeth
e023f190b0 folder/Maildir: Store only relative filename components
MaildirFolder.messagelist[*]['filename'] was storing the absolute file
paths for all stored emails. While this is convenient, it wastes much
space, as the folder prefix is always the same and it is known to the
MaildirFolder. Just 40 chars in a folder with 100k mails waste >4MB of
space. Adapt the few locations where we need the full path to construct
it dynamically.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-12 17:41:13 +02:00
Sebastian Spaeth
c1c200a487 folder/Maildir: cache getfullname() value
We use getfullname() very often (thousands to millions), yet we
dynamically calculate the very same value over and over. Optimize this
by caching the value once and be done with it.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-12 17:40:11 +02:00
Nicolas Sebrecht
4c8abb0bc6 Merge branch 'ss/imaplib2-v2.24' into next 2011-06-09 18:28:23 +02:00
Sebastian Spaeth
520e39d355 Update imaplib2 to 2.24
In some cases we had offlineimap trying to delete emails that shouldn't
be deleted. E.g. https://bugzilla.redhat.com/show_bug.cgi?id=708898.

It turns out that imaplib2 does not like FETCH responses that are
interrupted by other unsolicited server responses, e.g.

* OK Searched 43% of the mailbox, ETA 0:12\r\n

Bump imaplib2 to a version that can cope with these (legal) responses by
the IMAP server.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-09 18:26:39 +02:00
Sebastian Spaeth
e8b633b884 folder/IMAP: Remove buggy duplicate assignment
we do:
  for msgid in imapdata:
      maxmsgid = max(long(msgid), maxmsgid)
and then basically immediately:
   maxmsgid = long(imapdata[0])

throwing away the first assignment although the first method of
assigning is the correct one. The second had been forgotten to be
removed when we introduced the above iteration. This bug would fix a
regression with those broken ZIMBRA servers that send multiple EXISTS
replies.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:52:24 +02:00
Sebastian Spaeth
846070b240 Fix typos in months names
All months names are 3-letter abbreviated, but accidentally June and
July slipped through. Thanks to the heads up by
Philipp Kern <pkern@debian.org>.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:20:25 +02:00
Sebastian Spaeth
b20f5192fb Allow to specify remote hostname even for the Gmail case
Previously we hard-coded the imap server name in the case of Gmail
repositories, but often we need a different host name. So, allow people
to specify the hostname via the regular "remotehosteval" and
"remotehost" settings, and only falling back to imap.gmail.com when
nothing has been specified.

Cache the hostname, so we don't evaluate the whole thing each time we
query the host name.

Make the remotehosteval processing more robust, by catching any
Exceptions that occur, and throw a OfflineImapError, that explains where
exactly the error had occured. You can test this, e.g. by setting
remotehosteval to 1/"n" or some other invalid expression.

The whole IMAP.gethost() function has been documented code wise while
going through.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:19:07 +02:00
Sebastian Spaeth
1754bf4110 Allow to create the root MailDir directory
We currently do not allow nametrans rules such as
nametrans = lambda foldername: re.sub('^INBOX$', '', foldername)

because we crash with a traceback when running:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=499755

The underlying reason is that we cannot create the "top level" root
directory of the Maildir in the function makefolders(), it will bail
out. John Goerzen intentionally prevented offlineimap from creating the
top-level dir, so that a misconfiguration could not arbitrarily create
folders on the file system. I believe that it should be perfectly
possible to automatically create the root dirctory of the maildir. We
still protect against folder creations at arbitrary places in the file
system though.

This patch cleans up makefolders(), adds documentation, allows to
automatically create rootfolders if needed (using absolute paths) and
adds some robustness in case the folders already exist that we want to
create (rather than simply crapping out).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:15:31 +02:00
Sebastian Spaeth
f3ec6d9c7d Assert error hint when trying to create MailDir root
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=499755 shows the
cryptic output we have when, e.g. trying to put somethin in our mailDir
root via the nametrans options. This commit adds at least some hint as
to what went wrong using an "assert" message, although the correct thing
is to allow the creation of a maildir in the root folder.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-08 17:10:31 +02:00
Sebastian Spaeth
3dc9fc519a Throw errors on connection refused and on non-standard SSL ports
We were "crashing" with tracebacks when we could not connect to a host,
(e.g. because no service was on the port) and we were getting mysterious
SSL tracebacks when someone tried to connect via SSL to a non-ssl port.

In these cases, we will now throw an nice error message. On python<2.6
where no ssl module exists, we simply won't throw those errors.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-06-06 21:33:16 +02:00
Nicolas Sebrecht
a5a4cb214b Merge branch 'ns/fix-magicline-import' into next 2011-05-31 18:42:08 +02:00
Nicolas Sebrecht
e87f213046 fix magicline import
commit 0318c6a [Create LocalStatus or LocalStatusSQLite folders] changes import
of LocalStatus but doesn't preserve magicline.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-25 21:54:55 +02:00
Nicolas Sebrecht
fc6c12d96c improve message "error 111" if connection failed
Raise OfflineImapError with severity REPO explaining that the connection failed.
Before, no valuable information was given to the user.

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-25 20:32:41 +02:00
Tom Lawton
da36c5c1e7 Handle abort messages from GMail
Without this patch, we try to NOOP on a bad connection and crash messily.

Signed-off-by: Tom Lawton <tlawton@gmx.de>
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Ethan Glasser-Camp
2cc2ead503 Let the user configure how long to IDLE for
This commit was originally by James Bunton <jamesbunton@fastmail.fm>.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Ethan Glasser-Camp
36a0680191 Add check for IDLE in capabilities
If the server doesn't support IDLE, we fall back to the standard
noop() keepalive.

This commit was originally by James Bunton <jamesbunton@fastmail.fm>.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Ethan Glasser-Camp
d47bd1ff89 Change keepalive() to spawn IdleThreads
This is the commit that enables IDLE support. In order to do this, we
hijack the keepalive method. Instead of just sending NOOPs, it now
sends IDLE and responds accordingly, thanks to the IdleThread class.

This code was originally by James Bunton <jamesbunton@fastmail.fm>.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Ethan Glasser-Camp
9fa6bcdf82 Add IdleThread class
This encapsulates the logic for sending a keepalive/IDLE call,
including starting a sync if needed.

This code was originally by James Bunton <jamesbunton@fastmail.fm>.  I
modified the idle() method to put the select() call after
acquireconnection().

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Ethan Glasser-Camp
52cefb582c Recognize configuration for idlefolders
Mark this option as experimental and document its shortcomings in
MANUAL.rst.

This code was originally by James Bunton <jamesbunton@fastmail.fm>.

Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-24 18:55:46 +02:00
Nicolas Sebrecht
89a5d25263 fix broken ui Blinkenlights in multi-threaded mode
This is a regression introduced by commit d5493fe894
[threadutil: explicitly import get_ident from thread].

The threadid attribute was wrongly removed from the ExitNotifyThread class.
Restore it.

Tested-by: Mark Foxwell <fastfret79@archlinux.org.uk>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-18 18:38:05 +02:00
Nicolas Sebrecht
2d31abde76 v6.3.4-rc1
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-16 18:54:22 +02:00
Nicolas Sebrecht
ea8c1c0b6a Merge branch 'ns/threadutil-remove-os._exit' into next 2011-05-16 18:26:40 +02:00
Nicolas Sebrecht
41c21f5a9f fix: allow debugtype 'thread' from command line
commit f7e9d95 [Imply single-threaded mode with -d(ebug) command line option]
was broken. It pretends to imply singlethreading each time _unless_ user
explicitly asks for the debug 'thread' option.

Change the "force singlethreading" check from

  (force single threading if last column is true)

+------------------------------------------+----------------+
|                                          |                |
+---------------------------------+        |                |
|((SINGLETHREADING   THREAD_MODE) | "AND") | "NOT"          |
+------------------+--------------+--------+----------------+
| True             |  True        |  True  |  False (wrong) |
+------------------+--------------+--------+----------------+
| True             |  False       |  False |  True          |
+------------------+--------------+--------+----------------+
| False            |  True        |  False |  True (wrong)  |
+------------------+--------------+--------+----------------+
| False            |  False       |  False |  True          |
+------------------------------------------+----------------+

To the correct one

+--------------------------------------------------+-------+
|                                                  |       |
+-----------------------------------------+        |       |
|(("NOT" SINGLETHREADING    THREAD_MODE)  | "AND") | "NOT" |
+-------------------------+---------------+--------+-------+
| False                   | True          | False  | True  |
+-------------------------+---------------+--------+-------+
| False                   | False         | False  | True  |
+-------------------------+---------------+--------+-------+
| True                    | True          | True   | False |
+-------------------------+---------------+--------+-------+
| True                    | False         | False  | True  |
+-------------------------+---------------+----------------+

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-16 18:18:21 +02:00
Nicolas Sebrecht
694e5772da threadutil: remove broken os.exit()
We are missing the import of 'os' python module since commit d839be3c61
(Sat Apr 16 20:33:35 2005 +0100) which was when John switched from SVN to Git.

Happily, it help us today: we still had no feedback for this missing import,
6 years later.  So, we can remove the os.exit() call safely.

That beeing said, we still don't know if the above sys.exit() was ever touched.
My guess is that it never was. Keep this (hopefully) commented statement to
ensure the thread terminate and not play too much with the Murphy's law. :-)

Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-13 00:56:52 +02:00
Nicolas Sebrecht
21875852eb cleanup: remove unused initextnotify() function and inited variable
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-12 18:38:45 +02:00
Nicolas Sebrecht
136237b7dc refactoring: simplify the semaphorewait logic
The semaphorewait()/waitforthread() logic is usefull for IMAP starting
connections. We actually use it in imapserver only.

This patch removes the over-engineered factorized methods. It tend to simplify
the code by cleaning out a chain of two direct calls with no other processes.

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-12 18:30:39 +02:00
Nicolas Sebrecht
d5493fe894 threadutil: explicitly import get_ident from thread
The python module thread is the low-level module we should avoid to use in favor
of threading. We still need it to support old python because Thread.ident
doesn't exist before python 2.6:

	http://docs.python.org/library/threading.html#threading.Thread.ident

Make it clear we should avoid it.

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-12 18:28:49 +02:00
Nicolas Sebrecht
8a34edc8ca cleanup import satements
- conform to PEP8
- explicitly define symbols instead of 'import *'
- remove unused import

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-12 18:26:29 +02:00
Nicolas Sebrecht
a39128516b explicitly define symbols to import instead of 'import *'
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-09 22:43:03 +02:00
Nicolas Sebrecht
b25a72f8c8 cleanup: remove uneeded imports
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-09 22:42:15 +02:00