1) Rework the sleep abort request to set the skipsleep configuration
setting that the sleep() code checks.
2) Only output 15 rather than 50 debug messages on abort...
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The remote|local|statusrepo is an anttribute of each SyncableAccount()
anyway, so we don't need to pass it in, we can simply get it from the
Account().
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rename some variables, simplify the hotkeys treatment. Refresh/exit
signals still don't work as of yet, but will come.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Resizing a Blinkenlights terminal doesn't crash anymore, and actually
seems to be changing the size, with this patch.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We were still referring to s.gettf() in sleeping(self, ...) causing each
attempt to sleep to crash. Fix this, and the CursesAccountFrame.sleeping()
method. I am sure, there is still wrong and broken but we are getting there.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This function can IMHO lead to possible deadlocks when waiting for the
connectionlock. Do add a comment to that regard, this will need to audit.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The huge UI rework patch removed some obscure logic and special handling of
thread exit messages. It turns out that this was in fact still needed as a
specific exit message of the SyncRunner thread signified the threatmonitor
to quit.
We will want a nicer machinery for this in the future I guess, but fix the
eternal hang on exit by reintroducing a special exit message for the
SyncRunner thread, and return from the infinite monitor loop if SyncRunner
finishes.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
To make sure, the lock gets released even if we raise an exception between
acquire() and release()
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
All ExitNotifyThreads and InstanceLimitThreads are setDaemon(True) in their
constructor, so there is no need to do that again in the code.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
1) Differentiate error messages between imaplib.abort and imaplib.error
exceptions in the log.
2) Drop connections in the case of imapobj.error, it also might denote a
broken connection.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
During cleanup we often call releaseconnection in a finally: block. But
in cases of error, we might have dropped the connection earlier already
and set it to "None". In this case don't fail releaseconnection() but
make it a NOOP.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The default parameter value was "None", and we were comparing that
directly to the imaplib2 value of is_readonly which is False or True, so
the comparison always returned "False".
Fix this by setting the default parameter to "False" and not
"None". Also convert all users of that function to use False/True.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
It is basically unused by now. Rework to be able to make use of it
later, no functional changes.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Logging was flawed as the output was e.g. heavily buffered and people
complained about missing log entries. Fix this by making use of the
standard logging facilities that offlineimap offers.
This is one big ugly patch that does many things. It fixes the
Blinkenlights backend to work again with the logging facilities.
Resize windows and hotkeys are still not handled absolut correctly, this
is left for future fixing. THe rest of the backends should be working fine.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If folder creation failed, we would output the wrong repository and
folder name (copy'n paste error). Fix this so we actually output the
correct values.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Only import the lock, that we actually need. Also import the with statement
for use with python 2.5. We'll need it for sure in this file.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Drop a connection, if the NOOP to keep a connection open fails due to
broken connections.
Note that I believe this function is not working as intended. We grab
one random connection and send a NOOP. This is not enough to keep all
connections open, and if we invoke this function multiple times, we
might well always get the same connection to send a NOOP through.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
A repositories 'reference value is always prefixed to the full folder
path, so we should do so when creating a new one. The code had existed
but was commented out since 2003, I guess the "reference" option is not
too often used.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If port is None, we would try to format an empty string with %d wich
fails. Fix it by using %s.
Reported-by: Iain Dalton <iain.dalton@gmail.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This outputs a handy summary of your server configuration and version
strings etc, which is useful for bug reporting.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Was getting too large, split into an parse_cmd_options and a sync()
function. Moving config and ui to self.config and self.ui to make them
available through the OfflineImap instance.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
It happens quick, and clutters the log. So we can usually skip this. We
will output a log entry when we actually create a new folder anyway.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If we create a new folder we would previously not update our folder
list, which led to us skipping the synchronization of those new folders
during the initial run (subsequent runs would pick it up).
Invalidate the folder cache when we create a folder during folder
structure sync. Regetting the whole list from an IMAP server might be
slightly suboptimal from a performance point, but it is easy and will
lead to consistent results. Hopefully we will not have to create new
folders on each new run.
Reported-by: Daniel Shahaf <d.s@daniel.shahaf.name>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Modify the UI:acct and acctdone functions to keep tab of the time
inbetween. Put self.ui.acct() and acctdone() at the right places in
accounts.py so that the timing happens at the right places.
While modifying that loop, flatten the nested try: try: except: finally:
constructs, we require python 2.5 now which copes with that.
At the end of each account sync you will now see something like:
*** Finished account 'test' in 0:05
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Thread names are used to determine the logging header in the TTY ui. A
recent change made them too terse (basically only changing the account
name and not the folder names). Unbreak.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Output (2 of 500) when logging message copying. This required moving of
self.ui.copyingmessage into a different function where we actually have
the information about the progress handy.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than setting a global threadutil/profiledir variable, we make
set_profiledir a class function that sets the class variable profiledir.
While touching theprofiledir code, add warning to the user if the
profile directory existed before.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Registering a thread (associating it with a certain account name) would
fail if it was already registered. However, as we a) never unregister most
threads (bad) and b) single-threaded mode reuses threads, we failed when
syncing multiple accounts in single-threading mode.
This commit cleans up the functions to not make re-registering a thread
fatal (it could be legitimate, however it *should* not occur). Future
work needs to be done to unregister new threads at the appropriate places.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When checking for the IMAP4.abort() exception, we need of course to
perform:
except imapobj.abort:
and not
except imapobj.abort():
Thanks to Johannes Stezenbach <js@sig21.net> for pointing to the glitch.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
apply() has been deprecated since Python 2.3, and won't be working in
python 3 anymore. Use the functional equivalent throughout.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Dave identified a case where our new dropped connection handling did
not work out correctly: we use the retry_left variable to signify
success (0=success if no exception occured).
However, we were decrementing the variable AFTER all the exception
checks, so if there was one due to a dropped connection, it
could well be that we 1) did not raise an exception (because we want to
retry), and 2) then DECREMENTED retry_left, which indicated "all is
well, no need to retry".
The code then continued to check() the append, which failed with the
above message (because we obtained a new connection which had not even
selected the current folder and we were still in mode AUTH). The fix is
of course, to fix our logic: Decrement retry_left first, THEN decide
whether to raise() (retry_left==0) or retry (retry_left>0) which would
then correctly attempt another loop. I am sorry for this newbie type of
logic error. The retry count loop was too hastily slipped in, it seems.
Reported-by: Dave Abrahams <dave@boostpro.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If APPEND raises abort(), the (typ, dat) variables will not be set, so
we should not be using it for the OfflineImapError Exception
string. Fixing and prettifying the string formatting a bit at the same
time.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When syncfolder() fails, we output an error message containing the
foldername per the localfolder variable. However, the localfolder
variable is assigned inside our try: block and when the error occurs
there, we will have no localfolder variable to use for output. This
caused the errormsg to cause an Exception itself which unhelpfully
distracts from the root cause of the error.
Reconstruct the folder name in a bit more complex way, but in a way so
it is guaranteed to work (by relying on parameters passed in to the
function).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
A report by Dave Abrahams showed that the dequote() function failed when
invoked with an empty string. This fixes the function to be more robust.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
nametrans rules can lead to different visiblename names for the
top-level directory, specifically both '.' and '' (the latter was
recently introduced). However, we need to be able to compare folder
names to see if we need to create a new directory or whether a directory
already exists, so we need to be able to compare a repositories
visiblename (=transposed via nametrans rule) with another folder.
To make the top-level directory comparison happen, we enforce a
top-level name of '', so that comparisons work.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Some Webservers (I am looking at you Gmail) send different capabilities
before and after login, so they can tailor their server capabilities to
the user. While legal, this is uncommon and we were not updating our
server capabilities. Doing so allows us to detect that Gmail actually
supports the UIDPLUS extension, and we will stop mangling headers when
uploading to Gmail. This could lead to some performance gains when we
upload many messages to Gmail.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Commit b0e88622c4 changed dst_hash[folder.visiblename] to
dst_hash[folder.name] but we did not adapt all places where it is needed
to use visiblename again. This led to attempting to create a name on
REMOTE ignoring the nametrans setting on the LOCAL repository.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, we only checked if a LOCAL folder falls under the local
repositories folderfilter rule when deciding whether a folder should be
created on REMOTE.
However, we also do not want to create the folder on REMOTE if it would
fall under a folderfilter rule there. This patch prevents us from doing
so.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
It is not easy to think through when to use visiblenames() and whatnot. It seems I managed to not think it through properly. Which might be fixed now.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The thread ID is not really useful and looks ugly. It also makes lines
longer than needed, there is more useful information we can put in the
log. So do away with it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This will ignore any nametrans rules, so we might want to limit this
only to cases where no nametrans has been specified, or we might want to
use the nametrans setting of the dest repo.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
getvisiblename() was only defined on IMAP(derived) foldertypes, but we
want it on eg. Maildirs too, so we define it centrally in Folder.Base.py
rather than only in folder.IMAP.py.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Currently we only filtered IMAP repositories, this patch enables filtering
for Maildir repositories too.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We want to have these functions available for Maildir folders too, so we
can folderfilter a Maildir repository too (which is currently not possible)
This commit only move the corresponding functions from the IMAP to the Base
implementation. It should not change behavior in any way yet.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Also remove the removed parameters in the Gmail folder
initialization. This is one spot where I had forgotten to also strip the
parameters.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If nametrans translates to an empty directory we want to find the
top-level directory by name '' and not by name '.'. This unbreaks
nametrans rules that result in empty folder names.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, getfolder() would always construct new MaildirFolder()
objects, independent of whether the folder exists or not. Improve the
function to:
1) Scan and cache the folders if not already done
2) Return the same cached object if we ask for the same foldername twice
3) Reduce a tiny bit of code duplication
This is important because we handle stuff like folderfilter in the
scandir function and if we discard the scanned dir and create a new
object on folderget(), we will lose the folderfilter information.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Output a debug log line whenever we create a new folder on an IMAP
server. Also raise an OfflineImap Error in case we failed to create it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This variable shows if this folder should be synced or is disabled due to
a folderfilter statement. This lets us distinguish between a non-existent
folder and one that has been filtered out. Previously any filtered folder
would simply appear to be non-existing.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The Message UID is already the key to self.messagelist, so we have that
information. It is redundant to save the UID again as
self.messagelist[uid]{'uid': uid} and we never made use of the
information anyway.
The same thing should be done with the other 2 backends.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
A more pythonic and less verbose way to do the same. Add a comment what the
variable is all about.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Warn the user and abort when we attempt a plaintext login, but the
server has explicitly disabled plaintext logins.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Use the ui.error infrastructure that has been put in place and use
ui.terminate even if we received an Exception, so that we can output the
list of errors that we have. This does away with 2 now unused functions
in ui/UIBase.py
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We simply lock OfflineImap the same global way that we have always done
in addition to the previously implemented per-account lock. We can keep
both systems in parallel and then after a few stable releases, drop the
old-style global lock. by reverting this patch
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The next commit will make use of OfflineImapError but is transient (the
old-style lock). The commit is supposed to be reverted after a few
releases. So add the new import in a separate commit, because we might
need this even when reverting the commit.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, we were simply locking offlineimap whenever it was
running. Howver there is no reason why we shouldn't be able to invoke it
in parallel, e.g. to synchronize several accounts in one offlineimap
each.
This patch implements the locking per-account, so that it is possible to
sync different accounts at the same time. If in refresh mode, we will
attempt to loop three times before giving up.
This also fixes Debian bug #586655
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
open() and os.open() lead to different file permissions by default, and
while we have not changed the os.open that had been used, some code
changes led to these permissions slipping through. Fix this by setting
the permissions explicitly to 0666 (minus the users umask).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
repos.getuesr() asks for a username if none is specified, but in the
case of a tunnel connection, we don't need one, so we need to skip the
repos.getuser() call here.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
IMAPFolder has the repository and foldername values so it can get the
transposed (aka visiblename) of a folder itself just fine. There is no
need to pass it in as an separate parameter.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
They have the Repository() which contains the root, so no need to pass
it in as an extra parameter. Rename repository.LocalStatus()'s
self.directory to self.root for consistency with other backends.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It is possible to get the config parameter from the Repository() which is
set in BaseFolder, so we set self.config there and remove the various
methods and 'config' parameters that are superfluous.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We passed in the accountname to all derivatives of BaseFolder, such as
IMAPFolder(...,repository,...,accountname), although it is perfectly
possible to get the accountname from the Repository(). So remove this
unneeded parameter. Each backend had to define getaccountname() (although
the function is hardly used and most accessed .accountname directly).
On the other hand BaseFolder was using getaccountname but it never defined
the function. So make the sane thing, remove all definitions from backends
and define accountname() once in Basefolder. It was made a property and not
just a (public) attribute, so it will show up in our developer
documentation as public API.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
As all Folders share these parameters, we can safely handle them in
BaseFolder. This makes sense, as BaseFolder has a getname() function
that returns self.name but nothing actually set self.name.
It also saves a few lines of code.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In getmessage() we were releaseing a connection when we detected a
dropped connection, but it turns out that this was not enough, we need
to explicitely discard it when we detect a dropped one. So add the
drop_conn=True parameter that was recently introduced to force the
discarding of the dead conection.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The quickchanged() function was not handling dropped connections yet. If
IMAP4.select() throws a FOLDER_RETRY error, we will now discard the
connection, reconnect and retry.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Beauty of code is probably a subjective measure, but this patch hopefully
is an improvement over the previous incarnation without changing
functionality.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
repository.BaseRepository().restore_atime() was testing in complex ways
that it only operates on a Maildir and that the 'restoreatime' setting
is set. This is unecessary, we can simply make the base implementation a
NoOp, and move the implementation to MaildirRepository().
This will save a tad of work for everyone doing IMAP<->IMAP
synchronization and simplify the code. Also document the functions.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We only explicitly tested for 'yes' when we have a nice function to get
boolean settings which also works with Treu/False/NO, etc...
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The readonly feature was introduced to safeguard repositories from
accidental modifications. Unfortunately, my patch treated the readonly
setting as a string and not as a boolean, so if *anything* was set in
the configuration file as 'readonly', this value evaluated to True
Fortunately this was safe, we never treated a repository that we wanted
read-only as read-write. We always treated them readonly if something
was configured as "readonly=..." even if that was False.
The fix is simply to use getconfboolean() rather than getconf() which
checks for True/False/On/Off/yes/no/1/0 and hands back the correct boolean.
Reported-by: Tomasz Nowak <nowak2000@poczta.onet.pl>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We rely on the number of mails being returned by the imapobj.select()
call, however that only happens if we "force" a real select() to occur.
Pass in the force parameter that I dropped earlier (we did not make use
of the return value when I dropped it, that is how it slipped through).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were retrying indefinitely on imapobj.abort() (as that is what
imaplib2 suggests), but if the failure occurs repeatedly, we'll never
quit this loop. So implement a counter that errs out after unsuccessful
retries.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Finally, actually discard dropped connections when we detect them as an
imapobj.abort() has been thrown. In this case, invoke releaseconnection
with drop_conn=True.
We don't need the self.aborted attribute to get signified of dropped
connections. An Execption during the noop will do.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
while True: if a: return
is equivalent to
while not a:
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Variable name 'event' is as bad as it gets. Rename it to something that
actually describes what it is about.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Add code documentation throughout the idle() function.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Don't redefine the idle callback function on every run in the while
loop, define it once when we enter the function.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Python 2.5 has no ssl module, and we can therefor not get the server
certificate for fingerprint verification. Add a check that disables
fingerprint verification for python 2.5.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
imaplib2 has changed internally to use self.sock for its ssl socket when
it used to be sslobj. Reflect that change.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If we connect to a SSL server (not STARTTLS) and no CA cert has been
specified for verification, we check the configured SSL fingerprint and
bail out in case it has not been set yet, or it does not match.
This means one more mandatory option for SSL configuration, but it
improves security a lot.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If maxage is set too large, we would even SEARCH for negative
years. With devastating results. So implement some sanity check and err
out in case the year does not make sense.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We can use Imaplib's monthnames and shorten the construction of the date
by using them rather than hardcoding them again.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Results are delivered in a 1-element list, and somehow I managed to drop
a [0] in the previous patches. We need to look at the element of course,
or our string splitting will fail horribly. Sorry this somehow slipped
through.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If a folder is empty, most servers will return EXISTS 0 and imaplib2
passes back ['0'] as return value to a select(). It returns [None] if
no EXISTS response was given by the server at all.
Attempting to fetch the UIDs of 0 emails which leads to
various error messages (One server responds with "NO No matching
messages", Gmail seems to say "BAD Bad message sequence 1:*" for some
(although it is working fine for me with Gmail, so it might behave
different for different people).
In case we get an None or 0 back, we simply stop caching messages as the
folder is empty. This should fix the various error reports that have
popped up.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make sure that when a connection is dropped during append, we really
discard the broken connection and get a new one, retrying. We retry
indefinitely on the specific abort() Exception, as this is what imaplib2
suggests us to do.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If a connection is broken, we want to have it really dropped and not be
reused. So far, we are checking the .Terminate attribute for this, but
according to the imaplib2 author, it is only set on normal shutdown and
it is an undocumented attribute whose meaning could change any time.
This patch introduces the parameter drop_conn which allows to tell
releaseconnection() that we really want to connection being dropped from
the pool of available connections and properly destroy it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than output the full list of messages, coalesce it into number
ranges wherever possible.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This function was badly named and completely undocumented. Rework it to
avoid copying the full UID list using an iterator. Make it possible to
hand it a list of UIDs as strings rather than implicitely relying on the
fact that they are numeric already. Document the code.
The behavior off the function itself remained otherwise unchanged.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
SEARCH and FETCH were never checking that the IMAP server actually
returned OK. Throw OfflineImapErrors at severity FOLDER in case one of
them fails.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Some code cleanup. If we want to examine all messages of a folder, don't
try to find out how many there are and request a long list of all of them,
but simply request 1:*. This obliviates us from the need to force a select
even if we already had the folder selected and it requires us to send a
few less bytes over the wire.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Do away with the wrapping of this code in a try...except KeyError, as
this code cannot conceivably throw a KeyError. Even if it could, it
should be documented why we should simply return() in this case.
Shorten some of the variable names and minor code cleanup while taking
the git blame anyway.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than passing in huge lists of continuous numbers which eventually
overflow the maximum command line length, we coalesce number ranges
before passing the UID sequence to SEARCH. This should do away with the
error that has been reported with busy mailing lists and 'maxage'.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It is not needed. list(ALIST) will create a new copy of the list just
fine.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make sure that when a connection is dropped during append, we really
discard the broken connection and get a new one, retrying. We retry
indefinitely on the specific abort() Exception, as this is what imaplib2
suggests us to do.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If a connection is broken, we want to have it really dropped and not be
reused. So far, we are checking the .Terminate attribute for this, but
according to the imaplib2 author, it is only set on normal shutdown and
it is an undocumented attribute whose meaning could change any time.
This patch introduces the parameter drop_conn which allows to tell
releaseconnection() that we really want to connection being dropped from
the pool of available connections and properly destroy it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When invoked with FETCH 1:* (UID), imaplib returns [None] for empty
folders. We need to protect against this case and simply 'continue' here.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If a connection is dropped for some reason while fetching a message, the
imapobj.uid command throws an imapbj.abort() Exception which means we are
supposed to retry. Implement a fail loop that drops the connection, gets a
new one and attempts the command another time.
Remove obsolete comment that we need to catch nonexisting messages. We do
now.
GMail seems to drop connections left and right. This patch is a response to
the reported mail "4E5F8D8C.1020005@gmail.com" by zeek
<ezekiel.das@gmail.com>.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Handle the case gracefully where a server has closed an IMAP connection
that we want to use for IDLEing. Simply have it dropped and get a new one
in this case. THis should get rid of the errors reported by John Wiegley
in mail id:"m2sjohd16t.fsf@gmail.com".
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Imapserver.acquireconnection will succeed even whent the server connection
has been terminated and the first IMAP operation will throw an exception.
Often this is the folder SELECT operation (e.g. after an idle timeout), as
has been reported by John Wiegley. Catch this case and throw an
OfflineImapError with severity FOLDER_RETRY to notify consumers that they
are supposed to retry.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In cases where processing a folder failed, but a retry might well succeed
e.g. when the server connection had simply timed out and was disconnected,
we can throw a FOLDER_RETRY (which is less severe than FOLDER).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Name parameter that hands us a Status Repository 'status_repo' and not
'copyfolders' as was before:
a) make it clear that we pass in a repository and not folder
instances. That was very confusing before.
b) We were always only using one 'copyfolders' item anyway, so let us
not make it a list.
Go through the list and make the variable nameing consistent:
dst_repo rather than dest (is it a folder?) to match the status_repo
naming scheme.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
To preserve previous behavior and save a few CPU cycles, we were not sorting UID
lists and only collapsed them if they were alreay sorted. Vincent pointed out
that this is not always the case and unsorted lists lead to non-optimally
collapsing.
Force lists to numeric types and sort them before collapsing.
Reported-by: Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were using self.getfolderbasename(self.name) but the API is simply
getfolderbasename(). Fix this glitch.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
SafeConfigParser is very similar to the currently used ConfigParser but
it supports interpolation. This means values can contain format strings
which refer to other values in the same section, or values in a special
DEFAULT section. For example:
[My Section]
foodir: %(dir)s/whatever
dir=frob
would resolve the %(dir)s to the value of dir (frob in this case). All reference expansions are done on demand.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
as that method doesn't exist on sets. Rather call the inbuilt
sorted(flags). This fixes Exceptions being thrown when using the sqlite
backend.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were omitting an '%' where we needed it. Also include the traceback
information where it belongs in the new ui.error infrastructure.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
443 is of course the https and not the IMAPS standard port. Fix.
Thanks to Daniel Shahaf <d.s@daniel.shahaf.name> for the heads up.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Commit e023f190b0 changed the storing of
file paths in the messagelist variable to be relative paths, but we were
using the full absolute path anyway as we missed one spot.
Adapt this and construct the full file path in the one place where we
need it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than output the full list of messages, coalesce it into number
ranges wherever possible.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This function was badly named and completely undocumented. Rework it to
avoid copying the full UID list using an iterator. Make it possible to
hand it a list of UIDs as strings rather than implicitely relying on the
fact that they are numeric already. Document the code.
The behavior off the function itself remained otherwise unchanged.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
As this is essentially what it is, a set of values. This allows as
to do set arithmetics to see, e.g. the intersection of 2 flag sets
rather than clunkily having to do:
for flag in newflags:
if flag not in oldflags:
oldflags.append(flag)
Also some more code documenting.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Empty foldernames (as they could be created through nametrans) were
failing as the uidvalidity and status files names as determined by
folder/Base.py:getfolderbasename() lead to invalid file names ''.
Fix this by handling empty file names and translating them to '.' which
leads to the special file name 'dot'. (this special value existed before
and was not invented by this patch)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
I changed the API to pass in a folder rather than a list of folders, but
used getnicename() on the wrong object. It is not used on the folder but
on the ui object. Fix this and give the variable somewhat better names.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It works by fetching all headers of new messages from IMAP server and
searching for our X-OfflineIMAP marker by using regular expression.
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We only copy to a single folder anyway, so clean up the code to only
pass in a single folder.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
imaplib2 always attempts to verify a certificate if a verification
callback function is passed in, even the certificate is None
specified. Disable the verification excplictly by setting the
verification function to None in that case.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
we need errno.CONNREFUSED, but through some merging mishaps(?) the part
that actually imported errno was missing. Import the errno module.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
imaplib renamed self.sslobj to self.sock and our overriden open()
functions were failing for that reason when updating imaplib2 to
v2.28. It turns out that all of our custom initializations are being
done by stock imaplib2 now anyway, so there is no need to override them
anymore. This lets us simplify the code we have to worry about.
Move the verifycert() function to the imapserver.py file, it is now a
callback function that is being handed to imaplib from there, so it
makes sense to also define it in our imapserver function...
(this also lets us easily make use of the verifycert function in the
starttls case in the future)
TODO: we need to examine if and why we still need to override the
select() function, it is the only reason why we still wrap the IMAP4
classes.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Imaplib2 2.28 can deal with ID sequences, such as 1:*, so we need to
bump upstream in order to make use of these features.
Note that this revision will not run correctly as it requires
adaptations to our code, which happens in the next commit.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were not including the full server reply into our error message. Fix
that so we get better error logs.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This is a bug fix on several levels. 1) We were lacking the import of
OfflineImapError. 2) OfflineImap.ERROR.MESSAGE was misspelled as ERROR.Message.
3) COntinuing with the next message only worked in single-thread mode
(using debug) and not in multi-thread mode. The reason is that we were
invoking a new thread and catching Exceptions in the main thread. But
python immediately aborts if an Exception bubbles up to a thread start.
This was fixed by catching exceptions directly in copymessageto() which
is the new thread, rather than catching them in the main thread.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Allowing to specify the char to use in the BLinkenlights is a bit over
the top and bloats our default offlineimap.conf. The dot is just fine,
so let us settle for it and cut the example config file by an unneeded
section.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
append() raises an Exception, in case the IMAP server replies with 'BAD'
(but not when it responds with 'NO') but we were not catching that. Do
catch the situation and also raise an OfflineImapError at MESSAGE
severity, so that we can continue with the next message.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
sys.exc_traceback is long deprecated and is seems removed in python2.7,
so document the legitimate use of sys.exc_info()[2] instead.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We set an imapobj.mustquote which apparently was used in previous
incarnations of imaplib or imaplib2, however, nothing in our codebase
makes use of that. So let us remove it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This function was overridden as the IMAP version apparently had been
using imapobj.myrights() at some point in time, which was not
implemented in the Gmail version. However, IMAP is not using myrights()
anymore, and as that is an extension that needs to be advertised in
CAPABILITIES we should not unconditionally use it anyway.
So remove the function that is identical to it's ancestor's
function.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This code was unused and broken. It is still unused but this commit
fixes it. (We should retain the method in case we ever start calling
getfolders() on LocalStatus.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than throwing ValueError, we now properly throw OfflineImapError
when selecting a folder in folderincludes. So we also need to catch
OfflineImapErrors here. If they are of severity FOLDER, just ignore the
invalid folder and continue. If the error is more severe, bubble it up.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Remove a level of wrapper abstraction that is not needed. Just use
IMAPserver and be done with it.
We do this by passing in the IMAPRepository() instance rather than a
long list of single paramters to the IMAPServer instanciation. This way
we can retrieve all repository parameters ourselves, rather than passing
a dozen paramters into IMAPServer. Also, this enables us to pass the
repository() object into our WrappedIMAP4() instance, so that it can
query, e.g. the SSL fingerprint configuration.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We simply assert()ed that APPENDing a message returned OK, but in some
cases (e.g. Google chat messages) APPEND might return BAD or NO too. We
should be throwing an OfflineImapError here at MESSAGE level, so that we
can continue to sync all other messages, and still give the user some
details on what went wrong at the end of the sync run.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than using ui.warn, use ui.error() which outputs Exceptions to
the error log, saving them to a stack, so we get notified again at the
end of the sync run.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Output all raised Exceptions error strings to the error log. If we are
in debug mode, we also output the traceback of the exception.
Save all exceptions that occur during the run time to a Queue and output
them all again when the offlineimap sync is finished.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Message was stored to dstfolder, but we can't find it's UID. This means we can't
link current message to the one created in IMAP. So we just delete local message
and on next run we'll sync it back. Also fixed imap.savemessage description.
This was broken by e20d8b9679.
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It's not enough to place header after first newline, since this might break
multiline rfc0822 folded long header lines. Those are difined as CRLF followed
by white space. Instead we'll search for two successive CRLF sequences which
mark end of mail headers and place our header just before that.
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
That makes OfflineIMAP to use exclamation mark (!) instead of colon for storing
messages. Such files can be written to windows partitions. But you will probably
loose compatibility with other programs trying to read the same Maildir.
Signed-off-by: Vladimir Marek <vladimir.marek@oracle.com>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
getselectedfolder was using a cached variable that we were setting in
select(), but sometimes the IMAP4 instance got into the SELECTED state
without explicitely select()ing, it seems, and our variable was unset.
Let us just use the self.mailbox variable that imaplib2 is setting when
select()ing rather than doing our own caching. Also remove the part
where we were setting the cache.
Just access self.state rather than looking up self.state via
self.getstate() every time, it is just an unnecessary layer of
redirection.
Original-patch-by: Arnaud Fontaine <arnau@debian.org>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In commit 89cbdc9, usage of SSLError was dropped but later reintroduced
without importing SSLError exception.
Signed-off-by: Arnaud Fontaine <arnau@debian.org>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than the extremly verbose NO WARRANTY blurb, we output a somewhat
smaller initial text which should still make the GPL happy.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The traceback module has format_exc() for this purpose so let's use that.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We we using the variable 'severity' in a few places to throw
OfflineImapErrorrs of severity REPO. Somehow, that variable is now not
accessible in all places that refer to it, so we move where it is
defined to before all the 'if' checks which might make use of it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If we do not use a SSL connection anyway and if the server supports it,
authenticate automatically with STARTTLS.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In an IMAP run where we did not have to sync anything, I spend nearly a
fulls second in imaputil.debug() without even having debug output
enabled. imapsplit is mainly called by flagsplit() which will also do
debug output, so we get TONS of nearly duplicate debug output in the log
which makes it really hard to analyze.
Cut down the debug logging in imapsplit, we should be debug logging
stuff at a slightly higher level than here anyway.
This one-line change sped up my folder sync (without having to sync
anything) by 0.5 seconds even when debugging is disabled.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously, we instanciated an MappedImapFolder, and would cleverly (too
cleverly?) invoke methods on it casting it to an IMAPFolder by calling
methods such as: self._mb.cachemessages(self) where self._MB is the class
IMAPFolder and self and instance of MappedImapFolder. If
e.g. cachemessages() invokes a method uidexists() which exists for
MappedImapFolder, but not directly in IMAPFolder, I am not sure if
Python would at some point attempt to use the method of the wrong class.
Also, this leads to some twisted thinking as our class would in same
cases act as an IMAPFolder and in some cases as an MappedImapFOlder and
it is not always clear if we mean REMOTE UID or LOCAL UID.
This commit simplifies the class, by a)doing away with the complex Mixin
construct and directly inheriting from IMAPFOlder (so we get all the
IMAPFOlder methods that we can inherit). We instantiate self._mb as a
new instance of IMAPFolder which represents the local IMAP using local
UIDs, separating the MappedIMAPFolder construct logically from the
IMAPFolder somewhat.
In the long run, I would like to remove self._mb completely and simply
override any method that needs overriding, but let us take small and
understandable baby steps here.
Reported-and-tested-by: Vincent Beffara <vbeffara@gmail.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
sudir->subdir in a debug statement. Thanks ccxCZ on IRC for the heads
up.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
During a sync run, someone might remove or move IMAP messages. As we
only cache the list of UIDs in the beginning, we might be requesting
UIDs that don't exist anymore. Protect folder.IMAP.getmessage() against
the response that we get when we ask for unknown UIDs.
Also, if the server responds with anything else than "OK", (eg. Gmail
seems to be saying frequently ['NO', 'Dave I can't let you do that now']
:-) so we should also be throwing OfflineImapErrors here rather than
AssertionErrors.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
I discovered that we do not run ui.terminate in all circumstances, so
make sure that we call with properly at the end of each run (whether in
threaded or single-thread mode).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Commit 1754bf4110 introduced a blunder:
- for dirname in os.listdir(toppath) + ['.']:
+ for dirname in os.listdir(toppath) + [toppath]:
...
- if self.getsep() == '/' and dirname != '.':
+ if self.getsep() == '/' and dirname:
This change was plainly wrong and would never have worked, so this
commit reverts above bit. While touching the function, some minor code
documentation, cleanup and limiting line length to 80 chars.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Folder.savemessage() is supposed to return the new UID that a backend
assigned, and it BaseFolder.copymessageto() fails if we don't return a
non-negative number in the savemessage() there.
For some reason, the UIDMappedFolder was not returning anything in
savemessage, despite clearly stating in the code docs that it is
supposed to return a UID. Not sure how long this has already been the
case. This patch fixes the UIDMappedFolder to behave as it should.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
During a sync run, someone might remove or move IMAP messages. As we
only cache the list of UIDs in the beginning, we might be requesting
UIDs that don't exist anymore. Protect folder.IMAP.getmessage() against
the response that we get when we ask for unknown UIDs.
Also, if the server responds with anything else than "OK", (eg. Gmail
seems to be saying frequently ['NO', 'Dave I can't let you do that now']
:-) so we should also be throwing OfflineImapErrors here rather than
AssertionErrors.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Fix a gssapi issue where threads beyond the first would not
be able to authenticate against the imap server. This is
done by using the connection lock around the gssapi
authentication code and resetting (and releasing) the
kerberos state after success so that subsequent connections
may make use of kerberos.
Signed-off-by: Scott Henson <sjh@foolishpride.org>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We currently don't care about expiration dates of the servers SSL
certificate. This patch adds a check that fails Cert verification when
it is past its due date. There is no way or option to override this
check.
Unfortunately we only seem to be able to get SSL certificate data when
we passed in a CA cert file? How do we get that date when we don't have
a ca cert file?
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously we were attempting to save out mails according to
http://www.qmail.org/man/man5/maildir.html in 4 steps:
1 Create a unique filename
2 Do stat(tmp/<filename>). If it found a file, wait 2 sec and go back to 1.
3 Create and write the message to the tmp/<filename>.
4 Link from tmp/* to new/*
(we did step 2 up to 15 times) But as stated by
http://wiki1.dovecot.org/MailboxFormat/Maildir (see section 'Issues with
the specification'), this is a pointless approach, e.g. there are race
issues between stating that the filename does not exist and the actual
moving (when it might exist).
So, we can simplify the steps as suggested in the dovecot wiki and
tighten up our safety at the same time.
One improvement that we do is to open the file, guaranteeing that it did
not exist before in an atomic manner, thus our simplified approach is
really more secure than what we had before.
Also, we throw an OfflineImapError at MESSAGE level when the supposedly
unique filename already exists, so that we can skip this message and
still continue with other messages.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
MaildirFolder.messagelist[*]['filename'] was storing the absolute file
paths for all stored emails. While this is convenient, it wastes much
space, as the folder prefix is always the same and it is known to the
MaildirFolder. Just 40 chars in a folder with 100k mails waste >4MB of
space. Adapt the few locations where we need the full path to construct
it dynamically.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We use getfullname() very often (thousands to millions), yet we
dynamically calculate the very same value over and over. Optimize this
by caching the value once and be done with it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In some cases we had offlineimap trying to delete emails that shouldn't
be deleted. E.g. https://bugzilla.redhat.com/show_bug.cgi?id=708898.
It turns out that imaplib2 does not like FETCH responses that are
interrupted by other unsolicited server responses, e.g.
* OK Searched 43% of the mailbox, ETA 0:12\r\n
Bump imaplib2 to a version that can cope with these (legal) responses by
the IMAP server.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
we do:
for msgid in imapdata:
maxmsgid = max(long(msgid), maxmsgid)
and then basically immediately:
maxmsgid = long(imapdata[0])
throwing away the first assignment although the first method of
assigning is the correct one. The second had been forgotten to be
removed when we introduced the above iteration. This bug would fix a
regression with those broken ZIMBRA servers that send multiple EXISTS
replies.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
All months names are 3-letter abbreviated, but accidentally June and
July slipped through. Thanks to the heads up by
Philipp Kern <pkern@debian.org>.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Previously we hard-coded the imap server name in the case of Gmail
repositories, but often we need a different host name. So, allow people
to specify the hostname via the regular "remotehosteval" and
"remotehost" settings, and only falling back to imap.gmail.com when
nothing has been specified.
Cache the hostname, so we don't evaluate the whole thing each time we
query the host name.
Make the remotehosteval processing more robust, by catching any
Exceptions that occur, and throw a OfflineImapError, that explains where
exactly the error had occured. You can test this, e.g. by setting
remotehosteval to 1/"n" or some other invalid expression.
The whole IMAP.gethost() function has been documented code wise while
going through.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We currently do not allow nametrans rules such as
nametrans = lambda foldername: re.sub('^INBOX$', '', foldername)
because we crash with a traceback when running:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=499755
The underlying reason is that we cannot create the "top level" root
directory of the Maildir in the function makefolders(), it will bail
out. John Goerzen intentionally prevented offlineimap from creating the
top-level dir, so that a misconfiguration could not arbitrarily create
folders on the file system. I believe that it should be perfectly
possible to automatically create the root dirctory of the maildir. We
still protect against folder creations at arbitrary places in the file
system though.
This patch cleans up makefolders(), adds documentation, allows to
automatically create rootfolders if needed (using absolute paths) and
adds some robustness in case the folders already exist that we want to
create (rather than simply crapping out).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=499755 shows the
cryptic output we have when, e.g. trying to put somethin in our mailDir
root via the nametrans options. This commit adds at least some hint as
to what went wrong using an "assert" message, although the correct thing
is to allow the creation of a maildir in the root folder.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were "crashing" with tracebacks when we could not connect to a host,
(e.g. because no service was on the port) and we were getting mysterious
SSL tracebacks when someone tried to connect via SSL to a non-ssl port.
In these cases, we will now throw an nice error message. On python<2.6
where no ssl module exists, we simply won't throw those errors.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
commit 0318c6a [Create LocalStatus or LocalStatusSQLite folders] changes import
of LocalStatus but doesn't preserve magicline.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Raise OfflineImapError with severity REPO explaining that the connection failed.
Before, no valuable information was given to the user.
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Without this patch, we try to NOOP on a bad connection and crash messily.
Signed-off-by: Tom Lawton <tlawton@gmx.de>
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This commit was originally by James Bunton <jamesbunton@fastmail.fm>.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If the server doesn't support IDLE, we fall back to the standard
noop() keepalive.
This commit was originally by James Bunton <jamesbunton@fastmail.fm>.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This is the commit that enables IDLE support. In order to do this, we
hijack the keepalive method. Instead of just sending NOOPs, it now
sends IDLE and responds accordingly, thanks to the IdleThread class.
This code was originally by James Bunton <jamesbunton@fastmail.fm>.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This encapsulates the logic for sending a keepalive/IDLE call,
including starting a sync if needed.
This code was originally by James Bunton <jamesbunton@fastmail.fm>. I
modified the idle() method to put the select() call after
acquireconnection().
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Mark this option as experimental and document its shortcomings in
MANUAL.rst.
This code was originally by James Bunton <jamesbunton@fastmail.fm>.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This is a regression introduced by commit d5493fe894
[threadutil: explicitly import get_ident from thread].
The threadid attribute was wrongly removed from the ExitNotifyThread class.
Restore it.
Tested-by: Mark Foxwell <fastfret79@archlinux.org.uk>
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We are missing the import of 'os' python module since commit d839be3c61
(Sat Apr 16 20:33:35 2005 +0100) which was when John switched from SVN to Git.
Happily, it help us today: we still had no feedback for this missing import,
6 years later. So, we can remove the os.exit() call safely.
That beeing said, we still don't know if the above sys.exit() was ever touched.
My guess is that it never was. Keep this (hopefully) commented statement to
ensure the thread terminate and not play too much with the Murphy's law. :-)
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The semaphorewait()/waitforthread() logic is usefull for IMAP starting
connections. We actually use it in imapserver only.
This patch removes the over-engineered factorized methods. It tend to simplify
the code by cleaning out a chain of two direct calls with no other processes.
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The python module thread is the low-level module we should avoid to use in favor
of threading. We still need it to support old python because Thread.ident
doesn't exist before python 2.6:
http://docs.python.org/library/threading.html#threading.Thread.ident
Make it clear we should avoid it.
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
- conform to PEP8
- explicitly define symbols instead of 'import *'
- remove unused import
Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were outputting full message bodies to the debug log (often stderr),
and then again (as they go over the imaplib2 wire, imaplib logs
everything too). Not only is quite a privacy issue when sending in debug
logs but it can also freeze a console for quite some time. Plus it
bloats debug logs A LOT.
Only output the first and last 100 bytes of each message body to the
debug log (we still get the full body from imaplib2 logging). This
limits privacy issues when handing the log to someone else, but usually
still contains all the interesting bits that we want to see in a log.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The syntax was not right, and deleting messages from the LocalStatus
failed. (We passed in the full list of uids and we need to pass in one
uid at a time (as a tuple). Deleting messages works now.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The SigListener class was used to queue folders that we need to sync and
to receive "resync" and "abort" signals. It was undocumented and weird
and we had to pass "siglisteners" through the whole program.
Simply do away with it, and make 2 functions in the Account() class:
set_abort_event and get_abort_event which can be used to set and check
for such signals. This way we do not need to pass siglisteners all over
the place. Tested Blinkenlights and TTYUI uis to make sure that SIGUSR1
and SIGUSR2 actually still work.
Document those signals in MANUAL.rst. They were completly undocumented.
This simplifies the code and interdependencies by passing less stuff
around. Removes an undocumented and weirdly named class.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Throw an OfflineImapError when SELECTing a folder is unsuccessful and
bail out with a FOLDER serverity. In accounts.py catch all
OfflineImapErrors and either just log the error and skip the folder or
bubble it up if it's severe.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Use two %s in the message for both string parameters.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make the folder classes use uidexists() more. Add some code
documentation while going through.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Depending on the configuration we use the plain text or the new
experimental sqlite backend for the LocalStatus cache. Make plain text
the default status backend but allow people to configure
status_backend=sqlite in their [Account ...] section.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Test if sqlite is multithreading-safe and bail out if not. sqlite
versions since at least 2008 are.
But, as it still causes errors when 2
threads try to write to the same connection simultanously (We get a
"cannot start transaction within a transaction" error), we protect
writes with a per class, ie per-connection lock. Factor out the retrying
to write when the database is locked.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make getfolders() invoke getfolder() for each folder rather than
duplicating code. Also add a forgetfolders() implementation.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
doautosave was a useless variable before (it was *always* 1). So we
remove the self.dofsync variable and store in doautosave whether we
should fsync as often as possible (which really hurts performance).
The sqlite backend could (at one point) use the doautosave variable to
determine if it should autocommit after each modification.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Python's new style classes derive from object and str(class().__class__)
will return a slightly different format. class().__class.__name__ will
still work for both old and new style classes, so use that instead.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Based on patches by Stewart Smith, updated by Rob Browning.
plus:
- Inherit LocalStatusSQLFolder from LocalStatusFolder
This lets us remove all functions that are available via our ancestors
classes and are not needed.
- Don't fail if pysql import fails. Fail rather at runtime when needed.
- When creating the db file, create a metadata table which contains the
format version info, so we can upgrade nicely to other formats.
- Create an upgrade_db() function which allows us to upgrade from any
previous file format to the current one (including plain text)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
A typo prevented us from enforcing singlethreading mode when selecting debugging.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Using threading._VERBOSE=1 is broken since python 2.6 till at least
python 3.2, (http://bugs.python.org/issue4188) so we can't use it for
our thread debugging.
Remove the usage of threading._VERBOSE, and implement a "light thread
debug log" that for now outputs information when a new thread is being
registered and when it is being unregistered. I am sure we will be able
to add more thread debugging information over the time.
Besides '-d thread' this will re-enable the usage of -d 'all' for the
most verbose debugging of all categories.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When our LocalStatus cache is corrupt, ie e.g. it contains lines not in
the form number:number, we would previously just raise a ValueError
stating things like "too many values". In case we encounter clearly
corrupt LocalStatus cache entries, clearly raise an exception stating
the filename and the line, so that people can attempt to repair it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This should get rid of intermittent network failures, but lets us bail
out on permanent errors.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
By aborting the account syncing, the looping and logging an error
message. We will introduce a ui.error() rather than a ui.warn() function
which saves all Exceptions in a Queue and outputs them at the end of the
program.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In case we misconfigured a server name or are otherwise offline, a
socket.gaierror will be raised when attempting to connect. We catch that
case and raise an OfflineImapError with severity ERROR.REPO, meaning we
should stop syncing this account and continue with the next one.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This Exception can be thrown whenever a sync error occurs. It is used
for outputting sensible error messages to the user and it denotes a
"severity", it can tell offlineimap if we need to abort a message, a
folder, a repo sync, or whether we should indeed abort immediately.
The exception is not yet used in the code.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Currently, account.syncrunner() has 2 separate duplicated code paths
depending on whether we want to autorefresh after some waiting perios
or not. Unify those code paths by setting "looping = False" in case
self.refeshperiod == 0 after the first run. Behavior is identical to
before.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This commit enables true 1-way syncing between repositories. This has
often been demanded for backup purposes when you do not want to cause
accidental modifications of your backup that would be propagated to the
other side.
This has been implemented by allowing to configure a Repository as
'readonly' to forbid any modification on it.
'readonly' applies to all the type of repositories.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This debug type will always be enabled whenever any debugging is enables
and it outputs debug messages that cannot be categorized among any of
imap, maildir (e.g. things that concern the sync logic).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Improve the code documentation (still much more to do) and also add some
more meat to the structure of the developer documentation.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This function will need much more "robustifying", but the very least we
can do is to print the file name and line that are giving trouble.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
