IMAP, Maildir, and LocalStatus abort if in dry-run mode. IMAP and Maildir
will log that they "would have" created a new folder.
This will probably fail later on as we can not cache messagelists on
folder that don't exist, so --dry-run is not yet safe when new folders
have been created.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
1) Set attribute self.dryrun depending on whether we are in dry-run mode.
2) Don't actually call hooks in --dry-run (just log what you would
invoke
3) Don't write out the mbnames file in --dry-run mode.
Repository, and Folder levels still need to be protected in dry-run mode
as of now.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
And set the [general]dry-run=True setting if yes. It is not used yet.
Also set ui.dryrun to True so we can output what WE WOULD HAVE DONE in
dryrun mode.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
A convenience helper function that allows to set a configuration value
if the user has not explicitly configured anything ie the option does
not exist yet in the configuration. It won't do anything, if the option
exists.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Bail out with a better Exception and error text. The whole mapped
UID situation needs to be improved though.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If we cannot identify the new UID after a sendmessage(), log a better error
message, including the server response for better debugging.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
While improving the test suite, I noticed that we would not create folders on
the remote in some cases when we should (yay for test suites!). This is because
we were testing the untransposed LOCAL foldername and check if it existed on
the remote side when deciding whether we should potentially create a new folder.
Simplify the code by transposing the LOCAL folder names in dst_hash, saving us
to create another confusing "newsrc" temp variable. Make the code a bit more
readable by using dst_name_t to indicate we operate a transposed folder name.
This now passes test 03 (using invalid nametrans rules) when test 03 would pass
before.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rename variable src_name to src_name_t to indicate that it is the transposed
name. Also rather than testing the hash thingie, we can simply test for
"if source_name_t in dst_folders" now.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This allows to compare folders directly with strings. It also allows
constructs such as "if 'moo' in repo.getfolders()".
See the code documentation for the exact behavior (it basically is equal if
it's the same instance *or* a string matching the untranslated folder name.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
By default we sort folders alphabetically (for IMAP) according to their
transposed names. For python3, we need to bend a bit backwards to still
allow the use of a cmp() function for foldersort. While going through, I
discovered that we never sort folders for Maildir.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
There is no need to cast 0 to 'long' even if we want to compare it to long
numbers in modern pythons.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This won't work in python3 anymore, so just use sorted() when needed.
In one case, we could remove the sort() completely as were were sanity checking
one line above, that we only having one UID as response which makes sorting
unneeded.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
class BaseRepository(object, CustomConfig.ConfigHelperMixin):
led to TypeError: Cannot create a consistent method resolution
order (MRO) for bases ConfigHelperMixin, object. Switching the inherited
classes helps.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Replace low-level thread.get_ident() with threading.currentThread().ident.
This works both in python2.6 and python3. (thread is renamed _thread and its
direct use is not recommended)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
all we want to know is if we got some string'ish type and testing for isinstance
'basestring' is sufficient for that. Remove the import.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
All we want to do here is to test whether we got a string'ish type or a list
(literal), so testing for basestring will be fine.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
'set' is builtin since python2.6, so remove the imports. Also 'ssl' exists
since 2.6 and has everything we need, so no need for conditional import
tests here anymore.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This import failed in python3, we need to either specify "." (relative) or
from OfflineImap.ui. (absolute). Done the latter.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
To have the code work in python3, we need to convert all occurences of
raise Exception, "text" to be proper functions. This style also adheres to PEP8.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We do not use ui.locked() anymore to output an error message, the text comes
directly from the exception.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Attempt to load first ConfigParser and then configparser. At some point this
should be switched to do the python3 thing first.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We mean the (numeric) logging level here and not the info() function.
logger.isEnabledFor() takes the logging level as argument,
obviously. This was a stupid typo that failed under python3.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
It can lead to potential dataloss (see recent commit log where I added a
scary warning about it to offlineimap.conf).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Blinkenlights does not work well (at all) when using the --info
switch. All we really want here is an output that can be pasted as
debugging information.
Enforce the usage of the "Basic" ui, when the --info switch is used.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Most servers support the UIDPLUS extension, and we don't have to search
headers after each uploaded message. There is no need to CHECK the imap
server after each message when there is no need to search headers.
I have not measured the performance impact on real world servers, but
this lets us do less unneeded work in the common case.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We were not cleaning out possibly existing APPENDUID messages before
APPENDing a new message. In case an old message were still hanging
around, this could *possibly* lead to retrieving and old UID. Things
should have been fine, but we do want to play safe here.
Also, make use of the "official" imaplib2 .response() command rather
than the internal _get_untagged_response() function.
Remove the hack that we would be looking for APPENDUID responses even if
the server claimed not to support the UIDPLUS ext. We now poll server
CAPABILITIES after login, and Gmail does provide us with the UIDPLUS
capability after login.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We used "self" but the paramter was called "s". Fixes a crash when we
ui.warn() (only when using the MachineUI).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
imaputil.imapsplit did not cope with strings that contained encoded
quotation marks, e.g. a folder name '"Make" Magazine' would fail and
crash OfflineImap. Make it work by adapting the regex that we use to
extract the first quote to also work with encoded \" quotes. (We do no
sanity checks that there is an even number of such marks within a string
though)
This commit makes such folders work. This was reported and analyzed by
Mark Eichin.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Conflicts:
Changelog.draft.rst
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.
1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
from the imaplib2 result stack, so subsequent calls would return "None".
As various functions can invoke this, this led to some errors that we
avoid by caching the current UIDVALIDITY value in the Folder instance.
There are more simplifications and improvements to be made.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
commit f2a94af5 introduced the use of time.sleep in ui/TTY.py without
importing it. This caused a regression in 6.5.2, crashing OfflineIMap
when in refresh mode.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We were using super() on a class derived from logging.Formatter() which
worked fine in python 2.7. Apparently python 2.6 uses old-style classes
for this, so the TTYUI broke and crashed OfflineImap. This was
introduced in OLI 6.5.0, I think.
Fix it by calling logging.Formatter.... directly, rather than the
elegant super() (which I happen to like a lot more than is appropriate
in the python world).
Reported by Nik Reiman as github issue 23, should fix that issue.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When deleting many (eg 2000) mails using the SQLITE backend, this takes
a long time durig which OfflineImap can not be aborted via
CTRL-C. Thinking it had frozen permanently, I killed it hard, leaving a
corrupted db journal (which leads to awkwards complaints by OLI on
subsequent starts!). That shows that delete performance is critical and
needs improvement.
We were iterating through the list of messages to delete and deleted
them one-by-one execute()'ing a new SQL Query for each message. This
patch improves the situation by allowing us to use executemany(), which
is -despite still being one SQL query per message- much faster. This is
because rather than performing a commit() after each mail, we now do
only one commit() after all mails have been deleted.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
OfflineIMAP v6.5.2-rc1 (2012-01-09)
===================================
Commits v6.5.1.1 - v6.5.2-rc1:
note: Proper Changelog still in Changelog-draft.rst
d72bb88 Improve error message
3284e01 Revert "use .response() rather _get_untagged_response()"
81f194a mbnames should write out local and not nametransformed box names
7184ec2 Sanity check return value of UIDVALIDTY response
50de217 Allow to pass 'force' arg to selectro() to enforce a new select
ed71805 Changelog entry about "realdelete" option
0a275b9 Add scary warnings about "realdelete" option
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Recently the internal function use of imaplib2's _get_untagged_response()
was switched to use the public documented .response() function (which
should return the same data). However within a few fays we received reports
that both uses of a) the UIDVALIDITY fetching and b) the APPENDUID fetching
returned [None] as data although the IMAP log definitely shows that data
was returned. Revert to using the undocumented internal imaplib2 function,
that seemed to have worked without problems. This needs to be taken up to
the imaplib2 developer.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than to write out the nametrans'lated folder names for mbnames,
we now write out the local untransformed box names. This is generally
what we want. This became relevant since we support nametrans rules on
the local side since only a short time. Reported by Paul Collignan.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We have a reported case where response('UIDVALIDITY') returned [None]
which results in an ugly non-intuitive crash. Sanity check and report
something nicer.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Pass through the 'force' argument from selectro() to select() so that it
can also enforce a new SELECT even if we already are on that folder.
Also change the default parameter from '0' to 'False' to make clear that
this is a Bool.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Add a changelog to W. Trevor King's previous commit. Also make wording a
bit more consistent and and remove a now unneeded comparison (dirname is
always set when extension is set).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
On Tue, Dec 13, 2011 at 12:00:57PM -0500, W. Trevor King wrote:
> I've attached a patch that does fix the problem…
Oops, *now* I've attached the patch and logs ;).
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
From 3067b1b4dfb00d165bd9480ea49f446adb12991d Mon Sep 17 00:00:00 2001
From: W. Trevor King <wking@drexel.edu>
Date: Tue, 13 Dec 2011 11:26:00 -0500
Subject: [PATCH] Only scan children in _getfolders_scandir if extension is set.
When sep is '/', MaildirRepository._getfolders_scandir recursively
checks sub-directories for additional maildirs. The old loop logic
always checked the top directory and its children. This lead to
children being found twice, once from their parent, with dirname
matching their directory name, and once from themselves, with a
dirname of ''.
This patch fixes the problem by only checking the top directory when
extension is not set (i.e. for the root directory).
Do not read in custom maildir flags, or we would try to sync them over
the wire. The next step will be to merge flag writes with existing
custom flags, so we don't lose information.
The long term goal will be to attempt to sync flags to the other side,
of course.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Blinkenlights UI 6.5.0 regression fixes only.
* Sleep led to crash ('abort_signal' not existing)
* Make exit via 'q' key work again cleanly
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
With the new abort signal handler, we can send a signal that lets us
exit cleanly. Make use of this, rather than crashing out in ugly ways.
This affects only the Blinkenlights UI when pressing 'q'.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This is a regression introduced when renaming signals due to the
improved CTRL-C handling. Regression in 6.5.0
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
OfflineIMAP v6.5.1 (2012-01-07) - "Quest for stability"
=======================================================
* Fixed Maildir regression "flagmatchre" not found. (regressed in 6.5.0)
* Have console output go by default to STDOUT and not STDERR (regression
in 6.5.0)
* Fixed MachineUI to urlencode() output lines again, rather than
outputting multi-line items. It's ugly as hell, but it had been that
way for years.
* Remove the old global locking system. We lock only the accounts that
we currently sync, so you can invoke OfflineImap multiple times now as
long as you sync different accounts. This system is compatible with
all releases >= 6.4.0, so don't run older releases simultanous to this
one.
THe new logging framwork spit putput to STDERR by default (as that is
pythons default), but we used to have STDERR, so make it go there again.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We were using the internal imaplib2 _get_untagged_response() functions a
few times. Replace 3 of these calls with 2 calls to the public function
response() rather than fudging with internals that could change anytime.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Remove the old global locking system. We lock only the accounts that we
currently sync, so you can invoke OfflineImap multiple times now as long
as you sync different accounts. This system is compatible with all
releases >= 6.4.0, so don't run older releases simultanous to this one.
This mostly reverts commit 0d95651417,
disabling the old global lock system that we had in parallel to the new one.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The logging rework led to multipline output as we stopped urlencoding
the output lines. Urrg. Fixed this, so output is urlencoded again.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
These were needed for python <2.6 compatability, but since we depend on
python 2.6 now, these can go.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This is a CRITICAL bug fix release for everyone who is on the 6.4.x
series. Please upgrade to avoid potential data loss! The version has
been bumped to 6.5.0, please let everyone know to stay away from 6.5.x!
I am sorry for this.
See details in the Changelog and even more gory details in commit
message for commit 8fc7227189.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This change looks harmless, but it fixes a severe bugfix, potentially
leading to data loss! It fixes the "on n new uploads, it will redownload
n-1, n-2, n-3,... messages during the next syncs" condition, and this is
what happens:
If there are more than one Mails to upload to a server, we do that by
repeatedly invoking folder.IMAP.savemessage(). If the server supports
the UIDPLUS extension we query the resulting UID by doing a:
imapobj._get_untagged_response('APPENDUID', True)
and that is exactly the problem. The "True" part causes the reply to
remain in the "response stack" of the imaplib2 library. When we do
the same call on a subsequent message and the connection is still on the
same folder, we will get the same UID response back (imaplib2 only looks
for the first matching response and returns that). The only time we
clear the response stack, is when the IMAP connection SELECTS a
different folder.
This means that when we upload 10 messages, the IMAP server gives us
always the same UID (that of the first one) back. And trying to write
out 10 different messages with the same UID will confuse OfflineIMAP.
This is the reason why we saw the ongoing UPLOADING/DOWNLOADING behavior
that people reported. And this is the reason why we saw the
inconsistency in the UID mapping in the IMAP<->IMAP case.
I urge everyone to upgrade ASAP. Sorry for that, I don't know why the
problem only became prevalent in the recent few releases as this code
has been there for quite a while.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than always parsing the filename, we only need to do so if the flags
have actually changed, otherwise we can keep the filename.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, assigning a new UID to a mapped IMAP or Maildir repository
was done by loading the "local" item, saving it under a new UID and
deleting the old one. This involved lots of disk activity for nothing
more than an effective file rename in Maildirs, and lots of network
usage in the MappedUID cases.
We do this on every upload from a local to a remote item, so that can
potentially be quite expensive. This patch lets backends that support it
(Maildir, MappedUID) efficiently rename the file rather than having to
read the mail content, write it out as a new file and delete the old
file. This speeds up uploads from Maildir and the MappedUID server.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Various functions (such as change_message_uid) will want to construct
maildir filenames, so factor out the code into a helper.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Create a helper function that retrieves the UID, folder MD5, and Flags from
a message filename.
We need these items when we simply want to rename (=new UID) a Maildir
message file later. The new function can give us these components.
Rework, so we cache the calculation of the folder's md5 value once, it
never changes and we call it a lot.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If someone had a custom :2,a flag, adding a new flag would lead to the
invalid maildir filename ...a:2,... due to regex deficiencies not coping
with this. Fix this so we alway produce valid maildir names.
Note that custom flags are still problematic: as the syncing to the
remote IMAP server will fail, the next sync will assume that they have
been removed from the remote IMAP side and they will be removed from the
local Maildir then. We will need to think about how to handle this. At
least, with this patch we won't lose standard flags and won't produce
invalid maildir names.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The regex for catching Maildir message flags was
self.infosep + '.*2,([A-Z]+)' (infosep being ':').
The .* is bogus, as there is nothing between the : and the 2, per
maildir name specification, so remove that unneeded piece.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When a new remote folder was detected, we tried to create the folder
locally on the Maildir and called repository.forgetfolders() to force a
new scanning of the Maildir. However, that implementation used the
inherited base function that did nothing. We simply needed to implement
forgetfolders() to set self.folder=None, so we would force a new read in
of the updated local folder structure.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
1) Rename the unintuitive repository.syncfoldersto() to
sync_folder_structure()
2) We were checking if the local repository is readonly and then turning
off any folder creation. But as we can create folders on a remote
repository too, we need to be more fine grained here. Just don't create
a folder on the repository that is marked readonly=True.
This still does not do away with the error message that one currently
gets on missing local folders.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The quiet UI should only output errors, and the final "Finished account
X in 2 seconds" clearly is none, so the message debug level needed to be
reduced to INFO to suppress it in the quiet ui.
Fixes https://github.com/spaetz/offlineimap/issues/6
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Merge in the new logging mechanism, and provide an --info feature that
will help with debugging. There is still some unstableness, so there
will be another release soon, but this should be no worse than 6.4.2 at
least...
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, we would simply bail out in an ugly way, potentially leaving
temporary files around etc, or while writing status files. Hand SIGINT
and SIGTERM as an event to the Account class, and make that bail out
cleanly at predefined points. Stopping on ctrl-c can take a few seconds
(it will e.g. finish to transfer the ongoing message), but it will shut
down cleanly.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
As reported in https://github.com/spaetz/offlineimap/pull/2, we would
fail when files are empty because file.read() would throw attribute
errors.
Fix this by removing the superfluous read() check and additionally log
some warning message.
Reported-by: Ralf Schmitt
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When sep='/' in a Maildir, we were doing a os.path.join(dirname,'') on
the top level maildir, which results in a "dirname/", so all our maildir
folder names had slashes appended. Which is pretty much wrong, so this
fixes it by only using os.path.join when we actually have something to
append.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We need the list of folders and the folder delimiter, but it was not
always retrieved early enough. E.g. when doing IMAP<->IMAP sync and the
local IMAP being readonly, we would bunk out with a mysterious error
message become repository.getsel() would still return None.
This commit fixes this error.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
imapserver.getdelim() was not used at all, so remove this function. The
folder delimiter is available via the repository.getsep() call.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The folder delimiter is only initialized after a call to
acquireconnection(), so we must never call this function too
early. Include an assert() to make sure we get notified when we do.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than keeping a separate queue of all logged lines in memory, we
rely on the curses window scrolling functionality to scroll lines. On
resizing the terminal this means, we'll clear the screen and start
filling it afresh, but that should be acceptable.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
1) Rework the sleep abort request to set the skipsleep configuration
setting that the sleep() code checks.
2) Only output 15 rather than 50 debug messages on abort...
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The remote|local|statusrepo is an anttribute of each SyncableAccount()
anyway, so we don't need to pass it in, we can simply get it from the
Account().
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rename some variables, simplify the hotkeys treatment. Refresh/exit
signals still don't work as of yet, but will come.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Resizing a Blinkenlights terminal doesn't crash anymore, and actually
seems to be changing the size, with this patch.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We were still referring to s.gettf() in sleeping(self, ...) causing each
attempt to sleep to crash. Fix this, and the CursesAccountFrame.sleeping()
method. I am sure, there is still wrong and broken but we are getting there.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This function can IMHO lead to possible deadlocks when waiting for the
connectionlock. Do add a comment to that regard, this will need to audit.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The huge UI rework patch removed some obscure logic and special handling of
thread exit messages. It turns out that this was in fact still needed as a
specific exit message of the SyncRunner thread signified the threatmonitor
to quit.
We will want a nicer machinery for this in the future I guess, but fix the
eternal hang on exit by reintroducing a special exit message for the
SyncRunner thread, and return from the infinite monitor loop if SyncRunner
finishes.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
To make sure, the lock gets released even if we raise an exception between
acquire() and release()
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
All ExitNotifyThreads and InstanceLimitThreads are setDaemon(True) in their
constructor, so there is no need to do that again in the code.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
1) Differentiate error messages between imaplib.abort and imaplib.error
exceptions in the log.
2) Drop connections in the case of imapobj.error, it also might denote a
broken connection.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
During cleanup we often call releaseconnection in a finally: block. But
in cases of error, we might have dropped the connection earlier already
and set it to "None". In this case don't fail releaseconnection() but
make it a NOOP.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The default parameter value was "None", and we were comparing that
directly to the imaplib2 value of is_readonly which is False or True, so
the comparison always returned "False".
Fix this by setting the default parameter to "False" and not
"None". Also convert all users of that function to use False/True.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
It is basically unused by now. Rework to be able to make use of it
later, no functional changes.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Logging was flawed as the output was e.g. heavily buffered and people
complained about missing log entries. Fix this by making use of the
standard logging facilities that offlineimap offers.
This is one big ugly patch that does many things. It fixes the
Blinkenlights backend to work again with the logging facilities.
Resize windows and hotkeys are still not handled absolut correctly, this
is left for future fixing. THe rest of the backends should be working fine.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If folder creation failed, we would output the wrong repository and
folder name (copy'n paste error). Fix this so we actually output the
correct values.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Only import the lock, that we actually need. Also import the with statement
for use with python 2.5. We'll need it for sure in this file.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Drop a connection, if the NOOP to keep a connection open fails due to
broken connections.
Note that I believe this function is not working as intended. We grab
one random connection and send a NOOP. This is not enough to keep all
connections open, and if we invoke this function multiple times, we
might well always get the same connection to send a NOOP through.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
A repositories 'reference value is always prefixed to the full folder
path, so we should do so when creating a new one. The code had existed
but was commented out since 2003, I guess the "reference" option is not
too often used.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If port is None, we would try to format an empty string with %d wich
fails. Fix it by using %s.
Reported-by: Iain Dalton <iain.dalton@gmail.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This outputs a handy summary of your server configuration and version
strings etc, which is useful for bug reporting.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Was getting too large, split into an parse_cmd_options and a sync()
function. Moving config and ui to self.config and self.ui to make them
available through the OfflineImap instance.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
It happens quick, and clutters the log. So we can usually skip this. We
will output a log entry when we actually create a new folder anyway.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If we create a new folder we would previously not update our folder
list, which led to us skipping the synchronization of those new folders
during the initial run (subsequent runs would pick it up).
Invalidate the folder cache when we create a folder during folder
structure sync. Regetting the whole list from an IMAP server might be
slightly suboptimal from a performance point, but it is easy and will
lead to consistent results. Hopefully we will not have to create new
folders on each new run.
Reported-by: Daniel Shahaf <d.s@daniel.shahaf.name>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Modify the UI:acct and acctdone functions to keep tab of the time
inbetween. Put self.ui.acct() and acctdone() at the right places in
accounts.py so that the timing happens at the right places.
While modifying that loop, flatten the nested try: try: except: finally:
constructs, we require python 2.5 now which copes with that.
At the end of each account sync you will now see something like:
*** Finished account 'test' in 0:05
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Thread names are used to determine the logging header in the TTY ui. A
recent change made them too terse (basically only changing the account
name and not the folder names). Unbreak.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Output (2 of 500) when logging message copying. This required moving of
self.ui.copyingmessage into a different function where we actually have
the information about the progress handy.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than setting a global threadutil/profiledir variable, we make
set_profiledir a class function that sets the class variable profiledir.
While touching theprofiledir code, add warning to the user if the
profile directory existed before.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Registering a thread (associating it with a certain account name) would
fail if it was already registered. However, as we a) never unregister most
threads (bad) and b) single-threaded mode reuses threads, we failed when
syncing multiple accounts in single-threading mode.
This commit cleans up the functions to not make re-registering a thread
fatal (it could be legitimate, however it *should* not occur). Future
work needs to be done to unregister new threads at the appropriate places.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When checking for the IMAP4.abort() exception, we need of course to
perform:
except imapobj.abort:
and not
except imapobj.abort():
Thanks to Johannes Stezenbach <js@sig21.net> for pointing to the glitch.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
apply() has been deprecated since Python 2.3, and won't be working in
python 3 anymore. Use the functional equivalent throughout.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Dave identified a case where our new dropped connection handling did
not work out correctly: we use the retry_left variable to signify
success (0=success if no exception occured).
However, we were decrementing the variable AFTER all the exception
checks, so if there was one due to a dropped connection, it
could well be that we 1) did not raise an exception (because we want to
retry), and 2) then DECREMENTED retry_left, which indicated "all is
well, no need to retry".
The code then continued to check() the append, which failed with the
above message (because we obtained a new connection which had not even
selected the current folder and we were still in mode AUTH). The fix is
of course, to fix our logic: Decrement retry_left first, THEN decide
whether to raise() (retry_left==0) or retry (retry_left>0) which would
then correctly attempt another loop. I am sorry for this newbie type of
logic error. The retry count loop was too hastily slipped in, it seems.
Reported-by: Dave Abrahams <dave@boostpro.com>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If APPEND raises abort(), the (typ, dat) variables will not be set, so
we should not be using it for the OfflineImapError Exception
string. Fixing and prettifying the string formatting a bit at the same
time.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When syncfolder() fails, we output an error message containing the
foldername per the localfolder variable. However, the localfolder
variable is assigned inside our try: block and when the error occurs
there, we will have no localfolder variable to use for output. This
caused the errormsg to cause an Exception itself which unhelpfully
distracts from the root cause of the error.
Reconstruct the folder name in a bit more complex way, but in a way so
it is guaranteed to work (by relying on parameters passed in to the
function).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
A report by Dave Abrahams showed that the dequote() function failed when
invoked with an empty string. This fixes the function to be more robust.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
nametrans rules can lead to different visiblename names for the
top-level directory, specifically both '.' and '' (the latter was
recently introduced). However, we need to be able to compare folder
names to see if we need to create a new directory or whether a directory
already exists, so we need to be able to compare a repositories
visiblename (=transposed via nametrans rule) with another folder.
To make the top-level directory comparison happen, we enforce a
top-level name of '', so that comparisons work.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Some Webservers (I am looking at you Gmail) send different capabilities
before and after login, so they can tailor their server capabilities to
the user. While legal, this is uncommon and we were not updating our
server capabilities. Doing so allows us to detect that Gmail actually
supports the UIDPLUS extension, and we will stop mangling headers when
uploading to Gmail. This could lead to some performance gains when we
upload many messages to Gmail.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Commit b0e88622c4 changed dst_hash[folder.visiblename] to
dst_hash[folder.name] but we did not adapt all places where it is needed
to use visiblename again. This led to attempting to create a name on
REMOTE ignoring the nametrans setting on the LOCAL repository.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, we only checked if a LOCAL folder falls under the local
repositories folderfilter rule when deciding whether a folder should be
created on REMOTE.
However, we also do not want to create the folder on REMOTE if it would
fall under a folderfilter rule there. This patch prevents us from doing
so.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
It is not easy to think through when to use visiblenames() and whatnot. It seems I managed to not think it through properly. Which might be fixed now.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The thread ID is not really useful and looks ugly. It also makes lines
longer than needed, there is more useful information we can put in the
log. So do away with it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This will ignore any nametrans rules, so we might want to limit this
only to cases where no nametrans has been specified, or we might want to
use the nametrans setting of the dest repo.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
getvisiblename() was only defined on IMAP(derived) foldertypes, but we
want it on eg. Maildirs too, so we define it centrally in Folder.Base.py
rather than only in folder.IMAP.py.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Currently we only filtered IMAP repositories, this patch enables filtering
for Maildir repositories too.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We want to have these functions available for Maildir folders too, so we
can folderfilter a Maildir repository too (which is currently not possible)
This commit only move the corresponding functions from the IMAP to the Base
implementation. It should not change behavior in any way yet.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Also remove the removed parameters in the Gmail folder
initialization. This is one spot where I had forgotten to also strip the
parameters.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
If nametrans translates to an empty directory we want to find the
top-level directory by name '' and not by name '.'. This unbreaks
nametrans rules that result in empty folder names.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, getfolder() would always construct new MaildirFolder()
objects, independent of whether the folder exists or not. Improve the
function to:
1) Scan and cache the folders if not already done
2) Return the same cached object if we ask for the same foldername twice
3) Reduce a tiny bit of code duplication
This is important because we handle stuff like folderfilter in the
scandir function and if we discard the scanned dir and create a new
object on folderget(), we will lose the folderfilter information.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Output a debug log line whenever we create a new folder on an IMAP
server. Also raise an OfflineImap Error in case we failed to create it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This variable shows if this folder should be synced or is disabled due to
a folderfilter statement. This lets us distinguish between a non-existent
folder and one that has been filtered out. Previously any filtered folder
would simply appear to be non-existing.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The Message UID is already the key to self.messagelist, so we have that
information. It is redundant to save the UID again as
self.messagelist[uid]{'uid': uid} and we never made use of the
information anyway.
The same thing should be done with the other 2 backends.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
A more pythonic and less verbose way to do the same. Add a comment what the
variable is all about.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Warn the user and abort when we attempt a plaintext login, but the
server has explicitly disabled plaintext logins.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Use the ui.error infrastructure that has been put in place and use
ui.terminate even if we received an Exception, so that we can output the
list of errors that we have. This does away with 2 now unused functions
in ui/UIBase.py
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We simply lock OfflineImap the same global way that we have always done
in addition to the previously implemented per-account lock. We can keep
both systems in parallel and then after a few stable releases, drop the
old-style global lock. by reverting this patch
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The next commit will make use of OfflineImapError but is transient (the
old-style lock). The commit is supposed to be reverted after a few
releases. So add the new import in a separate commit, because we might
need this even when reverting the commit.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously, we were simply locking offlineimap whenever it was
running. Howver there is no reason why we shouldn't be able to invoke it
in parallel, e.g. to synchronize several accounts in one offlineimap
each.
This patch implements the locking per-account, so that it is possible to
sync different accounts at the same time. If in refresh mode, we will
attempt to loop three times before giving up.
This also fixes Debian bug #586655
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
open() and os.open() lead to different file permissions by default, and
while we have not changed the os.open that had been used, some code
changes led to these permissions slipping through. Fix this by setting
the permissions explicitly to 0666 (minus the users umask).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
repos.getuesr() asks for a username if none is specified, but in the
case of a tunnel connection, we don't need one, so we need to skip the
repos.getuser() call here.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
IMAPFolder has the repository and foldername values so it can get the
transposed (aka visiblename) of a folder itself just fine. There is no
need to pass it in as an separate parameter.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
They have the Repository() which contains the root, so no need to pass
it in as an extra parameter. Rename repository.LocalStatus()'s
self.directory to self.root for consistency with other backends.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It is possible to get the config parameter from the Repository() which is
set in BaseFolder, so we set self.config there and remove the various
methods and 'config' parameters that are superfluous.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We passed in the accountname to all derivatives of BaseFolder, such as
IMAPFolder(...,repository,...,accountname), although it is perfectly
possible to get the accountname from the Repository(). So remove this
unneeded parameter. Each backend had to define getaccountname() (although
the function is hardly used and most accessed .accountname directly).
On the other hand BaseFolder was using getaccountname but it never defined
the function. So make the sane thing, remove all definitions from backends
and define accountname() once in Basefolder. It was made a property and not
just a (public) attribute, so it will show up in our developer
documentation as public API.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
As all Folders share these parameters, we can safely handle them in
BaseFolder. This makes sense, as BaseFolder has a getname() function
that returns self.name but nothing actually set self.name.
It also saves a few lines of code.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In getmessage() we were releaseing a connection when we detected a
dropped connection, but it turns out that this was not enough, we need
to explicitely discard it when we detect a dropped one. So add the
drop_conn=True parameter that was recently introduced to force the
discarding of the dead conection.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The quickchanged() function was not handling dropped connections yet. If
IMAP4.select() throws a FOLDER_RETRY error, we will now discard the
connection, reconnect and retry.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Beauty of code is probably a subjective measure, but this patch hopefully
is an improvement over the previous incarnation without changing
functionality.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
repository.BaseRepository().restore_atime() was testing in complex ways
that it only operates on a Maildir and that the 'restoreatime' setting
is set. This is unecessary, we can simply make the base implementation a
NoOp, and move the implementation to MaildirRepository().
This will save a tad of work for everyone doing IMAP<->IMAP
synchronization and simplify the code. Also document the functions.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We only explicitly tested for 'yes' when we have a nice function to get
boolean settings which also works with Treu/False/NO, etc...
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
The readonly feature was introduced to safeguard repositories from
accidental modifications. Unfortunately, my patch treated the readonly
setting as a string and not as a boolean, so if *anything* was set in
the configuration file as 'readonly', this value evaluated to True
Fortunately this was safe, we never treated a repository that we wanted
read-only as read-write. We always treated them readonly if something
was configured as "readonly=..." even if that was False.
The fix is simply to use getconfboolean() rather than getconf() which
checks for True/False/On/Off/yes/no/1/0 and hands back the correct boolean.
Reported-by: Tomasz Nowak <nowak2000@poczta.onet.pl>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We rely on the number of mails being returned by the imapobj.select()
call, however that only happens if we "force" a real select() to occur.
Pass in the force parameter that I dropped earlier (we did not make use
of the return value when I dropped it, that is how it slipped through).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were retrying indefinitely on imapobj.abort() (as that is what
imaplib2 suggests), but if the failure occurs repeatedly, we'll never
quit this loop. So implement a counter that errs out after unsuccessful
retries.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Finally, actually discard dropped connections when we detect them as an
imapobj.abort() has been thrown. In this case, invoke releaseconnection
with drop_conn=True.
We don't need the self.aborted attribute to get signified of dropped
connections. An Execption during the noop will do.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
while True: if a: return
is equivalent to
while not a:
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Variable name 'event' is as bad as it gets. Rename it to something that
actually describes what it is about.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Add code documentation throughout the idle() function.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Don't redefine the idle callback function on every run in the while
loop, define it once when we enter the function.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Python 2.5 has no ssl module, and we can therefor not get the server
certificate for fingerprint verification. Add a check that disables
fingerprint verification for python 2.5.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
imaplib2 has changed internally to use self.sock for its ssl socket when
it used to be sslobj. Reflect that change.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If we connect to a SSL server (not STARTTLS) and no CA cert has been
specified for verification, we check the configured SSL fingerprint and
bail out in case it has not been set yet, or it does not match.
This means one more mandatory option for SSL configuration, but it
improves security a lot.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If maxage is set too large, we would even SEARCH for negative
years. With devastating results. So implement some sanity check and err
out in case the year does not make sense.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We can use Imaplib's monthnames and shorten the construction of the date
by using them rather than hardcoding them again.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Results are delivered in a 1-element list, and somehow I managed to drop
a [0] in the previous patches. We need to look at the element of course,
or our string splitting will fail horribly. Sorry this somehow slipped
through.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If a folder is empty, most servers will return EXISTS 0 and imaplib2
passes back ['0'] as return value to a select(). It returns [None] if
no EXISTS response was given by the server at all.
Attempting to fetch the UIDs of 0 emails which leads to
various error messages (One server responds with "NO No matching
messages", Gmail seems to say "BAD Bad message sequence 1:*" for some
(although it is working fine for me with Gmail, so it might behave
different for different people).
In case we get an None or 0 back, we simply stop caching messages as the
folder is empty. This should fix the various error reports that have
popped up.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make sure that when a connection is dropped during append, we really
discard the broken connection and get a new one, retrying. We retry
indefinitely on the specific abort() Exception, as this is what imaplib2
suggests us to do.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If a connection is broken, we want to have it really dropped and not be
reused. So far, we are checking the .Terminate attribute for this, but
according to the imaplib2 author, it is only set on normal shutdown and
it is an undocumented attribute whose meaning could change any time.
This patch introduces the parameter drop_conn which allows to tell
releaseconnection() that we really want to connection being dropped from
the pool of available connections and properly destroy it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than output the full list of messages, coalesce it into number
ranges wherever possible.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This function was badly named and completely undocumented. Rework it to
avoid copying the full UID list using an iterator. Make it possible to
hand it a list of UIDs as strings rather than implicitely relying on the
fact that they are numeric already. Document the code.
The behavior off the function itself remained otherwise unchanged.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
SEARCH and FETCH were never checking that the IMAP server actually
returned OK. Throw OfflineImapErrors at severity FOLDER in case one of
them fails.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Some code cleanup. If we want to examine all messages of a folder, don't
try to find out how many there are and request a long list of all of them,
but simply request 1:*. This obliviates us from the need to force a select
even if we already had the folder selected and it requires us to send a
few less bytes over the wire.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Do away with the wrapping of this code in a try...except KeyError, as
this code cannot conceivably throw a KeyError. Even if it could, it
should be documented why we should simply return() in this case.
Shorten some of the variable names and minor code cleanup while taking
the git blame anyway.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than passing in huge lists of continuous numbers which eventually
overflow the maximum command line length, we coalesce number ranges
before passing the UID sequence to SEARCH. This should do away with the
error that has been reported with busy mailing lists and 'maxage'.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It is not needed. list(ALIST) will create a new copy of the list just
fine.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Make sure that when a connection is dropped during append, we really
discard the broken connection and get a new one, retrying. We retry
indefinitely on the specific abort() Exception, as this is what imaplib2
suggests us to do.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If a connection is broken, we want to have it really dropped and not be
reused. So far, we are checking the .Terminate attribute for this, but
according to the imaplib2 author, it is only set on normal shutdown and
it is an undocumented attribute whose meaning could change any time.
This patch introduces the parameter drop_conn which allows to tell
releaseconnection() that we really want to connection being dropped from
the pool of available connections and properly destroy it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
When invoked with FETCH 1:* (UID), imaplib returns [None] for empty
folders. We need to protect against this case and simply 'continue' here.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
If a connection is dropped for some reason while fetching a message, the
imapobj.uid command throws an imapbj.abort() Exception which means we are
supposed to retry. Implement a fail loop that drops the connection, gets a
new one and attempts the command another time.
Remove obsolete comment that we need to catch nonexisting messages. We do
now.
GMail seems to drop connections left and right. This patch is a response to
the reported mail "4E5F8D8C.1020005@gmail.com" by zeek
<ezekiel.das@gmail.com>.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Handle the case gracefully where a server has closed an IMAP connection
that we want to use for IDLEing. Simply have it dropped and get a new one
in this case. THis should get rid of the errors reported by John Wiegley
in mail id:"m2sjohd16t.fsf@gmail.com".
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Imapserver.acquireconnection will succeed even whent the server connection
has been terminated and the first IMAP operation will throw an exception.
Often this is the folder SELECT operation (e.g. after an idle timeout), as
has been reported by John Wiegley. Catch this case and throw an
OfflineImapError with severity FOLDER_RETRY to notify consumers that they
are supposed to retry.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
In cases where processing a folder failed, but a retry might well succeed
e.g. when the server connection had simply timed out and was disconnected,
we can throw a FOLDER_RETRY (which is less severe than FOLDER).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Name parameter that hands us a Status Repository 'status_repo' and not
'copyfolders' as was before:
a) make it clear that we pass in a repository and not folder
instances. That was very confusing before.
b) We were always only using one 'copyfolders' item anyway, so let us
not make it a list.
Go through the list and make the variable nameing consistent:
dst_repo rather than dest (is it a folder?) to match the status_repo
naming scheme.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
To preserve previous behavior and save a few CPU cycles, we were not sorting UID
lists and only collapsed them if they were alreay sorted. Vincent pointed out
that this is not always the case and unsorted lists lead to non-optimally
collapsing.
Force lists to numeric types and sort them before collapsing.
Reported-by: Beffara <vbeffara@ens-lyon.fr>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were using self.getfolderbasename(self.name) but the API is simply
getfolderbasename(). Fix this glitch.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
SafeConfigParser is very similar to the currently used ConfigParser but
it supports interpolation. This means values can contain format strings
which refer to other values in the same section, or values in a special
DEFAULT section. For example:
[My Section]
foodir: %(dir)s/whatever
dir=frob
would resolve the %(dir)s to the value of dir (frob in this case). All reference expansions are done on demand.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
as that method doesn't exist on sets. Rather call the inbuilt
sorted(flags). This fixes Exceptions being thrown when using the sqlite
backend.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We were omitting an '%' where we needed it. Also include the traceback
information where it belongs in the new ui.error infrastructure.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
443 is of course the https and not the IMAPS standard port. Fix.
Thanks to Daniel Shahaf <d.s@daniel.shahaf.name> for the heads up.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Commit e023f190b0 changed the storing of
file paths in the messagelist variable to be relative paths, but we were
using the full absolute path anyway as we missed one spot.
Adapt this and construct the full file path in the one place where we
need it.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Rather than output the full list of messages, coalesce it into number
ranges wherever possible.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
This function was badly named and completely undocumented. Rework it to
avoid copying the full UID list using an iterator. Make it possible to
hand it a list of UIDs as strings rather than implicitely relying on the
fact that they are numeric already. Document the code.
The behavior off the function itself remained otherwise unchanged.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
As this is essentially what it is, a set of values. This allows as
to do set arithmetics to see, e.g. the intersection of 2 flag sets
rather than clunkily having to do:
for flag in newflags:
if flag not in oldflags:
oldflags.append(flag)
Also some more code documenting.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Empty foldernames (as they could be created through nametrans) were
failing as the uidvalidity and status files names as determined by
folder/Base.py:getfolderbasename() lead to invalid file names ''.
Fix this by handling empty file names and translating them to '.' which
leads to the special file name 'dot'. (this special value existed before
and was not invented by this patch)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
I changed the API to pass in a folder rather than a list of folders, but
used getnicename() on the wrong object. It is not used on the folder but
on the ui object. Fix this and give the variable somewhat better names.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
It works by fetching all headers of new messages from IMAP server and
searching for our X-OfflineIMAP marker by using regular expression.
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
We only copy to a single folder anyway, so clean up the code to only
pass in a single folder.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
imaplib2 always attempts to verify a certificate if a verification
callback function is passed in, even the certificate is None
specified. Disable the verification excplictly by setting the
verification function to None in that case.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
we need errno.CONNREFUSED, but through some merging mishaps(?) the part
that actually imported errno was missing. Import the errno module.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
imaplib renamed self.sslobj to self.sock and our overriden open()
functions were failing for that reason when updating imaplib2 to
v2.28. It turns out that all of our custom initializations are being
done by stock imaplib2 now anyway, so there is no need to override them
anymore. This lets us simplify the code we have to worry about.
Move the verifycert() function to the imapserver.py file, it is now a
callback function that is being handed to imaplib from there, so it
makes sense to also define it in our imapserver function...
(this also lets us easily make use of the verifycert function in the
starttls case in the future)
TODO: we need to examine if and why we still need to override the
select() function, it is the only reason why we still wrap the IMAP4
classes.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
Imaplib2 2.28 can deal with ID sequences, such as 1:*, so we need to
bump upstream in order to make use of these features.
Note that this revision will not run correctly as it requires
adaptations to our code, which happens in the next commit.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
