Commit Graph

105 Commits

Author SHA1 Message Date
Ben Boeckel
4217fccb82 mbnames: add option to write the file once per account
The basic problem is in the context of syncing multiple accounts where
one is fast and the others are slower (due to the number of folders).
When the fast account completes, the other accounts are partially written
through the list and if the file is read during this time, the list can
be useless. However, in the general case, the file is probably left
around from a previous run of offlineimap and is more correct, so add an
option to leave it alone until all syncing is done.

Incremental is still the default since this running offlineimap using
its own timer setup is likely the most common setup. Turning it off
works best with one-shot mode triggered by cron or systemd timers.

Signed-off-by: Ben Boeckel <mathstuf@gmail.com>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-03-22 21:15:46 +01:00
Nicolas Sebrecht
461554b7b1 more style consistency
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-02-10 17:25:00 +01:00
Nicolas Sebrecht
41fa3ae4fc more style consistency
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-13 18:16:25 +01:00
Giovanni Mascellani
29e9b7ab39 Drop caches after having processed folders.
This enhances a lot memory consumption when you have many folders
to process.

Signed-off-by: Giovanni Mascellani <mascellani@poisson.phc.unipi.it>
2015-01-12 17:17:42 +01:00
Nicolas Sebrecht
63e499c6f2 more consistent style
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-12 17:17:42 +01:00
Nicolas Sebrecht
a44718130d minor: add comments
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-12 17:17:42 +01:00
Eygene Ryabinkin
e7fabf9e57 Properly re-raise exception to save original tracebacks
We usually mutate some exceptions to OfflineImapError() and it is
a whole lot better if such exception will show up with the original
traceback, so all valid occurrences of such mutations were transformed
to the 3-tuple form of "raise".  Had also added coding guidelines
document where this re-raise strategy is documented.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2015-01-12 17:51:21 +03:00
Nicolas Sebrecht
0f40ca4799 more style consistency
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-08 23:36:19 +01:00
Nicolas Sebrecht
61021260cb more consistent style
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2015-01-07 21:31:43 +01:00
Eygene Ryabinkin
a1bf8db517 Brought CustomConfig.py into more proper shape
- Multi-line documentation for functions and methods
   now has ending triple-double-quotes on an own line,
   as per PEP 257.

 - Added documentation and comments to almost all functions
   and methods.

 - Added stub implementations for getconfig() and getsection()
   inside CustomConfig.ConfigHelperMixin to provide sane
   run-time diagnostics for classes that doesn't implement them.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-10-09 19:24:11 +04:00
Eygene Ryabinkin
968ffc20bd More clearly show results of folder name translation
For cases like
  http://article.gmane.org/gmane.mail.imap.offlineimap.general/6468
it is beneficial to see that folder name was translated and the result
of this translation on a single line: having log like
{{{
Folder Boring/Wreck [acc: tmarble@info9.net]:
 Syncing Boring/Breck: Gmail -> Maildir
}}}
with translated name on the "Folder" line and original one on the
"Syncing" line isn't very intuitive.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-06-24 18:48:58 +04:00
Eygene Ryabinkin
6cbd2498ae Refactoring: make functions to be private if we can
Make external API of class/module to be smaller, explicitely mark
all internal functions.  Also annotate methods that are implemented
as the part of the parent class interface.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2014-04-30 19:35:56 +04:00
Eygene Ryabinkin
95aea5e489 Add new expansion key for mbnames.peritem config variable
It is called localfolders and holds expanded name for the same
variable for the local repository of the account that is being
processed.

GitHub issue: https://github.com/OfflineIMAP/offlineimap/issues/21
Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-08-27 20:25:23 +04:00
Eygene Ryabinkin
f4140cbbed Create global instance of command-line options
This eases testing of option values inside the code.  This instance
is implemented as the read-only copy of the obtained 'options' object,
so callers won't be able to modify its contents.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
2013-02-11 23:34:18 +04:00
X-Ryl669
5c842c01bd Fix deadlock for IMAP folder synced in singlethreaded mode
The problem lies in the fact that offlineimap.folder.Base's method
syncmessagesto_copy() uses threaded code everytime it is suggested by
the derived class's suggeststhreads() (currently, only IMAP does this
suggestion), but offlineimap/init.py will not spawn the
exitnotifymonitorloop() from offlineimap.threadutil.

The root cause is that ExitNotifyThread-derived threads need
offlineimap.threadutil's exitnotifymonitorloop() to be running the
cleaner for the exitthreads Queue(), because it fills the queue via
the run() method from this class: it wants to put() itself to the
Queue on exit, so when no exitnotifymonitorloop() is running, the
queue will fill up.  And if this thread is an instance of
InstanceLimitedThread that hits the limit on the number of threads,
then it will hold the instancelimitedsems[] semaphore will prevent
other InstanceLimitedThread()s of the same name to pass its start()
method.

The fix is to avoid using threaded code if we're running
single-threaded.

Signed-off-by: Eygene Ryabinkin <rea@codelabs.ru>
Obtained-from: X-Ryl669 <boite.pour.spam@gmail.com>
2013-02-05 07:53:25 +04:00
Sebastian Spaeth
3476e9ab36 Fix fallout when filtering folders
Previous commit e7ca5b25cb combined
checks for filtered folders in one place. However, it turns out there
was a reason to have them separate. getfolder() on a non-existent Maildir
fails and there might not be an equivalent local Maildir folder for a
filtered out IMAP folder.

Fix this by first checking if the remote folder should be filtered, and
only then retrieving the local folder (which should exist then).

This bug was found by our test suite!

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-09-01 03:15:08 +02:00
Dave Abrahams
e7ca5b25cb Combine checks for ignored folders in one place
- Factor out the code to find a local folder given a remote folder

Patch by Dave, split and modified by Sebastian.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-09-01 02:16:06 +02:00
Sebastian Spaeth
1b54b85f20 Reuse LocalStatus() folders rather than recreate instances
If we ask twice for a LocalStatusFolder via getfolder(), we would
get a newly created instance each time. This can lead to problems,
as e.g. write locks protecting files only work within the same Folder
instance. Make it so, that we cache all Folder instances that we have
asked for and hand back the existing one if we ask again for it,
rather than recreate a new instance.

Also, make getfolders() a noop for LocalStatus. We attempted to
derive the foldername from the name of the LocalStatusfile. However,
this is not really possible, as we do file name mangling
(".$" -> "dot", "/" -> ".") and there is no way to get the original folder
name from the LocalStatus file name anyway.

This commit could potentially solve the "file not found" errors, that people
have been seeing with their LocalStatusCache files. If we have 2
instances of a LocalStatusFolder pointing to the same file, our locking
system would not work.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-09-01 01:19:19 +02:00
Sebastian Spaeth
04ffae2f00 Exit with nice error message failing to get repositories
If we throw an OfflineImapError in case of the Repository()
initialization, we display the nice error message and exit rather
than bomb out with a traceback. Misconfiguring a repository name in
the configuration file is now nicely pointed out to the user.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-08-31 20:34:54 +02:00
Sebastian Spaeth
33f55b5362 Implement dry-run on Account() level
1) Set attribute self.dryrun depending on whether we are in dry-run mode.
2) Don't actually call hooks in --dry-run (just log what you would
   invoke
3) Don't write out the mbnames file in --dry-run mode.

Repository, and Folder levels still need to be protected in dry-run mode
as of now.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-17 10:53:10 +01:00
Sebastian Spaeth
9df7f34d4c Remove unused locked() function
We do not use ui.locked() anymore to output an error message, the text comes
directly from the exception.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
55da31c84b octal notation 0700 -> 0o700
Use octal notation that python3 understands. Works >=python2.6

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
0844d27f9f except Ex, e: --> except Ex as e:
Nudge us towards python3 compatability by converting deprecated python2 syntax.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-02-06 17:41:42 +01:00
Sebastian Spaeth
de1e03b065 Tweak code documentation
Minor comment changes

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
d3c693be7d Rename folder.saveuidvalidity()
Simply rename to folder.save_uidvalidity()

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
eb616da602 Rename folder.isuidvalidityok()
Use nicer name folder.check_uidvalidity() and improve code documentation

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:03:03 +01:00
Sebastian Spaeth
9453e1d955 Fix getuidvalidity crash (UIDVALIDITY returning None)
Rename getuidvalidity -> get_uidvalidity and cache the IMAP result.

1) Start modernizing our function names using more underscores
2) IMAPs implementation of get_uidvalidity was removing the UIDVALIDITY result
   from the imaplib2 result stack, so subsequent calls would return "None".
   As various functions can invoke this, this led to some errors that we
   avoid by caching the current UIDVALIDITY value in the Folder instance.

There are more simplifications and improvements to be made.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-19 12:02:13 +01:00
Sebastian Spaeth
81f194adca mbnames should write out local and not nametransformed box names
Rather than to write out the nametrans'lated folder names for mbnames,
we now write out the local untransformed box names. This is generally
what we want. This became relevant since we support nametrans rules on
the local side since only a short time. Reported by Paul Collignan.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-08 12:48:21 +01:00
Sebastian Spaeth
1b85e35256 Disable old global lock system
Remove the old global locking system. We lock only the accounts that we
currently sync, so you can invoke OfflineImap multiple times now as long
as you sync different accounts. This system is compatible with all
releases >= 6.4.0, so don't run older releases simultanous to this one.

This mostly reverts commit 0d95651417,
disabling the old global lock system that we had in parallel to the new one.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-06 23:54:28 +01:00
Sebastian Spaeth
3e28073f98 Do not create folders on readonly repositories
1) Rename the unintuitive repository.syncfoldersto() to
sync_folder_structure()

2) We were checking if the local repository is readonly and then turning
off any folder creation. But as we can create folders on a remote
repository too, we need to be more fine grained here. Just don't create
a folder on the repository that is marked readonly=True.

This still does not do away with the error message that one currently
gets on missing local folders.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-05 14:05:51 +01:00
Sebastian Spaeth
0ccf06d5e6 Implement clean CTRL-C termination
Previously, we would simply bail out in an ugly way, potentially leaving
temporary files around etc, or while writing status files. Hand SIGINT
and SIGTERM as an event to the Account class, and make that bail out
cleanly at predefined points. Stopping on ctrl-c can take a few seconds
(it will e.g. finish to transfer the ongoing message), but it will shut
down cleanly.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-01-04 19:31:27 +01:00
Sebastian Spaeth
c93f8710a3 Init folder list early enough
We need the list of folders and the folder delimiter, but it was not
always retrieved early enough. E.g. when doing IMAP<->IMAP sync and the
local IMAP being readonly, we would bunk out with a mysterious error
message become repository.getsel() would still return None.

This commit fixes this error.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-12-01 10:31:43 +01:00
Sebastian Spaeth
f4a32bafd6 Pass ui.registerthread an Account() and not a name as string
This way, we can use all the account functions such as set_abort_event()
from the ui if needed.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-03 13:45:44 +01:00
Sebastian Spaeth
ab184d84e2 Reduce parameter list to account.syncfolder call
The remote|local|statusrepo is an anttribute of each SyncableAccount()
anyway, so we don't need to pass it in, we can simply get it from the
Account().

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-03 13:29:43 +01:00
Sebastian Spaeth
d54859a931 Don't setDaemon explicitly, it's done inherently
All ExitNotifyThreads and InstanceLimitThreads are setDaemon(True) in their
constructor, so there is no need to do that again in the code.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-11-02 11:29:23 +01:00
Sebastian Spaeth
cbec8bb5b2 Rework UI system to make use of the logging module
Logging was flawed as the output was e.g. heavily buffered and people
complained about missing log entries. Fix this by making use of the
standard logging facilities that offlineimap offers.

This is one big ugly patch that does many things. It fixes the
Blinkenlights backend to work again with the logging facilities.

Resize windows and hotkeys are still not handled absolut correctly, this
is left for future fixing. THe rest of the backends should be working fine.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-10-27 16:23:55 +02:00
Sebastian Spaeth
3885acf87d Implement server diagnostics
This outputs a handy summary of your server configuration and version
strings etc, which is useful for bug reporting.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-30 09:41:45 +02:00
Sebastian Spaeth
4e89bbfe6e Recache folder list after creating a new folder
If we create a new folder we would previously not update our folder
list, which led to us skipping the synchronization of those new folders
during the initial run (subsequent runs would pick it up).

Invalidate the folder cache when we create a folder during folder
structure sync. Regetting the whole list from an IMAP server might be
slightly suboptimal from a performance point, but it is easy and will
lead to consistent results. Hopefully we will not have to create new
folders on each new run.

Reported-by: Daniel Shahaf <d.s@daniel.shahaf.name>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-30 09:05:49 +02:00
Sebastian Spaeth
3b647d65be Output sync timing
Modify the UI:acct and acctdone functions to keep tab of the time
inbetween. Put self.ui.acct() and acctdone() at the right places in
accounts.py so that the timing happens at the right places.

While modifying that loop, flatten the nested try: try: except: finally:
constructs, we require python 2.5 now which copes with that.

At the end of each account sync you will now see something like:
*** Finished account 'test' in 0:05

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 17:26:46 +02:00
Sebastian Spaeth
ba396cf0ef More verbose loggin
Thread names are used to determine the logging header in the TTY ui. A
recent change made them too terse (basically only changing the account
name and not the folder names). Unbreak.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-29 16:59:36 +02:00
Sebastian Spaeth
1b6d76345a Robustify error msg against more failure
When syncfolder() fails, we output an error message containing the
foldername per the localfolder variable. However, the localfolder
variable is assigned inside our try: block and when the error occurs
there, we will have no localfolder variable to use for output. This
caused the errormsg to cause an Exception itself which unhelpfully
distracts from the root cause of the error.

Reconstruct the folder name in a bit more complex way, but in a way so
it is guaranteed to work (by relying on parameters passed in to the
function).

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-26 15:32:24 +02:00
Sebastian Spaeth
8ba17c5bd1 add sync_this variable to all Folder() instances
This variable shows if this folder should be synced or is disabled due to
a folderfilter statement. This lets us distinguish between a non-existent
folder and one that has been filtered out. Previously any filtered folder
would simply appear to be non-existing.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 11:35:52 +02:00
Sebastian Spaeth
0d95651417 Perform legacy global lock in addition to new per-account lock
We simply lock OfflineImap the same global way that we have always done
in addition to the previously implemented per-account lock. We can keep
both systems in parallel and then after a few stable releases, drop the
old-style global lock. by reverting this patch

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:34:25 +02:00
Sebastian Spaeth
c7938dc081 Per-account locking
Previously, we were simply locking offlineimap whenever it was
running. Howver there is no reason why we shouldn't be able to invoke it
in parallel, e.g. to synchronize several accounts in one offlineimap
each.

This patch implements the locking per-account, so that it is possible to
sync different accounts at the same time. If in refresh mode, we will
attempt to loop three times before giving up.

This also fixes Debian bug #586655

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2011-09-19 09:34:25 +02:00
Sebastian Spaeth
fe57140224 Fix repository 'readonly' configuration
The readonly feature was introduced to safeguard repositories from
accidental modifications. Unfortunately, my patch treated the readonly
setting as a string and not as a boolean, so if *anything* was set in
the configuration file as 'readonly', this value evaluated to True

Fortunately this was safe, we never treated a repository that we wanted
read-only as read-write. We always treated them readonly if something
was configured as "readonly=..." even if that was False.

The fix is simply to use getconfboolean() rather than getconf() which
checks for True/False/On/Off/yes/no/1/0 and hands back the correct boolean.

Reported-by: Tomasz Nowak <nowak2000@poczta.onet.pl>
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-09-15 19:20:27 +02:00
Vladimir Marek
c2fc81dd3d Make syncfoldersto to accept a single folder only
It is just historic relict

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-17 21:02:21 +02:00
Sebastian Spaeth
b47bc44783 accounts.py: Use ui.error when raising exceptions
Rather than using ui.warn, use ui.error() which outputs Exceptions to
the error log, saving them to a stack, so we get notified again at the
end of the sync run.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-08-11 19:14:28 +02:00
Nicolas Sebrecht
b25a72f8c8 cleanup: remove uneeded imports
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-09 22:42:15 +02:00
Nicolas Sebrecht
40900dcfb6 Merge branch 'ss/remove-siglistener-class' into next
Conflicts:
	offlineimap/accounts.py
2011-05-08 14:37:47 +02:00
Sebastian Spaeth
89619838b0 Remove weird SigListener class
The SigListener class was used to queue folders that we need to sync and
to receive "resync" and "abort" signals. It was undocumented and weird
and we had to pass "siglisteners" through the whole program.

Simply do away with it, and make 2 functions in the Account() class:
set_abort_event and get_abort_event which can be used to set and check
for such signals. This way we do not need to pass siglisteners all over
the place. Tested Blinkenlights and TTYUI uis to make sure that SIGUSR1
and SIGUSR2 actually still work.

Document those signals in MANUAL.rst. They were completly undocumented.

This simplifies the code and interdependencies by passing less stuff
around. Removes an undocumented and weirdly named class.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Signed-off-by: Nicolas Sebrecht <nicolas.s-dev@laposte.net>
2011-05-08 14:25:16 +02:00